Web universal vulnerability mining system and method, program product, equipment and medium

Abstract

The embodiment of the invention provides a Web universal vulnerability mining system and method, a program product, equipment and a medium, and relates to the technical field of network security. The system comprises an information crawling agent, a data tampering agent and a vulnerability verification agent. The information crawling agent is used for simulating a service operation executed by a user on a webpage of a target Web application and acquiring a request data packet; the data tampering agent is used for implanting a target attack load at a target parameter position in the request data packet for each type of universal vulnerabilities in multiple types of universal vulnerabilities of the Web application, and generating a verification data packet corresponding to the current universal vulnerabilities; the target parameter position and the target attack load are determined according to the current universal vulnerability; and the vulnerability verification agent is used for replaying the verification data packet and determining whether the request data packet has the current universal vulnerability according to a replaying result. According to the embodiment of the invention, the universal vulnerabilities in the Web application can be automatically and comprehensively mined, and the Web universal vulnerability mining efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and more specifically, to a Web general vulnerability mining system, method, program product, device, and medium. Background Art

[0002] Traditional web application vulnerability mining methods can be primarily categorized as manual analysis and automated tool detection. Manual analysis relies on testers' experience and technical expertise to manually analyze code, use penetration testing tools, and write customized scripts to scan and test web applications for vulnerabilities. This results in slow vulnerability mining and low accuracy. Automated tool detection uses rule-based tools designed for the known architecture and development languages of web applications. These tools exploit known vulnerabilities in web applications, making it difficult to mine common vulnerabilities in unknown web applications. Therefore, how to automatically and comprehensively mine common vulnerabilities in web applications and improve their efficiency has become a major challenge that urgently needs to be addressed. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a Web common vulnerability mining system, method, program product, device and medium to achieve the technical effect of automatically and comprehensively mining common vulnerabilities in Web applications and improving the efficiency of Web common vulnerability mining.

[0004] In a first aspect, an embodiment of the present application provides a general Web vulnerability mining system, including an information crawling agent, a data tampering agent, and a vulnerability verification agent;

[0005] The information crawling agent is used to simulate the business operations performed by the user on the web page of the target web application and obtain the request data packet;

[0006] The data tampering agent is configured to implant a target attack payload at a target parameter position in the request data packet for each of multiple common vulnerabilities in a Web application, and generate a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and the target attack payload are determined based on the current common vulnerability;

[0007] The vulnerability verification agent is used to replay the verification data packet and determine whether the request data packet contains the current common vulnerability based on the replay result.

[0008] In the above implementation process, by establishing an information crawling agent, a data tampering agent and a vulnerability verification agent, the information crawling agent is used to simulate the business operations performed by the user on the web page of the target Web application, and the request data packet is obtained. The data tampering agent is used to determine the target parameter position and target attack payload for each type of common vulnerability in the Web application according to the current common vulnerability, and the target attack payload is implanted in the target parameter position in the request data packet to generate a verification data packet corresponding to the current common vulnerability. The vulnerability verification agent is used to replay the verification data packet, and it is determined whether the request data packet contains the current common vulnerability based on the replay result. Based on AI technology, it can collaborate with multiple AI agents to mine multiple types of common vulnerabilities in unknown Web applications, automatically and comprehensively mine common vulnerabilities in Web applications, and improve the efficiency of Web common vulnerability mining.

[0009] Furthermore, the simulation of the business operation performed by the user on the webpage of the target web application to obtain the request data packet includes:

[0010] Loading the target web application's web page through a pre-configured headless browser, and after completing the loading of the target web application's web page, simulating the business operations performed by the user on the target web application's web page based on the user's operational behavior characteristics;

[0011] The pre-configured proxy tool intercepts the request data packet after completing the business operation performed by the simulated user on the web page of the target Web application.

[0012] In the above implementation process, by utilizing an information crawling agent, a pre-configured headless browser is used to load the web page of the target Web application, and after completing the loading of the target Web application web page, the business operations performed by the user on the target Web application web page are simulated according to the user's operation behavior characteristics, and after completing the simulation of the business operations performed by the user on the target Web application web page through a pre-configured proxy tool, the request data packet is intercepted, which can ensure that the information crawling agent simulates the user's real operations on the target Web application web page and effectively obtains the request data packet.

[0013] Furthermore, the information crawling agent is also used to parse the request data packet through the proxy tool and store the parsed data of the request data packet.

[0014] In the above implementation process, by utilizing the information crawling agent, parsing the request data packet through the proxy tool, and storing the parsed data of the request data packet, subsequent users can quickly obtain the parsed data of the request data packet for manual analysis.

[0015] Furthermore, for each of the multiple common vulnerabilities of the Web application, a target attack payload is implanted at a target parameter position in the request data packet to generate a verification data packet corresponding to the current common vulnerability, including:

[0016] Identifying a target parameter associated with the current common vulnerability from the request data packet and determining a location of the target parameter; wherein the target parameter includes at least one of a uniform resource locator (URL) query parameter, a request header parameter, and a request body parameter;

[0017] Constructing an initial attack payload targeting the current common vulnerability, and encoding the initial attack payload according to a target encoding method to obtain the target attack payload; wherein the target encoding method is determined according to the security policy of the attack target;

[0018] The target attack payload is implanted in the target parameter position to generate the verification data packet.

[0019] In the above implementation process, by utilizing the data tampering intelligent agent, for each of the multiple common vulnerabilities in Web applications, the target parameters associated with the current common vulnerability are identified from the request data packet, that is, at least one of the URL query parameters, request header parameters and request body parameters, the target parameter position is determined, and the initial attack payload for the current common vulnerability is constructed. The initial attack payload is encoded according to the target encoding method determined according to the security policy of the attack object to obtain the target attack payload, so as to implant the target attack payload at the target parameter position and generate a verification data packet. It can simulate attack detection for various common vulnerabilities in Web applications, covering multiple input points such as URL query parameters, request header parameters and request body parameters, and further ensure comprehensive mining of common vulnerabilities in Web applications.

[0020] Furthermore, replaying the verification data packet and determining whether the request data packet has the current common vulnerability according to the replay result includes:

[0021] Replaying the verification data packet to obtain the replay result;

[0022] A target analysis method is adopted to analyze whether the current common vulnerability actually exists according to the replay result, so as to determine whether the request data packet has the current common vulnerability; wherein the target analysis method is determined according to the current common vulnerability.

[0023] In the above implementation process, by utilizing the vulnerability verification intelligent agent, the verification data packet corresponding to any type of common vulnerability in the Web application is replayed, and the target analysis method determined according to the current common vulnerability is adopted. According to the obtained replay results, it is analyzed whether the current common vulnerability actually exists, so as to determine whether the request data packet contains the current common vulnerability. This can ensure the effective and accurate verification of the real existence of common vulnerabilities in Web applications, and further improve the efficiency of Web common vulnerability mining.

[0024] Furthermore, the system also includes a script generating agent;

[0025] The script generation agent is used to generate a test script based on the request data packet and the response data packet corresponding to the request data packet when it is determined that the current common vulnerability exists in the request data packet; wherein the test script includes a proof-of-concept PoC script and / or a vulnerability exploitation EXP script.

[0026] In the above implementation process, a script generation agent is established. When it is determined that a request data packet contains any type of common vulnerability of a Web application, the script generation agent generates test scripts such as PoC scripts and / or EXP scripts based on the request data packet and the response data packet corresponding to the request data packet, so that subsequent users can directly run the test scripts to mine common vulnerabilities in the target Web application.

[0027] In a second aspect, an embodiment of the present application provides a method for mining a general vulnerability on a Web, which is applicable to a general vulnerability mining system on a Web, wherein the system includes an information crawling agent, a data tampering agent, and a vulnerability verification agent;

[0028] The method comprises:

[0029] By using the information crawling agent, the business operations performed by the user on the web page of the target web application are simulated to obtain the request data packet;

[0030] By means of the data tampering agent, for each of multiple common vulnerabilities of a Web application, a target attack payload is implanted at a target parameter position in the request data packet, thereby generating a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and the target attack payload are determined based on the current common vulnerability;

[0031] The vulnerability verification agent replays the verification data packet, and determines whether the request data packet contains the current common vulnerability based on the replay result.

[0032] In a third aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a computer, the computer implements the method as described above.

[0033] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the method described above is implemented.

[0034] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A schematic diagram of the structure of a general Web vulnerability mining system provided in the first embodiment of the present application;

[0037] Figure 2 A flowchart of a general Web vulnerability mining method provided in the second embodiment of the present application;

[0038] Figure 3 A schematic structural diagram of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0040] It should be noted that in the description of this application, the terms "first" and "second" are used only to distinguish descriptions and should not be understood to indicate or imply relative importance. Furthermore, the step numbers herein are used only to facilitate the explanation of the embodiments of this application and do not limit the order in which the steps are to be executed.

[0041] In the relevant art, web application vulnerability mining methods can be mainly divided into manual analysis methods and automated tool detection methods. Manual analysis methods rely on testers' experience and technical accumulation to manually analyze code, use penetration testing tools, and write customized scripts to scan and test web applications for vulnerabilities and discover vulnerabilities within web applications. This method is slow and has low accuracy. Automated tool detection methods design rule-based tools based on the known architecture and development language of web applications. These tools are used to penetrate web applications for known vulnerabilities, but are difficult to detect common vulnerabilities in unknown web applications. Therefore, how to automatically and comprehensively discover common vulnerabilities in web applications and improve the efficiency of web common vulnerability mining has become a major challenge that urgently needs to be solved.

[0042] To this end, the present application proposes a method for mining common Web vulnerabilities, which establishes an information crawling agent, a data tampering agent and a vulnerability verification agent, utilizes the information crawling agent to simulate the business operations performed by users on the web pages of the target Web application, obtains the request data packet, utilizes the data tampering agent to determine the target parameter position and target attack payload for each of the multiple common vulnerabilities of the Web application according to the current common vulnerability, implants the target attack payload at the target parameter position in the request data packet, generates a verification data packet corresponding to the current common vulnerability, utilizes the vulnerability verification agent to replay the verification data packet, and determines whether the request data packet contains the current common vulnerability according to the replay result. Based on AI technology, it can collaborate with multiple AI agents to mine multiple types of common vulnerabilities in unknown Web applications, automatically and comprehensively mine common vulnerabilities in Web applications, and improve the efficiency of Web common vulnerability mining.

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0044] Please see Figure 1 , Figure 1A schematic diagram of the structure of a Web general vulnerability mining system provided in the first embodiment of the present application. The first embodiment of the present application provides a Web general vulnerability mining system, comprising an information crawling agent 101, a data tampering agent 102, and a vulnerability verification agent 103; the information crawling agent 101 is used to simulate the business operations performed by a user on the web page of a target Web application to obtain a request data packet; the data tampering agent 102 is used to, for each of multiple types of common vulnerabilities in a Web application, implant a target attack payload at the target parameter position in the request data packet to generate a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and target attack payload are determined based on the current common vulnerability; and the vulnerability verification agent 103 is used to replay the verification data packet and determine whether the request data packet contains the current common vulnerability based on the replay result.

[0045] As an example, based on actual business scenarios, AI (Artificial Intelligence) technology is applied to establish an information crawling agent 101, a data tampering agent 102, and a vulnerability verification agent 103. It is understandable that the information crawling agent 101, the data tampering agent 102, and the vulnerability verification agent 103 are AI agents.

[0046] The information crawling agent 101 simulates the business operations performed by the user on the web page of the target Web application selected for testing by the user and obtains the request data packet.

[0047] Web application web pages typically display elements such as network links, data forms, and operation controls. Users can open the web application web page through a front-end browser, view the network links, data forms, and operation controls on the web application web page, and select to perform a business operation by clicking a network link, a business operation by clicking an operation control, and a business operation by editing any data in the data form. The information crawling agent 101 can apply AI technology to simulate the business operations performed by users on the target web application web page and obtain request data packets.

[0048] OWASP (Open Web Application Security Project) has published many common security vulnerabilities in web applications, namely, many general vulnerabilities, including: Injection, such as SQL injection; Broken Authentication and Session Management; Sensitive Data Exposure; XML External Entities (XXE); Broken Access Control; Security Misconfiguration; Cross-Site Scripting (XSS); Insecure Deserialization; Using Components with Known Vulnerabilities; and Log Forging.

[0049] The data tampering agent 102 applies AI technology to analyze the data structure of the request data packet for each of the multiple common vulnerabilities in Web applications, determines the location of the tamperable parameters in the request data packet, that is, the target parameter location, and applies AI technology in combination with the collected historical vulnerability data, such as historical vulnerability reports, to perform analysis, determine the target attack payload that is likely to trigger the current common vulnerability, implant the target attack payload at the target parameter location in the request data packet, and generate a verification data packet corresponding to the current common vulnerability.

[0050] After obtaining the verification data packet corresponding to the current common vulnerability, the vulnerability verification agent 103 replays the verification data packet to obtain the replay result, and applies AI technology to analyze the actual existence of the current common vulnerability based on the replay result to determine whether the request data packet contains the current common vulnerability.

[0051] In actual applications, after the vulnerability verification agent 103 completes the verification of the verification data packets corresponding to multiple types of common vulnerabilities in Web applications, if it is determined based on the verification data packets corresponding to multiple types of common vulnerabilities in Web applications that the request data packet does not have multiple types of common vulnerabilities in Web applications, then the request data packet is marked as not having vulnerabilities.

[0052] The embodiment of the present application establishes an information crawling agent 101, a data tampering agent 102 and a vulnerability verification agent 103, uses the information crawling agent 101 to simulate the business operations performed by the user on the web page of the target Web application, obtains the request data packet, uses the data tampering agent 102 to determine the target parameter position and the target attack payload for each type of common vulnerability in the Web application according to the current common vulnerability, implants the target attack payload at the target parameter position in the request data packet, generates a verification data packet corresponding to the current common vulnerability, uses the vulnerability verification agent 103 to replay the verification data packet, and determines whether the request data packet contains the current common vulnerability based on the replay result. Based on AI technology, it can collaborate with multiple AI agents to mine multiple types of common vulnerabilities in unknown Web applications, automatically and comprehensively mine common vulnerabilities in Web applications, and improve the efficiency of Web common vulnerability mining.

[0053] In an optional embodiment, simulating a business operation performed by a user on a web page of a target web application to obtain a request data packet includes: loading the web page of the target web application through a preconfigured headless browser, and after completing the loading of the target web application web page, simulating the business operation performed by the user on the web page of the target web application based on the user's operation behavior characteristics; and intercepting the request data packet through a preconfigured proxy tool after completing the simulation of the business operation performed by the user on the web page of the target web application.

[0054] As an example, the information crawling agent 101 is pre-configured with a headless browser and proxy tools.

[0055] A headless browser is a web browser without a graphical user interface (GUI), such as Chrome headless. This type of browser programmatically simulates user interactions on web pages through a browser, while providing a rich API (Application Programming Interface) for the crawler agent 101 to perform web page interactions, which can reduce or even replace manual processing tasks. Because headless browsers render web pages in memory and not on the device front end, they have faster processing speeds.

[0056] In actual applications, the proxy tool can be configured by setting the proxy-server information of the headless browser when it is started.

[0057] In actual applications, in order to avoid being detected as an automated operation by the target Web application, anti-detection configuration is also required. The information crawling agent 101 can randomly set the User-Agent information of the headless browser to simulate different devices and browser environments.

[0058] After the information crawling agent 101 completes the environment configuration, it starts a pre-configured headless browser to load the web page of the target web application.

[0059] In actual applications, the information crawling agent 101 can use DOM (Document Object Model) to parse the web page of the target Web application, that is, the HTML structure, identify elements such as network links, data forms and operation controls in the web page, and render the web page in memory.

[0060] After determining that the headless browser has completed loading the web page of the target Web application, the information crawling agent 101 continues to simulate the business operations performed by the user on the web page of the target Web application through the headless browser according to the user's operation behavior characteristics.

[0061] In actual applications, the information crawling agent 101 can determine whether the headless browser has completed loading the target web application's web page based on a preset delay waiting strategy. For example, the information crawling agent 101 determines whether the time from the moment the headless browser was started to load the target web application's web page to the current moment has reached a preset waiting time, such as 30 seconds. If so, it is determined that the headless browser has completed loading the target web application's web page. Otherwise, it is determined that the headless browser has not completed loading the target web application's web page and the waiting process continues. The information crawling agent 101 can also detect whether the headless browser has completed loading the target web application's web page by querying memory.

[0062] In actual applications, the user's operation behavior characteristics include the user's mouse movement trajectory and movement speed in order to click on network links, operation controls and other elements. The information crawling agent 101 can simulate the user clicking on network links, operation controls and other elements in a slow, curved or jittery manner.

[0063] It is understandable that by learning the user's operational behavior characteristics to simulate the business operations performed by the user on the web page of the target Web application, it is possible to effectively avoid being detected as an automated operation by the target Web application.

[0064] The information crawling agent 101 intercepts the request data packets between the headless browser and the back-end server after completing the business operations performed by the simulated user on the web page of the target Web application through the pre-configured proxy tool.

[0065] The embodiment of the present application utilizes the information crawling agent 101 to load the web page of the target Web application through a pre-configured headless browser, and after completing the loading of the target Web application web page, simulates the business operations performed by the user on the target Web application web page according to the user's operation behavior characteristics, and intercepts the request data packet after completing the simulation of the business operations performed by the user on the target Web application web page through a pre-configured proxy tool, thereby ensuring that the information crawling agent 101 simulates the user's real operations on the target Web application web page and effectively obtains the request data packet.

[0066] In an optional embodiment, the information crawling agent 101 is also used to parse the request data packet through a proxy tool and store the parsed data of the request data packet.

[0067] As an example, after the information crawling agent 101 intercepts the request data packet through the proxy tool, it also parses the request data packet through the proxy tool and stores the parsed data of the request data packet.

[0068] For example, assume that the information crawling agent 101 simulates a user performing a business operation of submitting a data form on a web page of a target web application, and obtains a request data packet. In this case, the request data packet includes a GET request for reading data, a POST request for submitting data, a PUT request for updating data, and a DELETE request for deleting data. The agent parses the request data packet through a proxy tool to obtain parsed data of the request data packet, and stores the parsed data of the request data packet. The parsed data of the request data packet includes form parameters, cookies, and identity authentication information.

[0069] In actual applications, after the information crawling agent 101 intercepts the request data packet through the proxy tool, if the request data packet is an encrypted data packet, it is necessary to first use the proxy tool to use a man-in-the-middle attack (MITM) method to decrypt the request data packet, and then parse the decrypted request data packet.

[0070] In actual applications, the information crawling agent 101 can also extract key information from the parsed data of the request data packet through the proxy tool, such as the requested URL, parameters, response status code, response content, timestamp, user operation steps and execution results, etc. for storage.

[0071] The embodiment of the present application utilizes the information crawling agent 101, parses the request data packet through a proxy tool, and stores the parsed data of the request data packet, which can facilitate subsequent users to quickly obtain the parsed data of the request data packet for manual analysis.

[0072] In an optional embodiment, for each type of common vulnerability in multiple types of Web application common vulnerabilities, a target attack payload is implanted at a target parameter position in a request data packet, and a verification data packet corresponding to the current common vulnerability is generated, including: identifying a target parameter associated with the current common vulnerability from the request data packet and determining the target parameter position; wherein the target parameter includes at least one of a uniform resource locator (URL) query parameter, a request header parameter, and a request body parameter; constructing an initial attack payload for the current common vulnerability, and encoding the initial attack payload according to a target encoding method to obtain a target attack payload; wherein the target encoding method is determined according to the security policy of the attack target; implanting the target attack payload at the target parameter position, and generating a verification data packet.

[0073] As an example, after receiving the request data packet, the data tampering agent 102 applies AI technology to analyze the data structure of the request data packet for each of the multiple common vulnerabilities in the Web application, identifies the parameters in the request data packet, determines the target parameter associated with the current common vulnerability in the request data packet, and determines the position of the target parameter as the target parameter position.

[0074] Request packets typically contain URL query parameters (such as id=1), request header parameters (such as User-Agent and Cookie), and request body parameters (such as data form or JSON data in a POST request). The identification process requires understanding the semantics and functionality of the request packet. For example, the id parameter in the URL query parameter may be related to a database query, and the file parameter in the request body may involve file operations. This allows you to identify target parameters in the request packet that are associated with various common vulnerabilities.

[0075] The data tampering agent 102 also applies AI technology, such as natural language processing (NLP) and deep learning technology, to deeply analyze the data structure, parameter characteristics and historical vulnerability data of the request data packet, construct an initial attack payload that is easy to trigger the current common vulnerability, and determine the target encoding method according to the security policy of the attack target, such as URL encoding method, Base64 encoding method or Unicode encoding method, etc., encode the initial attack payload according to the target encoding method to obtain the target attack payload.

[0076] After determining the target parameter position and the target attack payload, the data tampering agent 102 implants the target attack payload into the target parameter position in the request data packet and generates a verification data packet corresponding to the current common vulnerability.

[0077] In actual applications, the data tampering agent 102 can dynamically adjust the encoding method or attack method of the attack payload according to the response content of the back-end server to the verification data packet.

[0078] For example, after sending the attack request, the data tampering agent 102 will make a judgment based on the response content returned by the back-end server: if the response status code is 200 and the return result is abnormal, the bypass will be detected as successful, and the Payload, that is, the target attack payload, will be recorded. The more effective Payload can be used for subsequent attacks; if the response content contains prompts such as "403Forbidden" or "WAF detected", the bypass will be detected as failed, and the encoding method or attack method will be automatically adjusted.

[0079] The embodiment of the present application utilizes the data tampering agent 102 to identify the target parameters associated with the current common vulnerability from the request data packet, i.e., at least one of the URL query parameters, request header parameters, and request body parameters, for each of the multiple common vulnerabilities in Web applications, determine the target parameter position, construct an initial attack payload for the current common vulnerability, and encode the initial attack payload in accordance with the target encoding method determined according to the security policy of the attack target to obtain the target attack payload, so as to implant the target attack payload at the target parameter position and generate a verification data packet. This can simulate attack detection for various common vulnerabilities in Web applications, covering multiple input points such as URL query parameters, request header parameters, and request body parameters, thereby further ensuring comprehensive mining of common vulnerabilities in Web applications.

[0080] In an optional embodiment, the verification data packet is replayed, and whether the request data packet has the current common vulnerability is determined based on the replay result, including: replaying the verification data packet to obtain a replay result; using a target analysis method to analyze whether the current common vulnerability actually exists based on the replay result to determine whether the request data packet has the current common vulnerability; wherein the target analysis method is determined based on the current common vulnerability.

[0081] As an example, after obtaining the verification data packet corresponding to any type of common vulnerability in the Web application, the vulnerability verification agent 103 uses a replay attack method to replay the verification data packet to obtain a replay result, and determines a target analysis method based on the current common vulnerability. The target analysis method is used to analyze whether the current common vulnerability actually exists based on the replay result. If the current common vulnerability actually exists, it is determined that the request data packet has the current common vulnerability. Otherwise, it is determined that the request data packet does not have the current common vulnerability.

[0082] In practical applications, there are mainly the following analysis methods for analyzing the real existence of vulnerabilities: Analysis method based on feature matching: The vulnerability verification agent 103 identifies common vulnerability response features through training models, such as error messages, abnormal return status codes, page content changes, etc., and matches the response features obtained after replaying the verification data packet with the previously identified known response features. If the match is successful, it is considered that the target attack payload successfully triggers the current common vulnerability, and it is determined that the request data packet has the current common vulnerability. Otherwise, it is considered that the target attack payload has not triggered the current common vulnerability, and it is determined that the request data packet has the current common vulnerability. Analysis method based on response time: For SQL injection and blind remote code execution (Blind Injection), the target attack payload is considered to have successfully triggered the current common vulnerability. RCE) and other common vulnerabilities that will cause abnormal processing time of the back-end server, the vulnerability verification intelligent agent 103 obtains the response time of the back-end server to the security data packet, and compares the response time of the back-end server to the verification data packet with the obtained response time. If the response time of the back-end server to the verification data packet is longer than the obtained response time, it is considered that the target attack payload has caused a processing delay problem on the back-end server, and it is determined that the request data packet has the current common vulnerability. Otherwise, it is considered that the target attack payload has not caused a processing delay problem on the back-end server, and it is determined that the request data packet does not have the current common vulnerability.

[0083] The embodiment of the present application utilizes the vulnerability verification agent 103 to replay the verification data packet corresponding to any type of common vulnerability in the Web application, adopts the target analysis method determined according to the current common vulnerability, and analyzes whether the current common vulnerability actually exists based on the obtained replay result to determine whether the request data packet contains the current common vulnerability. This can ensure the effective and accurate verification of the real existence of common vulnerabilities in Web applications, and further improve the efficiency of Web common vulnerability mining.

[0084] In an optional embodiment, the system further includes a script generation agent; the script generation agent is configured to generate a test script based on the request data packet and the response data packet corresponding to the request data packet when it is determined that the request data packet has a current common vulnerability; wherein the test script includes a proof-of-concept PoC script and / or a vulnerability exploitation EXP script.

[0085] As an example, AI technology is applied to establish a script-generating intelligent agent. It is understandable that the script-generating intelligent agent is an AI intelligent agent.

[0086] When the script generation agent determines that a request data packet contains any common vulnerability of a Web application, it extracts key information from the request data packet and the response data packet corresponding to the request data packet, such as the URL, method (such as GET / POST request), parameters (such as id=1'OR 1=1--) and request header in the request data packet, as well as the status code, content characteristics (such as SQL syntax error) and time delay in the response data packet, which contain information containing the complete request interaction logic. Based on its own semantic parsing and code generation capabilities, it converts these interaction logics into scripts that can be run independently and repeatedly verified to generate test scripts, where the test scripts include PoC scripts and / or EXP scripts.

[0087] It should be noted that the test script includes a PoC script and / or an EXP script, which is mainly divided into the following situations: the test script includes a PoC script; the test script includes an EXP script; the test script includes a PoC script and an EXP script.

[0088] The embodiment of the present application establishes a script generation agent. When the script generation agent determines that a request data packet contains any type of common vulnerability of a Web application, it generates a test script such as a PoC script and / or an EXP script based on the request data packet and the response data packet corresponding to the request data packet, making it convenient for subsequent users to directly run the test script to mine common vulnerabilities in the target Web application.

[0089] Please see Figure 2 , Figure 2 This is a flow chart of a method for mining a common Web vulnerability provided in the second embodiment of the present application. The second embodiment of the present application provides a method for mining a common Web vulnerability, applicable to a common Web vulnerability mining system, the system comprising an information crawling agent 101, a data tampering agent 102, and a vulnerability verification agent 103; the method comprises steps S201 to S203:

[0090] S201, using the information crawling agent 101, simulates the business operations performed by the user on the target web application web page to obtain the request data packet;

[0091] S202: For each of the multiple common vulnerabilities in the Web application, the data tampering agent 102 implants a target attack payload into the target parameter position of the request data packet, thereby generating a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and the target attack payload are determined based on the current common vulnerability;

[0092] S203, replay the verification data packet through the vulnerability verification agent 103, and determine whether the request data packet has the current common vulnerability based on the replay result.

[0093] In an optional embodiment, simulating a business operation performed by a user on a web page of a target web application to obtain a request data packet includes: loading the web page of the target web application through a preconfigured headless browser, and after completing the loading of the target web application web page, simulating the business operation performed by the user on the web page of the target web application based on the user's operation behavior characteristics; and intercepting the request data packet through a preconfigured proxy tool after completing the simulation of the business operation performed by the user on the web page of the target web application.

[0094] In an optional embodiment, the method further includes step S204:

[0095] S204. Through the information crawling agent 101, the request data packet is parsed through the proxy tool, and the parsed data of the request data packet is stored.

[0096] In an optional embodiment, for each type of common vulnerability in multiple types of Web application common vulnerabilities, a target attack payload is implanted at a target parameter position in a request data packet, and a verification data packet corresponding to the current common vulnerability is generated, including: identifying a target parameter associated with the current common vulnerability from the request data packet and determining the target parameter position; wherein the target parameter includes at least one of a uniform resource locator (URL) query parameter, a request header parameter, and a request body parameter; constructing an initial attack payload for the current common vulnerability, and encoding the initial attack payload according to a target encoding method to obtain a target attack payload; wherein the target encoding method is determined according to the security policy of the attack target; implanting the target attack payload at the target parameter position, and generating a verification data packet.

[0097] In an optional embodiment, the verification data packet is replayed, and whether the request data packet has the current common vulnerability is determined based on the replay result, including: replaying the verification data packet to obtain a replay result; using a target analysis method to analyze whether the current common vulnerability actually exists based on the replay result to determine whether the request data packet has the current common vulnerability; wherein the target analysis method is determined based on the current common vulnerability.

[0098] In an optional embodiment, the system further includes a script generating agent; the method further includes step S205:

[0099] S205. Generate an agent through a script. When it is determined that a current common vulnerability exists in a request data packet, generate a test script based on the request data packet and the response data packet corresponding to the request data packet; wherein the test script includes a proof-of-concept PoC script and / or a vulnerability exploitation EXP script.

[0100] The implementation process of the corresponding steps in the above method is specifically detailed in the implementation process of the functions and roles of each intelligent agent in the system described in the first embodiment of this application, and will not be repeated here.

[0101] The third embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a computer, the computer implements the method described in the second embodiment of the present application and can achieve the same beneficial effects.

[0102] The method described in the second embodiment of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in each embodiment of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model), or other programmable device.

[0103] A computer program or instruction can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, a computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. A computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media, such as floppy disks, hard disks, or magnetic tapes; optical media, such as digital video disks; or semiconductor media, such as solid-state drives. The computer-readable storage medium can be volatile or non-volatile, or can include both volatile and non-volatile types of storage media.

[0104] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in the fourth embodiment of the present application. The fourth embodiment of the present application provides an electronic device 30, comprising a processor 301, a memory 302, and a computer program stored in the memory 302 and configured to be executed by the processor 301; when the processor 301 executes the computer program, it implements the method described in the second embodiment of the present application and can achieve the same beneficial effects as described in the second embodiment.

[0105] In which, when the processor 301 reads the computer program from the memory 302 through the bus 303 and executes the computer program, it can implement the method of any embodiment included in the method described in the second embodiment of the present application.

[0106] Processor 301 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 301 can be a microprocessor.

[0107] The memory 302 can be used to store instructions executed by the processor 301 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of this application. The processor 301 of the embodiment of the present disclosure can be used to execute the instructions in the memory 302 to implement the method described in the second embodiment of this application. The memory 302 includes dynamic random access memory, static random access memory, flash memory, optical storage, or other memory known to those skilled in the art.

[0108] The fifth embodiment of the present application provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method described in the second embodiment of the present application, and can achieve the same beneficial effects as the method described in the second embodiment of the present application.

[0109] The method described in the second embodiment of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in each embodiment of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model), or other programmable device.

[0110] The computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0111] In summary, the embodiments of the present application provide a Web general vulnerability mining system, method, program product, device and medium, wherein the Web general vulnerability mining system includes an information crawling agent 101, a data tampering agent 102 and a vulnerability verification agent 103; the information crawling agent 101 is used to simulate the business operations performed by the user on the web page of the target Web application to obtain a request data packet; the data tampering agent 102 is used to implant a target attack payload at the target parameter position in the request data packet for each type of general vulnerability in the Web application, and generate a verification data packet corresponding to the current general vulnerability; wherein the target parameter position and the target attack payload are determined based on the current general vulnerability; the vulnerability verification agent 103 is used to replay the verification data packet and determine whether the request data packet has the current general vulnerability based on the replay result. The embodiment of the present application establishes an information crawling agent 101, a data tampering agent 102 and a vulnerability verification agent 103, uses the information crawling agent 101 to simulate the business operations performed by the user on the web page of the target Web application, obtains the request data packet, uses the data tampering agent 102 to determine the target parameter position and the target attack payload for each type of common vulnerability in the Web application according to the current common vulnerability, implants the target attack payload at the target parameter position in the request data packet, generates a verification data packet corresponding to the current common vulnerability, uses the vulnerability verification agent 103 to replay the verification data packet, and determines whether the request data packet contains the current common vulnerability based on the replay result. Based on AI technology, it can collaborate with multiple AI agents to mine multiple types of common vulnerabilities in unknown Web applications, automatically and comprehensively mine common vulnerabilities in Web applications, and improve the efficiency of Web common vulnerability mining.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A general Web vulnerability mining system, characterized in that: Including information crawling agents, data tampering agents and vulnerability verification agents; The information crawling agent is used to simulate the business operations performed by the user on the web page of the target web application and obtain the request data packet; The data tampering agent is configured to implant a target attack payload at a target parameter position in the request data packet for each of multiple common vulnerabilities in a Web application, and generate a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and the target attack payload are determined based on the current common vulnerability; The vulnerability verification agent is used to replay the verification data packet and determine whether the request data packet contains the current common vulnerability based on the replay result.

2. The system according to claim 1, wherein: The process of simulating a business operation performed by a user on a webpage of a target web application and obtaining a request data packet includes: Loading the target web application's web page through a pre-configured headless browser, and after completing the loading of the target web application's web page, simulating the business operations performed by the user on the target web application's web page based on the user's operational behavior characteristics; The pre-configured proxy tool intercepts the request data packet after completing the business operation performed by the simulated user on the web page of the target Web application.

3. The system according to claim 2, characterized in that The information crawling agent is also used to parse the request data packet through the proxy tool and store the parsed data of the request data packet.

4. The system according to claim 1, wherein: For each of the multiple common vulnerabilities of the Web application, a target attack payload is implanted at the target parameter position in the request data packet to generate a verification data packet corresponding to the current common vulnerability, including: Identifying a target parameter associated with the current common vulnerability from the request data packet and determining a location of the target parameter; wherein the target parameter includes at least one of a uniform resource locator (URL) query parameter, a request header parameter, and a request body parameter; Constructing an initial attack payload targeting the current common vulnerability, and encoding the initial attack payload according to a target encoding method to obtain the target attack payload; wherein the target encoding method is determined according to the security policy of the attack target; The target attack payload is implanted in the target parameter position to generate the verification data packet.

5. The system according to claim 1, wherein: The replaying of the verification data packet and determining whether the request data packet has the current common vulnerability according to the replay result includes: Replaying the verification data packet to obtain the replay result; A target analysis method is adopted to analyze whether the current common vulnerability actually exists according to the replay result, so as to determine whether the request data packet has the current common vulnerability; wherein the target analysis method is determined according to the current common vulnerability.

6. The system according to any one of claims 1 to 5, characterized in that The system also includes a script generating an intelligent agent; The script generation agent is used to generate a test script based on the request data packet and the response data packet corresponding to the request data packet when it is determined that the current common vulnerability exists in the request data packet; wherein the test script includes a proof-of-concept PoC script and / or a vulnerability exploitation EXP script.

7. A method for mining common web vulnerabilities, characterized in that: Applicable to a general Web vulnerability mining system, the system includes an information crawling agent, a data tampering agent, and a vulnerability verification agent; The method comprises: By using the information crawling agent, the business operations performed by the user on the web page of the target web application are simulated to obtain the request data packet; By means of the data tampering agent, for each of multiple common vulnerabilities of a Web application, a target attack payload is implanted at a target parameter position in the request data packet, thereby generating a verification data packet corresponding to the current common vulnerability; wherein the target parameter position and the target attack payload are determined based on the current common vulnerability; The vulnerability verification agent replays the verification data packet, and determines whether the request data packet contains the current common vulnerability based on the replay result.

8. A computer program product, characterized in that The computer program product comprises instructions which, when executed by a computer, cause the computer to implement the method according to claim 7 .

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the method according to claim 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to claim 7.

Images