Web API asset automatic mining system and method based on IAST technology

Abstract

The invention provides a Web AP I asset automatic mining system and method based on an IAST technology, and the system comprises a flow agent module which is used for capturing original flow data between a Web client and a server through a flow bypass agent; the flow cleaning module is used for cleaning and filtering the original flow data and screening out target flow data related to the Web AP I request; the framework identification module is used for identifying the framework type of the Web AP I request according to the data characteristics of the target flow data; the information extraction module is used for extracting asset information in the Web AP I request according to the frame type of the Web AP I request, and the asset information comprises an authentication parameter, a resource identifier and an authority identifier; and the asset library module is used for storing the asset information after duplicate removal and providing a management interface.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a Web API asset automatic mining system and method based on IAST technology. Background Art

[0002] As the critical hub for inter-enterprise connectivity and information exchange in the data age, Web APIs have become vital digital assets. Unfortunately, they have also become a prime target for attackers, making the security of these Web APIs a top priority for development and operations teams. This is where Web API asset mining technology emerges, becoming a critical link between Web API operation and security. Web API asset mining technology aims to help organizations discover and manage all Web APIs running in their network environments. This involves not only identifying existing Web API endpoints but also understanding their behavioral patterns, data exchange formats, and security measures.

[0003] However, traditional Web API asset mining technology has numerous limitations. First, it relies on dynamic testing, performing asset mining through external scanning and simulated attacks. This technology relies on expert experience, has a low degree of automation, and offers limited mining coverage for diverse API asset types. Second, it relies on static testing, instrumenting the web application's code layer to monitor its API calls. This serial instrumentation approach is highly invasive to businesses and can increase web application performance overhead.

[0004] Therefore, there is an urgent need for a universal automatic mining method for Web API assets to help enterprises automatically discover, classify and label API assets, thereby improving the efficiency and accuracy of API asset management, so as to better formulate security policies and respond to potential security threats. Summary of the Invention

[0005] In a first aspect of the present disclosure, a Web API asset automatic mining system based on IAST technology is provided, the system comprising:

[0006] Traffic proxy module, used to capture the original traffic data between the web client and the server through a traffic bypass proxy;

[0007] A traffic cleaning module is used to clean and filter the raw traffic data and select target traffic data related to Web API requests;

[0008] A framework identification module, configured to identify the framework type of the Web API request based on data characteristics of the target traffic data;

[0009] An information extraction module, configured to extract asset information from the Web API request, including authentication parameters, resource identifiers, and permission identifiers, based on the frame type of the Web API request;

[0010] The asset library module is used to store the deduplicated asset information and provide a management interface.

[0011] In combination with the first aspect, the traffic proxy module interactively simulates user behavior based on crawler technology and captures the original traffic data by intercepting and retransmitting HTTP / HTTPS data packets.

[0012] A second aspect of the present disclosure provides a method for automatically mining Web API assets based on IAST technology, the method comprising:

[0013] Based on IAST, interactively simulate user behavior on the web client, execute website functions, trigger web API requests, and generate raw traffic data;

[0014] Cleaning and filtering the raw traffic data to select target traffic data related to the Web API request;

[0015] Identify the framework type of the Web API request based on the request structure, usage protocol, and data format of the target traffic data;

[0016] Extracting asset information from the Web API request based on the framework type of the Web API request;

[0017] Build an API asset library, store the deduplicated asset information in the API asset library, and provide a management interface.

[0018] Combined with the second aspect, based on the IAST idea, interactively simulating user behavior on the Web client, executing website functions, triggering Web API requests, and generating original traffic data include using a traffic bypass proxy to capture the original traffic data between the Web client and the server.

[0019] In conjunction with the second aspect, the cleaning and filtering of the original traffic data to screen out target traffic data related to the Web API request includes:

[0020] Matching the target host name, port number, and request URL features, filtering the original traffic data to obtain first traffic data;

[0021] Excluding static resource request data and HTML response traffic data from the first traffic data to obtain second traffic data;

[0022] The second traffic data is filtered for traffic that meets the communication characteristics of the Web API request and deduplication is performed to obtain the target traffic data.

[0023] In conjunction with the second aspect, identifying the framework type of the Web API request based on the request structure, usage protocol, and data format of the target traffic data includes:

[0024] If the target traffic data uses a protocol containing / graphql / , it is determined to be GraphQL.

[0025] If the Content-Type field in the data format of the target traffic data contains application / soap+xml or exists <soap:envelope>Node, determined to be SOAP;

[0026] If the Content-Type field in the data format of the target traffic data is application / grpc or contains grpc-status, it is determined to be gRPC;

[0027] If no other features are matched, the default classification is REST framework type.

[0028] In conjunction with the second aspect, extracting the asset information from the Web API request according to the framework type of the Web API request includes:

[0029] If the framework type of the Web API request is REST, extract the resource identifier, permission identifier, and authentication parameters from the URL path, request header, and request body;

[0030] If the framework type of the Web API request is GraphQL, parse the query, mutation, or subscription fields in the request body and extract the field names, query parameters, and permission-related fields;

[0031] If the framework type of the Web API request is SOAP, parse the request body based on XML and extract <soap:envelope>or <soap:body>The resource identifier and permission-related fields in ;

[0032] If the framework type of the Web API request is gRPC, parse the .proto structure in the binary request body and extract the message fields and authentication-related parameters.

[0033] In conjunction with the second aspect, the step of constructing an API asset library, storing the deduplicated asset information in the API asset library, and providing a management interface includes:

[0034] Define the data structure of the API asset library, including API endpoints, framework types, resource identifiers, permission identifiers, authentication parameters and their locations, permission levels, sensitivity levels, and request and response examples;

[0035] Deduplication and storage of asset information, including:

[0036] Deduplication of extracted asset information based on API endpoints,

[0037] For recorded API assets, if the extracted permission level or sensitivity level changes, the record will be updated.

[0038] Insert the deduplicated asset information into the data table and generate a unique index.

[0039] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0040] one or more processors;

[0041] A storage unit is used to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the above-mentioned Web API asset automatic mining methods based on IAST technology.

[0042] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can implement any of the above-mentioned methods for automatic mining of Web API assets based on IAST technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a structural diagram of a Web API asset automatic mining system based on IAST technology according to an embodiment of the present disclosure;

[0044] Figure 2 A flowchart of a method for automatically mining Web API assets based on IAST technology according to an embodiment of the present disclosure;

[0045] Figure 3 Schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure.

[0047] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0049] like Figure 1 FIG2 is a schematic diagram of a Web API asset automatic mining system based on IAST technology according to an embodiment of the present disclosure. The system includes:

[0050] The traffic proxy module 110 is used to capture the original traffic data between the Web client and the server through a traffic bypass proxy;

[0051] The traffic cleaning module 120 is used to clean and filter the raw traffic data and select target traffic data related to the Web API request;

[0052] A framework identification module 130 is configured to identify the framework type of the Web API request based on data characteristics of the target traffic data;

[0053] An information extraction module 140 is configured to extract asset information from the Web API request, including authentication parameters, resource identifiers, and permission identifiers, based on the frame type of the Web API request.

[0054] The asset library module 150 is used to store the deduplicated asset information and provide a management interface.

[0055] The traffic proxy module 110 is the entry point of the entire system and is responsible for capturing the original traffic data between the web client and the server. The traffic proxy module 110 operates as a bypass proxy, monitoring and intercepting bidirectional HTTP / HTTPS communication traffic.

[0056] Traffic is transmitted to the analysis device through a mirror port or network diversion device without affecting normal business operations.

[0057] Crawler support: Simulates user interactions, such as form submissions and button clicks, to trigger API calls. It captures the HTTP / HTTPS packets corresponding to these triggering actions. This allows the module to resend data if traffic is not fully captured.

[0058] The traffic cleaning module 120 processes the original traffic data, removes irrelevant data (such as static resource requests, HTML page responses), and filters out target traffic data related to Web API requests.

[0059] Cleaning process:

[0060] Match target traffic: Filters raw data based on target host name, port number, and URL characteristics.

[0061] Eliminate invalid traffic: Static resource requests (such as images, CSS files, etc.) and HTML response traffic are excluded.

[0062] Feature screening: Further extract valid target traffic data based on API-specific communication features (such as request headers, content types, etc.).

[0063] Extended functionality: Machine learning models can be combined to identify API traffic characteristics to improve filtering efficiency.

[0064] Supports real-time traffic cleaning to avoid affecting the processing of downstream modules due to delays.

[0065] The framework identification module 130 automatically identifies the framework type (such as REST, GraphQL, SOAP, gRPC, etc.) used by the Web API request by analyzing the protocol, format, and structure of the target traffic data.

[0066] Identification logic:

[0067] REST framework: Based on the general HTTP protocol, requests and responses are usually sent in JSON format, with resource identifiers in the URL structure.

[0068] GraphQL framework: The request contains query, mutation, or subscription fields.

[0069] SOAP framework: uses XML format, with <soap:envelope>or <soap:body>Logo.

[0070] gRPC framework: The request is in binary format, and the protocol header contains application / grpc or grpc-status.

[0071] The information extraction module 140 extracts key asset information from the API request based on the identified frame type, including:

[0072] Authentication parameters: tokens, keys, or session IDs used for authentication and permission management.

[0073] Resource identifier: a unique identifier for accessing a resource, such as a file path or database ID.

[0074] Permission identifier: Defines the scope of the permission for the requested action, such as role permissions or access levels.

[0075] Extraction logic:

[0076] REST framework: Extract resource and authentication related fields from URL path, request headers, and request body.

[0077] GraphQL framework: parses the content of the query field in the request body and extracts the query field name and permission information.

[0078] SOAP framework: Parse the request body based on XML, extract <soap:envelope>or <soap:body>The data in .

[0079] gRPC framework: parses binary data and extracts fields and related authentication parameters from the .proto structure.

[0080] Asset library module 150: stores and manages the extracted asset information to ensure data consistency and traceability.

[0081] Management interface: Provides interfaces for querying, updating, and deleting assets, including:

[0082] Query API endpoints, framework types, and related permission information;

[0083] Update clearance levels or sensitivity classifications;

[0084] Data export or report generation function.

[0085] Data structure design: The core data structure of the asset library includes:

[0086] Basic fields: API endpoint, framework type, resource identifier, permission identifier, authentication parameters and their locations.

[0087] Extended fields: request and response examples, permission level, sensitivity level.

[0088] Beneficial effects: Through modular design, the system can efficiently and automatically mine Web API assets, adapt to complex scenarios of various Web development frameworks, and ensure the management convenience and integrity of asset information.

[0089] like Figure 2 FIG. 1 is a flow chart of a method for automatically mining Web API assets based on IAST technology according to an embodiment of the present disclosure, including:

[0090] S201: Based on IAST, interactively simulate user behavior on the web client, execute website functions, trigger web API requests, and generate raw traffic data;

[0091] S202: Cleaning and filtering the original traffic data to select target traffic data related to the Web API request;

[0092] S203: Identify the framework type of the WebAPI request based on the request structure, usage protocol, and data format of the target traffic data;

[0093] S204: Extracting asset information from the Web API request according to the framework type of the Web API request;

[0094] S205: Build an API asset library, store the deduplicated asset information in the API asset library, and provide a management interface.

[0095] Furthermore, based on the IAST idea, the web client interactively simulates user behavior, executes website functions, triggers web API requests, and generates raw traffic data, including using a traffic bypass proxy to capture raw traffic data between the web client and the server.

[0096] Specifically, in order to be compatible with different Web API framework attack tests, the HTTP / HTTPS data packet processing capabilities of mitmdump can be used to cut off the response of the generated traffic, thereby extracting all the request and response information of the traffic data packet without affecting the real-time effect of the traffic trigger simulation. For sites using HTTP 1.1 and below, use requests_raw to complete the traffic retransmission based on the original text of the request. For sites using HTTP 2.0, use httpx to complete the traffic retransmission based on the reorganization of various traffic information. On this basis, the traffic data generated by the interactive simulation of user behavior on the Web client based on the crawler script can be fully obtained in this step, but the traffic obtained in this step is only used as the original input traffic and needs to be further cleaned and processed.

[0097] Furthermore, the cleaning and filtering of the original traffic data to select target traffic data related to the Web API request includes:

[0098] Matching the target host name, port number, and request URL features, filtering the original traffic data to obtain first traffic data;

[0099] Excluding static resource request data and HTML response traffic data from the first traffic data to obtain second traffic data;

[0100] Filter the traffic in the second traffic data that meets the communication characteristics of the Web API request and remove duplicates to obtain the target traffic data.

[0101] Specifically, the original network traffic captured by the traffic proxy needs to be cleaned. This traffic may include Web API request response traffic, non-API traffic, and other network communication data.

[0102] In the Web API traffic cleaning flow, the target host name and target port cleaning are based on the user entry configuration or system configuration parameters. First, a full string match is performed. If it fails, a regular match attempt is made (applicable to scenarios such as fuzzy matching) to obtain the first traffic data.

[0103] Then, the traffic requesting static resources and the traffic with unexpected response formats are filtered. At this point, it can be basically determined that the traffic is Web API traffic, and the second traffic data is obtained.

[0104] Finally, by analyzing the communication behavior and characteristics of the traffic data (such as request method, URL pattern, Header field, Payload format, etc.), we can identify the traffic that meets the characteristics of Web API requests, and remove the duplicate request information in the filtered traffic data to obtain the target traffic data.

[0105] This step can be performed through predefined rules or machine learning-based models, such as identifying typical REST API requests (such as GET, POST and containing specific path parameters) or other types of APIs (such as SOAP, GraphQL).

[0106] Duplication may come from capturing the same operation multiple times or repeatedly transmitting network packets. Deduplication is achieved by comparing key fields of the traffic (such as request URL, parameters, authentication information, etc.).

[0107] Furthermore, identifying the framework type of the Web API request based on the request structure, usage protocol, and data format of the target traffic data includes:

[0108] If the target traffic data uses a protocol containing / graphql / , it is determined to be GraphQL.

[0109] If the Content-Type field in the data format of the target traffic data contains application / soap+xml or exists <soap:envelope>Node, determined to be SOAP;

[0110] If the Content-Type field in the data format of the target traffic data is application / grpc or contains grpc-status, it is determined to be gRPC;

[0111] If no other features are matched, the default classification is REST framework type.

[0112] Specifically, this step identifies different Web API framework types, such as REST, GraphQL, gRPC, SOAP, etc., by analyzing traffic characteristics such as the request structure, the protocol used, and the data format.

[0113] For SOAP framework, its response.headers["Content-Type"] is "application / soap+xml" or "text / xml", and response.body contains " <soap:envelope> ”、" <soap:body>"; For the gRPC framework, request.protocol is usually "HTTP / 2.0", request.method is usually "POST", and its response.headers["Content-Type"] is "application / grpc" or response.headers contains the "grpc-status" field; For the GraphQL framework, if request.method is "GET", request.url often contains a single endpoint such as " / graphql / ". If request.method is "POST", then its request.headers["Content-Type"] needs to be "a The most common REST framework is determined after the framework is determined above. Its request.method is not restricted, request.headers["Content-Type"] is usually "application / json" or "application / xml", and request.url often has a structure such as " / api / ", " / v1 / ", or " / v2 / ". Based on the above traffic characteristics and framework identification strategy, the four mainstream Web API frameworks of REST, GraphQL, gRPC, and SOAP can be effectively identified.

[0114] Furthermore, extracting the asset information from the Web API request according to the framework type of the Web API request includes:

[0115] If the framework type of the Web API request is REST, extract the resource identifier, permission identifier, and authentication parameters from the URL path, request header, and request body;

[0116] If the framework type of the Web API request is GraphQL, parse the query, mutation, or subscription fields in the request body and extract the field names, query parameters, and permission-related fields;

[0117] If the framework type of the Web API request is SOAP, parse the request body based on XML and extract <soap:envelope>or <soap:body>The resource identifier and permission-related fields in ;

[0118] If the framework type of the Web API request is gRPC, parse the .proto structure in the binary request body and extract the message fields and authentication-related parameters.

[0119] Specifically, this step uses a key parameter extraction strategy to deeply analyze Web API requests and extract key parameters that influence authentication decisions, such as authentication parameters (such as API keys and tokens), permission identifiers (such as user roles and permission levels), and resource identifiers (such as resource IDs and user IDs). This extracted parameter information is used to build a detailed profile of the API asset. The Web API key parameter information extraction strategy is as follows.

[0120] This step constructs different asset parsing functions based on the framework recognition results, but all perform asset parsing at four levels: URL query, request header, request cookie, and request body.

[0121] For GET API requests and gRPC APIs containing a ".Proto" request body (which can be difficult to unpack for binary data and can be ignored for attack testing), the request body is not important. The following describes the parsing strategy for the URL query, request header, request cookie, and request body.

[0122] Parsing of URL queries is relatively simple; you can directly extract the query and store it in a dictionary. Parsing of request cookies is based on request.headers["Cookie"]. If this field is in the request header, the cookie sub-items are parsed one by one and stored in a dictionary. Parsing of request bodies is mainly based on the request.headers["Content-Type"] field for preliminary judgment, and JSON format request body processing is implemented based on json.loads, and XML format request body processing is implemented based on xml.etree.ElementTree.fromstring. Regardless of the request body type, it is processed as a parameter key-value dictionary, including nested dictionaries. Parsing of request headers focuses on unconventional headers, such as excluding other request headers after the "User-Agent", "Accept", "Accept-Language", "Accept-Encoding", "Content-Type", "Content-Length", and "Cookie" fields, and storing these request header key values ​​in a dictionary.

[0123] On the basis of implementing Web API parameter parsing, it is necessary to identify the key parameters among these parameters, namely authentication items, resource identifiers, and permission identifiers.

[0124] To this end, the present invention constructs a detailed key parameter traffic feature library, which exists in a two-dimensional list structure, such as "[["parameter name","parameter value type","parameter name variant list","parameter value regular expression"],...]".

[0125] Based on these feature libraries, the present invention constructs a scalable and adaptive key parameter extraction strategy. First, all English letters in parameter names and their variant lists are capitalized, and all special characters are underlined. The two-dimensional list is sorted in ascending order by parameter name, and the variant lists themselves are also sorted in ascending order to facilitate search and comparison. For key parameter identification and extraction, the parameter name preprocessing logic is consistent with the feature library expansion logic, capitalizing letters, unifying special characters, and sorting in ascending order.

[0126] Then, a parameter name similarity matching with tolerance is performed (which can be based on Levenshtein's distance character similarity algorithm) to ensure that the parameter name can be correctly identified even when the parameter name changes slightly.

[0127] Regular expression matching is performed on parameters that remain unrecognized to increase recognition accuracy. Finally, to ensure the correct extraction of key parameters, this paper uses the traffic proxy's retransmission capability to verify each key parameter to be determined, i.e., "key parameter dependency verification," based on the logic that if key parameter replay requests are removed, the response will inevitably change significantly. Finally, a list of confirmed key parameter information is output.

[0128] Furthermore, the construction of the API asset library, storing the deduplicated asset information in the API asset library, and providing a management interface include:

[0129] Define the data structure of the API asset library, including API endpoints, framework types, resource identifiers, permission identifiers, authentication parameters and their locations, permission levels, sensitivity levels, and request and response examples;

[0130] Deduplication and storage of asset information, including:

[0131] Deduplication of extracted asset information based on API endpoints,

[0132] For recorded API assets, if the extracted permission level or sensitivity level changes, the record will be updated.

[0133] Insert the deduplicated asset information into the data table and generate a unique index.

[0134] Specifically, this step can be implemented based on a lightweight library such as sqlite3, predefine the api_assets data table structure, and encapsulate the corresponding table creation function.

[0135] This step creates or loads the corresponding data table based on the host address and port number during the first run, and the data table primary key id (integer) is automatically generated. At the same time, two functions are predefined: inserting API asset records and querying API asset records. The API endpoint structure and API duplicate asset calculation are based on the same basis; API permission level 0 represents the no permission level (unauthorized public users), 1 represents the normal permission level (normal users), and 2 represents the high permission level (administrators); API sensitivity levels increase from 0 to 3; API frameworks correspond to the four identifiers of "REST", "GraphQL", "gRPC", and "SOAP"; the host address and port number are based on the actual address of the traffic packet; request and response message examples are generated by the actual situation of the traffic agent; authentication parameters, resource identifiers, and permission identifiers refer to their parameter names, which can be empty depending on the extraction situation; the locations of authentication parameters, resource identifiers, and permission identifiers are generally "header", "query", "body", and "cookie", which can be empty depending on the extraction situation.

[0136] The present invention provides a Web API asset automatic mining system and method based on IAST technology. The system adopts a traffic proxy module to capture the original traffic data between the Web client and the server, and combines bypass proxy and crawler technology to cover various traffic scenarios; the traffic cleaning module cleans and filters the original traffic data to screen out the target traffic data related to the API request; the framework identification module identifies the framework type of the Web API request according to the protocol and format characteristics of the target traffic data; the information extraction module accurately extracts asset information such as authentication parameters, resource identifiers and permission identifiers according to different framework types; and finally, the asset library module deduplicates and stores the extracted asset information and provides a management interface, thereby achieving the purpose of fully automated mining of Web API assets and significantly improving the efficiency, accuracy and management convenience of asset discovery.

[0137] The electronic device 300 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 300 may include but is not limited to a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 This is merely an example of the electronic device 300 and does not constitute a limitation of the electronic device 300. The electronic device 300 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0138] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0139] The memory 302 can be an internal storage unit of the electronic device 300, such as a hard disk or memory of the electronic device 300. The memory 302 can also be an external storage device of the electronic device 300, such as a plug-in hard disk equipped on the electronic device 300, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the memory 302 can also include both an internal storage unit of the electronic device 300 and an external storage device. The memory 302 is used to store the computer program 303 and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or is about to be output.

[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0141] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0142] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0143] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0144] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0146] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0147] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should all be included in the scope of protection of the present disclosure.< / soap:body> < / soap:envelope> < / soap:body> < / soap:envelope> < / soap:envelope> < / soap:body> < / soap:envelope> < / soap:body> < / soap:envelope> < / soap:body> < / soap:envelope> < / soap:envelope>

Claims

1. A Web API asset automatic mining system based on IAST technology, characterized by: The system comprises: Traffic proxy module, used to capture the original traffic data between the web client and the server through a traffic bypass proxy; A traffic cleaning module, configured to clean and filter the raw traffic data and select target traffic data related to Web API requests; A framework identification module, configured to identify the framework type of the Web API request based on data characteristics of the target traffic data; An information extraction module, configured to extract asset information from the Web API request, including authentication parameters, resource identifiers, and permission identifiers, based on the frame type of the Web API request; The asset library module is used to store the deduplicated asset information and provide a management interface.

2. The system according to claim 1, wherein: The traffic proxy module interactively simulates user behavior based on crawler technology and captures the original traffic data by intercepting and retransmitting HTTP / HTTPS data packets.

3. A Web API asset automatic mining method based on IAST technology, characterized in that: The method comprises: Based on IAST, interactively simulate user behavior on the Web client, execute website functions, trigger Web API requests, and generate raw traffic data; Cleaning and filtering the raw traffic data to select target traffic data related to the Web API request; Identify the framework type of the Web API request based on the request structure, usage protocol, and data format of the target traffic data; Extracting asset information from the Web API request according to a frame type of the Web API request; Build an API asset library, store the deduplicated asset information in the API asset library, and provide a management interface.

4. The method according to claim 1, wherein Based on the IAST idea, the web client interactively simulates user behavior, executes website functions, triggers web API requests, and generates raw traffic data, including using a traffic bypass proxy to capture the raw traffic data between the web client and the server.

5. The method according to claim 1, wherein The cleaning and filtering of the original traffic data to select target traffic data related to the Web API request includes: Matching the target host name, port number, and request URL features, filtering the original traffic data to obtain first traffic data; Excluding static resource request data and HTML response traffic data from the first traffic data to obtain second traffic data; Filter the traffic in the second traffic data that meets the communication characteristics of the Web API request and remove duplicates to obtain the target traffic data.

6. The method according to claim 1, characterized in that The identifying the framework type of the Web API request according to the request structure, usage protocol, and data format of the target traffic data includes: If the target traffic data uses a protocol containing / graphql / , it is determined to be GraphQL. If the Content-Type field in the data format of the target traffic data contains application / soap+xml or exists <soap:envelope> Node, determined to be SOAP;< / soap:envelope> If the Content-Type field in the data format of the target traffic data is application / grpc or contains grpc-status, it is determined to be gRPC; If no other features are matched, the default classification is REST framework type.

7. The method according to claim 1, characterized in that The extracting the asset information from the Web API request according to the framework type of the Web API request includes: If the framework type of the Web API request is REST, extract the resource identifier, permission identifier, and authentication parameters in the URL path, request header, and request body; If the framework type of the Web API request is GraphQL, parse the query, mutation, or subscription fields in the request body to extract the field names, query parameters, and permission-related fields; If the framework type of the Web API request is SOAP, the request body is parsed based on XML to extract <soap:envelope>or <soap:body> The resource identifier and permission-related fields in ;< / soap:body> < / soap:envelope> If the framework type of the Web API request is gRPC, parse the .proto structure in the binary request body and extract the message fields and authentication-related parameters.

8. The method according to claim 1, characterized in that The steps of constructing an API asset library, storing the deduplicated asset information in the API asset library, and providing a management interface include: Define the data structure of the API asset library, including API endpoints, framework types, resource identifiers, permission identifiers, authentication parameters and their locations, permission levels, sensitivity levels, and request and response examples; Deduplication and storage of asset information, including: Deduplication of extracted asset information based on API endpoints, For recorded API assets, if the extracted permission level or sensitivity level changes, the record will be updated. Insert the deduplicated asset information into the data table and generate a unique index.

9. An electronic device, characterized in that: include: one or more processors; A storage unit for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for automatic mining of Web API assets based on IAST technology according to any one of claims 3 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can implement the automatic Web API asset mining method based on IAST technology according to any one of claims 3 to 8.

Images