A specific network resource extraction method

Abstract

The application belongs to the technical field of network resources, and particularly relates to a specific network resource extraction method, which comprises an Android simulator sandbox, a third-party network agent, a network resource analysis module, and performs network communication with a server through the third-party network agent. The specific network resource extraction method extracts and collects specific network resources through the application sandbox and the automatic information extraction technology, and can extract and collect part of the data encrypted through HTTPS and the network data in the HTTPS request compared with the historical scheme.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network resources, and particularly relates to a specific network resource extraction method. BACKGROUND

[0002] As an important basic technology of network security, the specific network resource extraction technology is used to draw a network node and network connection relationship diagram of a specific network asset device through active or passive detection methods, so as to portrait the network device. In historical research, there has been extraction and statistics of part of single instance type network resource data. However, with the passage of time, the use proportion of part of the resources has shown a downward trend, so that the statistical analysis result based on this part of data cannot reflect the current network overall state, and therefore it is necessary to expand the information extraction and analysis capability and carry out extraction and analysis of other network resources.

[0003] The current network resource extraction method is mainly used for extraction and collection of network resources without encrypted transmission, and only a small amount of data information (such as Host data) can be extracted for network data with encrypted transmission, and further data (such as URI, request header, request body, response header and response body) extraction and collection work cannot be carried out. SUMMARY

[0004] The application aims to provide a specific network resource extraction method to solve the problem that the network resources are mainly extracted and collected for network resources without encrypted transmission, and only a small amount of data information (such as Host data) can be extracted for network data with encrypted transmission, and further data (such as URI, request header, request body, response header and response body) extraction and collection work cannot be carried out.

[0005] To achieve the above purpose, the application provides the following technical scheme: a specific network resource extraction method, which consists of the following parts:

[0006] The Android emulator sandbox, the third-party network agent and the network resource analysis module are used to communicate with the server through the third-party network agent.

[0007] The Android emulator sandbox provides a running environment for mobile applications and customizes the network layer, so that the network behavior triggered in the application running process can be monitored, and the network data to be monitored can be forwarded to the third-party network agent module for processing.

[0008] The third-party network proxy module is used for proxy forwarding of network requests monitored by the Android emulator sandbox. The network proxy module actively attempts to intercept and analyze network resources of specific carriers (mainly encrypted and unverified HTTPS protocols), and marks and transmits the specific carrier resources to the network resource analysis module.

[0009] After the network resource analysis module obtains the transmitted network resources, it classifies the network resources according to types, and marks specific types and specific carriers of network resources according to requirements, thereby realizing the function of extracting specific types and specific carriers of network resources.

[0010] Preferably, the Android emulator sandbox uses a deeply customized ROM to provide a running environment for target applications, and realizes functions such as network resource recording, operation recording, sensitive behavior marking, and other functions to support network resource extraction and analysis.

[0011] Preferably, the Android emulator sandbox includes the following parts:

[0012] The basic layer: through deep customization and development of the system and kernel, a set of Hook mechanism is integrated, which can add breakpoints to key system APIs, mark sensitive behaviors and related triggering operations generated during application execution, and provide network resource recording function for subsequent data export.

[0013] The data layer: stores the operations, marked behaviors and generated network resources during the execution of the target application, and provides an extraction interface.

[0014] The business layer: provides a description of the current running state of the application, which is used to associate the application state with the related behaviors and network resources generated by it.

[0015] The result layer: outputs the final result in a specific format, which contains the network resources extracted during the application running process, application state and other information.

[0016] After the third-party network proxy module obtains the network request of the target application forwarded by the Android sandbox, it intercepts the network request through man-in-the-middle technology, records and marks the requests that have not been verified through the encrypted channel.

[0017] Preferably, the third-party network proxy module intercepts the HTTPS handshake request of the application, and then sends the handshake request to the server. After the server returns the certificate and the third-party network proxy module completes the encryption key negotiation, the third-party network proxy module transmits its own certificate to complete the encryption key negotiation with the application.

[0018] Compared with the prior art, the specific network resource extraction method has the beneficial effects that the specific network resource extraction method extracts and collects specific network resources by applying a sandbox and automatic information extraction technology, can extract and collect part of data encrypted by HTTPS, and extracts network data in an HTTPS request. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are used to explain the application, and do not constitute a limitation of the application. In the drawings:

[0020] Figure 1 is a flowchart of the application;

[0021] Figure 2 is a structure diagram of the emulator sandbox of the application;

[0022] Figure 3 is a schematic diagram of the man-in-the-middle technology of the application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0024] A specific network resource extraction method, which consists of the following parts:

[0025] An Android emulator sandbox, a third-party network agent, and a network resource analysis module, and network communication with a server through the third-party network agent;

[0026] The Android emulator sandbox provides a running environment for a mobile application and customizes a network layer, so that network behaviors triggered in the running process of the application can be monitored, and network data that needs to be monitored can be forwarded to the third-party network agent module for processing;

[0027] The third-party network agent module is used to proxy and forward network requests monitored by the Android emulator sandbox. The network agent module actively attempts to intercept and analyze network resources of specific carriers (mainly encrypted and unchecked HTTPS protocols), marks the specific carrier resources, and transmits the specific carrier resources to the network resource analysis module;

[0028] The network resource analysis module classifies the transmitted network resources according to categories, and marks the network resources of specific categories and specific carriers according to requirements, thereby realizing the extraction of network resources of specific categories and specific carriers.

[0029] Further, the Android emulator sandbox uses a deeply customized ROM to provide a running environment for target applications, realizes network resource recording, operation recording, sensitive behavior marking and other functions to support network resource extraction and analysis.

[0030] Further, the Android emulator sandbox includes the following parts:

[0031] The base layer: through deep customization and development of the system and kernel, a set of Hook mechanism is integrated, which can add breakpoints to key system APIs, mark sensitive behaviors and related triggering operations generated during application execution, and provide network resource recording functions for subsequent data export.

[0032] The data layer: stores the operations, marked behaviors and generated network resources during the execution of the target application, and provides an extraction interface.

[0033] The business layer: provides a description of the current running state of the application, which is used to associate the application state with the related behaviors and network resources generated thereby.

[0034] The result layer: outputs the final results in a specific format, which includes the network resources extracted during the execution of the application, the application state and other information.

[0035] After obtaining the network request of the target application forwarded by the Android sandbox, the third-party network proxy module intercepts the network request through the man-in-the-middle technology, records and marks the requests that have not been verified through the encrypted channel.

[0036] Further, the third-party network proxy module intercepts the HTTPS handshake request of the application, and then sends the handshake request to the server. After the server returns the certificate and the third-party network proxy module completes the encryption key negotiation, the third-party network proxy module transmits its own certificate to complete the encryption key negotiation with the application.

[0037] The third-party network proxy module intercepts the HTTPS handshake request of the application, and then sends the handshake request to the server. After the server returns the certificate and the third-party network proxy module completes the encryption key negotiation, the third-party network proxy module transmits its own certificate to complete the encryption key negotiation with the application. If the application does not check the server certificate of this HTTPS request, the application transmits encrypted data to the third-party network proxy module first, and then the third-party network proxy module decrypts and re-encrypts the data with the key negotiated with the server before transmitting the data to the server. When the server returns data, the data is first decrypted by the third-party network proxy module and then encrypted with the key negotiated with the application before being transmitted to the server. Therefore, the encrypted network resources can be extracted and marked by the third-party network proxy module.

[0038] During the running of the target application, there may be various network resource transmissions, including unencrypted transmission, encrypted unverified transmission, and encrypted verified transmission. If the man-in-the-middle is used for the encrypted verified transmission network resource transmission request, the request will fail, resulting in incorrect application running. Therefore, the third-party network proxy module will take release measures for the request that has failed a man-in-the-middle attack, that is, not using the man-in-the-middle technology for the request.

[0039] Because there may be both encrypted verified transmission and encrypted unverified transmission in multiple requests to the same IP, to improve the extraction rate of specific network resources, the third-party network proxy module will dynamically adjust the number of release requests for the IP according to the number of historical man-in-the-middle attack failures of the IP.

[0040] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for extracting specific network resources, characterized in that: The specific network resource extraction method consists of the following parts: Android emulator sandbox, third-party network proxy, network resource parsing module, and communicate with the server via the third-party network proxy; The Android emulator sandbox provides a runtime environment for mobile applications and customizes the network layer. It can monitor network behaviors triggered during application operation and forward network data that needs to be monitored to a third-party network proxy module for processing. The third-party network proxy module is used to forward network requests detected by the Android emulator sandbox. The network proxy module will actively attempt to intercept and parse network resources on a specific carrier, which is an encrypted unverified HTTPS protocol. It will mark these specific carrier resources and transmit them to the network resource parsing module. After obtaining the transmitted network resources, the network resource parsing module will classify the network resources according to their types and mark network resources of specific types and carriers according to the requirements, thereby realizing the function of extracting network resources of specific types and carriers. The third-party network proxy module intercepts the application's HTTPS handshake request and then sends the handshake request to the server. After the server returns its certificate and completes the encryption key negotiation with the third-party network proxy module, the third-party network proxy module then transmits its own certificate to complete the encryption key negotiation with the application. If the application does not verify the server certificate for this HTTPS request, the application will first transmit encrypted data to the third-party network proxy module. After the third-party network proxy module decrypts the data, it will re-encrypt it using the key negotiated with the server before transmitting it to the server. When the server returns data, it will also first decrypt it using the third-party network proxy module before using the key negotiated with the application for encrypted communication.

2. The method for extracting specific network resources according to claim 1, characterized in that: The Android emulator sandbox uses a deeply customized ROM to provide a runtime environment for target applications, and implements functions such as network resource recording, operation recording, and sensitive behavior marking, supporting the extraction and analysis of network resources.

3. The method for extracting specific network resources according to claim 2, characterized in that: The Android emulator sandbox includes the following components; The base layer integrates a Hook mechanism through deep customization of the system and kernel. It can add breakpoints to key APIs of the system, mark sensitive behaviors and related triggering operations generated during application execution, and provide network resource recording function for subsequent data export. Data layer: Stores the operations, marked behaviors, and generated network resources during the execution of the target application, and provides an extraction interface; Business layer: Provides a description of the application's current running state, used to associate the application state with its related behaviors and network resources; Results layer: Outputs the final results in a specific format, including network resources extracted during application execution and application status information. After obtaining the target application's network requests forwarded by the Android sandbox, the third-party network proxy module intercepts the network requests using man-in-the-middle technology, recording and marking any requests that have passed through an encrypted channel but have not been verified.

Images