Storage bucket identification method

Abstract

The present application relates to the technical field of cloud storage, and relates to a storage bucket identification method. The method comprises: S1, collecting enterprise name information, and acquiring enterprise domain name information on the basis of the enterprise name information and a preset data acquisition mode; S2, performing information crawling on the basis of the enterprise domain name information to obtain web page content in a corresponding domain name; S3, parsing the page content to obtain a page content parsing result, and determining storage bucket addresses on the basis of the page content parsing result; and S4, screening the storage bucket addresses on the basis of preset screening rules to generate a final storage bucket address set.

Description

Bucket identification method

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 2024108364902, filed on June 26, 2024, and entitled "A Bucket Identification Method", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present application belongs to the technical field of cloud storage, and particularly relates to a Bucket identification method. BACKGROUND

[0004] Bucket is a concept in cloud storage services, especially widely used in object storage services. It is a container for storing objects, which can be files, pictures, videos, and various data types. Each bucket has a unique name, and usually specifies a region when created to determine the location of data storage.

[0005] The traditional technology mainly aims at the discovery method of subdomain name, and the existing method mainly aims at the discovery of enterprise associated domain name. The enterprise may have multiple cloud service platform accounts. If the management is not standardized, it is difficult to collect all the accounts and obtain the corresponding AK / SK (wherein, AK is Access Key ID, and SK is Secret Access Key). The method of querying and obtaining the storage bucket address through the API (Application Programming Interface) provided by the public cloud is not enough to realize the discovery of all storage bucket addresses. SUMMARY

[0006] The present application provides a Bucket identification method, which comprises:

[0007] S1, collecting enterprise name information, and obtaining enterprise domain name information based on the enterprise name information and according to a preset data acquisition method;

[0008] S2, performing information crawling based on the enterprise domain name information to obtain web page content in the corresponding domain name;

[0009] S3, parsing the page content to obtain a page content parsing result, and determining a storage bucket address based on the page content parsing result;

[0010] S4, screening the storage bucket address based on a preset screening rule to generate a final storage bucket address set.

[0011] Further, in S1, the preset data acquisition manner at least includes searching enterprise domain name on a network space mapping platform, searching enterprise domain name on an attack surface management platform, discovering enterprise subdomain name by using an open source project, collecting domain name through a security device deployed by a company, searching enterprise domain name on a search engine, obtaining from company data, and obtaining domain name by using an API of a cloud service provider.

[0012] Further, S1 includes: performing deduplication processing on all obtained domain names, and storing the deduplicated domain names into a file or a database.

[0013] Further, the domain names are saved in a JSON format, and each domain name at least includes two fields, which are respectively used for recording domain name information and domain name source information.

[0014] Further, S3 includes: calling webpage content corresponding to the enterprise domain name information, analyzing the enterprise domain name information based on regular matching, and determining whether the enterprise domain name information contains a storage bucket address; if the enterprise domain name information contains the storage bucket address, the storage bucket address is recorded.

[0015] Further, S3 includes: calling webpage content corresponding to the enterprise domain name information, determining whether the enterprise domain name is in an XML format; if the webpage content corresponding to the enterprise domain name is all in the XML format, the corresponding storage bucket address is recorded.

[0016] Further, S3 includes: calling webpage content corresponding to the enterprise domain name information, calling header information, extracting a storage bucket feature from the header information, and performing feature matching based on the extracted storage bucket feature; if the matching is successful, the corresponding storage bucket address is extracted.

[0017] Further, S3 includes: if the extraction of the storage bucket feature from the header information fails, body information is called, a recognition feature is extracted from the body information, and feature matching is performed according to the recognition feature; if the matching is successful, the corresponding storage bucket address is extracted.

[0018] Further, in S3, if the matching based on the header information and the body information is unsuccessful, it is determined that the enterprise domain name information does not contain a storage bucket address.

[0019] Further, the preset screening rule at least includes determining a derivation source of the domain name, verifying an API of a cloud service provider, verifying whether the domain name is from enterprise data, and determining whether a storage bucket usage behavior matches the enterprise. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art based on the accompanying drawings without creative effort should be within the protection scope of the present application.

[0021] FIG. 1 is a flowchart of a storage bucket identification method provided by an embodiment of the present application;

[0022] FIG. 2 is a schematic diagram of obtaining enterprise domain name information provided by an embodiment of the present application;

[0023] FIG. 3 is a flowchart of storage bucket identification provided by an embodiment of the present application;

[0024] FIG. 4 is a flowchart of determining whether a storage bucket address is company-related provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should be within the protection scope of the present application.

[0026] In addition, in the following description, the description of the known structures and technologies is omitted to avoid unnecessary confusion of the concepts disclosed in the present application.

[0027] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance. The terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0028] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is presented in the context of the drawings, where like numbers represent like or similar elements throughout the several views. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0029] The application provides a storage bucket identification method, as shown in FIG. 1, comprising:

[0030] S1, collecting enterprise name information, and acquiring enterprise domain name information based on the enterprise name information and according to a preset data acquisition mode.

[0031] In this step, as shown in FIGS. 1 and 2, enterprise-related domain names are acquired through multiple ways. As shown in FIG. 2, the information of company / subsidiary / holding company main domain name and name needs to be collected. The information can be collected through the domain name information filing management system of the Ministry of Industry and Information Technology, or collected through a commercial query platform. These information are basic information for the discovery of subdomain names. Taking company A as an example, the full name of A is A Cloud Technology Co., Ltd. A search in the domain name information filing management system of the Ministry of Industry and Information Technology (https: / / beian.miit.gov.cn / ) can find 46 main domain name records. In the commercial query platform, it can be seen that A company has 83 branch organizations, which are the subsidiary and branch company information of A company. It can also be seen that A company has invested in three companies, B company, C company and D company, which are the holding companies of A company. The domain name of these three companies also needs to be searched in the domain name information filing management system.

[0032] The application proposes to acquire enterprise-related domain names through seven ways. Since the domain names acquired by a single way may not be comprehensive, multiple ways are needed to collect and summarize the domain names.

[0033] (1) Searching for enterprise-related domain names in a network space mapping platform.

[0034] There are currently many network space mapping platforms at home and abroad, such as Shodan, ZoomEye, FOFA, etc. Taking FOFA as an example, when it is known that ctyun.cn is one of the main domain names of Tianyi Cloud, the search syntax host="ctyun.cn" can be used to acquire all the subdomain name information under this main domain name.

[0035] (2) Searching for enterprise-related domain names in an attack surface management platform.

[0036] The current attack surface management platform also provides domain name discovery services for enterprises. For example, searching for Tianyi Cloud Technology Co., Ltd. on 0.zone (https: / / 0.zone / ) shows that there are 4244 domain name records.

[0037] (3) Use open source projects to discover enterprise subdomains.

[0038] There are also some open source projects for subdomain discovery, such as OneForAll (https: / / github.com / shmilylty / OneForAll), subfinder (https: / / github.com / projectdiscovery / subfinder), and subDomainsBrute (https: / / github.com / lijiejie / subDomainsBrute). For example, if you want to discover the subdomains of ctyun.cn, you can use the following command: python3 oneforall.py --target ctyun.cn run.

[0039] (4) Collect domain names through security devices deployed by the company.

[0040] Companies generally deploy many security devices, and some security devices also record domain name access records, such as Internet behavior management devices. However, the domain name records of these devices cannot directly distinguish whether the domain name is related to the company itself or whether it is a storage bucket related to the company. In S3, the present application will handle this situation.

[0041] (5) Search for enterprise-related domain names in search engines.

[0042] Subdomains can be collected in commonly used search engines such as Google, Baidu, and Bing. For example, you can search for site: ctyun.cn in Google to obtain subdomains related to ctyun.cn.

[0043] (6) Query company data.

[0044] Company data will have domain name records, but these information may not be complete, and sometimes due to departmental walls, it may not be possible to obtain the full amount of data. In addition, storage buckets are generally not recorded by enterprise asset managers. Therefore, the present application uses a combination of various means to discover enterprise-related domain names and obtain enterprise domain name information.

[0045] (7) Use cloud service provider APIs.

[0046] Cloud service providers will provide APIs to obtain the storage bucket address corresponding to the account. If the AK / SK of the account of the enterprise purchasing public cloud services can be obtained, it can be obtained in this way. However, different departments of an enterprise may have different accounts, and it is difficult to ensure that all relevant persons can be contacted to obtain the AK / SK of all accounts.

[0047] Finally, the collected domain names are de-duplicated and saved to a file or a database. For example, they can be saved as a JSON file, each containing two fields: domain, which records domain information, and from, which records domain sources. Note that there may be multiple domain sources. A record is shown as follows:

[0048] S2, based on enterprise domain name information, crawling information to obtain web page content corresponding to the domain name.

[0049] In this step, the web page content corresponding to the domain name is crawled by writing a crawler. There are many ways to implement a crawler. For example, in Python, it can be implemented through the requests library. The crawling results are saved to a file or a database, and both HTTP headers and bodies are saved. The format can be referred to as follows. Since the content of the header and the body is very long, only an example is given.

[0050] S3, parsing the page content to obtain page content parsing results, and determining the storage bucket address based on the page content parsing results.

[0051] In this step, as shown in FIG. 3, (1) some public cloud storage bucket addresses have a relatively fixed format, which can be identified by regular matching of the format. Taking Alibaba Cloud as an example, the address format of the Alibaba Cloud storage bucket is as follows: https: / / [bucket-name].oss-[region].aliyuncs.com

[0052] Among them, [bucket-name] is the name of the storage bucket, and [region] is the region where the Alibaba Cloud storage bucket is located. Therefore, regular expressions can be used to identify storage bucket addresses that conform to this format. However, not all public cloud storage bucket address formats are very standardized, and there are also private clouds and dedicated clouds. In addition, there are many public clouds, and it is difficult to ensure that none are missed. Therefore, only the address formats of several mainstream public clouds can be regularized. Other storage bucket addresses are identified in the following way.

[0053] (2) Determine whether the webpage of the domain name is in XML format. It can be determined by the Content-Type field in the header. If the webpage is in XML format, the content of this field will contain application / xml. Through the analysis of the storage buckets of multiple cloud vendors, these storage buckets will follow the Amazon bucket protocol format, and the webpage content is in XML format. Therefore, if the webpage is not in XML format, it must not be a storage bucket.

[0054] (3) The header part of the storage bucket address will have some characteristics to indicate that it is a storage bucket. For example, Huawei Cloud, the following content will appear in the header: Server: OBS x-obs-request-id: 000001***6F696 x-obs-bucket-location: cn-east-3 x-obs-id-2: 32AAAQ****k4NPt4e

[0055] Therefore, the features that can be extracted are Server: OBS, x-obs-request-id, x-obs-bucket-location, and x-obs-id-2, which can be used as Huawei Cloud storage bucket features. If they match, it means it is a Huawei Cloud storage bucket. Other clouds can also extract similar feature information.

[0056] (4) If the header part does not match the characteristics of the storage bucket, enter the body matching link. If the files in the storage bucket do not need authentication and can be accessed, the body will contain the ListBucketResult string, which can be used as a storage bucket feature. However, if the storage bucket needs a key to access, since the current probe does not have a key, it will be denied access. For example, Amazon Cloud, the return content of its denial of access is as follows: <?xml version="1.0" encoding="UTF-8"?> <error> <code>Access Denied< / code> <message>Access Denied< / message> <requestid>8H9B6J2K0V3B7CWK< / requestid> <hostid>oc6HOMY8***jKQ=< / hostid> < / error>

[0057] Features such as <code>Access Denied< / code> can also be extracted as storage bucket identification features. Note that the features may not be unique, but the core is to be able to distinguish this type of storage bucket.

[0058] (5) If neither the header nor the body features match, the domain name is not a storage bucket address.

[0059] Save the result to file or database. The format can refer to the following, domain is the domain address, isBucket indicates whether this domain is a storage bucket, bucketProvider indicates which cloud service provider provides this storage bucket. from is used to record the source of the domain name.

[0060] S4, screening the storage bucket address based on the preset screening rule to generate a final storage bucket address set.

[0061] In this step, the storage bucket address is filtered to obtain a final storage bucket address set. According to the following judgment mode, the priority decreases step by step, as shown in FIG. 4.

[0062] (1) If the domain name is derived from the company / subsidiary / branch / holding company master domain name, the domain name is a company-related storage bucket address.

[0063] (2) If the domain name comes from the API of the cloud service provider, the domain name is a company-related storage bucket address.

[0064] (3) If the domain name comes from the company's internal asset record table, the domain name is a company-related storage bucket address.

[0065] (4) If the domain name comes from the security equipment deployed by the company, and can match the company's storage bucket usage behavior rules, the domain name is a company-related storage bucket address.

[0066] A storage bucket address recorded by the security equipment deployed by the company may also be used by a third-party website when a company employee downloads external materials. However, the storage bucket address in this case will not be accessed by multiple people and frequently. Therefore, if a storage bucket address is accessed by multiple people and multiple times within a certain period of time, it can be considered that the address is related to the company. The specific time, number of people and number of times can be adjusted according to the actual situation of the company. For example, it can be considered that the storage bucket address is related to the company if more than 10 people access it more than 20 times within a month.

[0067] (5) Otherwise, the domain name is not related to the company's storage bucket address.

[0068] The application proposes a method for automatically identifying a storage bucket according to a URL format and page content, and can identify a cloud service provider providing a storage bucket service; and proposes a method for judging whether a storage bucket address is related to a company according to a storage bucket use behavior rule. Even if a storage bucket address is not operated by a company, but comes from a third-party SaaS (Software as a Service) service, if the address is accessed multiple times, it can also be identified, greatly improving the identification efficiency and comprehensiveness of the storage bucket.

[0069] The application proposes a storage bucket identification method, solves the problem of insufficient enterprise storage bucket identification capability, and can help an enterprise discover as many storage bucket addresses related to the enterprise as possible, and provide comprehensive data sources for subsequent sensitive data identification.

[0070] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered as falling within the scope of the present application.

[0071] The above-described embodiments only express several implementation manners of the application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patentable scope of the application. It should be pointed out that, for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A storage bucket identification method, comprising: S1, collecting enterprise name information, and acquiring enterprise domain name information based on the enterprise name information and according to a preset data acquisition mode; S2, performing information crawling based on the enterprise domain name information to acquire web page content in a corresponding domain name; S3, parsing the page content to obtain a page content parsing result, and determining a storage bucket address based on the page content parsing result; S4, screening the storage bucket address based on a preset screening rule to generate a final storage bucket address set.

2. The method of claim 1, wherein, In S1, the preset data acquisition mode at least includes enterprise domain name retrieval on a network space mapping platform, enterprise domain name retrieval on an attack surface management platform, enterprise subdomain name discovery using an open source project, domain name collection through a company-deployed security device, enterprise domain name retrieval on a search engine, acquisition from company materials, and domain name acquisition using an API of a cloud service provider.

3. The method of claim 1, wherein, S1 comprises: de-duplicating all acquired domain names, and storing the de-duplicated domain names in a file or a database.

4. The method of claim 3, wherein, The method further comprises: saving the domain names in a JSON format, and each domain name comprises at least two fields for recording domain name information and domain name source information.

5. The method of claim 1, wherein, S3 comprises: calling web page content corresponding to the enterprise domain name information, analyzing the enterprise domain name information based on regular matching, determining whether the storage bucket address is contained in the enterprise domain name information, and recording the storage bucket address if the storage bucket address is contained in the enterprise domain name information.

6. The method of claim 1, wherein, S3 comprises: calling web page content corresponding to the enterprise domain name information, determining whether the enterprise domain name is in an XML format, and recording the corresponding storage bucket address if the web page content corresponding to the enterprise domain name is in the XML format.

7. The method of claim 1, wherein, S3 comprises: calling the web page content corresponding to the enterprise domain name information, calling header information, extracting a storage bucket feature from the header information, and performing feature matching based on the extracted storage bucket feature; if the matching is successful, the corresponding storage bucket address is extracted.

8. The method of claim 7, wherein, In S3, if the matching of the storage bucket feature extracted from the header information fails, the body information is called, the identification feature is extracted from the body information, and feature matching is performed according to the identification feature; if the matching is successful, the corresponding storage bucket address is extracted.

9. The method of claim 8, wherein, In S3, if the matching according to the header information and the body information fails, it is determined that the enterprise domain name information does not contain the storage bucket address.

10. The method of claim 1, wherein, The preset screening rule at least includes determining the derived source of the domain name, verifying the API of the cloud service provider, verifying whether the domain name is from enterprise materials, and determining whether the storage bucket usage behavior matches the enterprise.

Images