Asset acquisition method, apparatus, device, and readable storage medium
By determining the relationship between query type and IP address, and combining full port scanning and reverse DNS lookup, the problem of low success rate in acquiring target assets was solved, resulting in more efficient asset acquisition and improved search engine stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING HUASHUN XIN AN TECH CO LTD
- Filing Date
- 2023-08-01
- Publication Date
- 2026-06-19
Smart Images
Figure CN117112878B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cyberspace mapping, and in particular to an asset acquisition method, apparatus, device, and readable storage medium. Background Technology
[0002] Currently, in the field of cyberspace mapping, when users search for and obtain target assets through search engines, in order to ensure the timeliness of the assets, search engines generally perform a full network scan of important ports to obtain the latest assets of all IPs for querying. After obtaining the specific information queried by the user, the search engine finds the latest assets of several IPs under the paths of important ports corresponding to the specific information queried by the user in the latest assets, so as to determine the target asset.
[0003] However, due to the impracticality of performing a full network scan of all ports to obtain the latest assets of all IPs, the latest assets are generally obtained by scanning the important ports of all IPs, which may result in the remaining assets not being included in the latest assets. Therefore, when obtaining the target asset from the latest assets, the target asset may not be obtained because it is in the remaining assets. Here, the remaining assets refer to the assets corresponding to the remaining ports of all IPs, and the remaining ports refer to the ports other than the important ports in all ports.
[0004] Therefore, how to provide a solution to the low probability of successfully acquiring target assets is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0005] In order to improve the success rate of asset acquisition, this application provides an asset acquisition method, apparatus, device and readable storage medium.
[0006] Firstly, this application provides a method for acquiring assets, employing the following technical solution:
[0007] An asset acquisition method includes:
[0008] Upon receiving a query statement, the system determines, through matching, whether the corresponding asset exists in the database. The database includes assets obtained by scanning multiple important ports.
[0009] If it does not exist, then determine whether the query statement is a domain name;
[0010] If not, then based on the preset correspondence between query statements and IP addresses, several first IP addresses corresponding to the query statement are obtained;
[0011] Based on the aforementioned first IP addresses, the first enhanced target asset is obtained by scanning all ports.
[0012] By adopting the above technical solution, when a query statement that is not a domain name cannot be matched with a corresponding asset in the database, it is determined whether the query statement is a domain name. If it is not a domain name, it means that the query statement needs to indirectly obtain the target asset through an IP address. Specifically, based on the preset correspondence between query statements and IP addresses and the query statement itself, several first IP addresses corresponding to the query statement can be obtained. Based on these first IP addresses, the first enhanced target asset can be obtained by scanning all ports. Compared to the impossibility of performing a full network scan on all ports of all IP addresses to obtain the latest assets, performing a full port scan on several IP addresses is equivalent to performing a full port scan on a small number of assets, which is feasible. Moreover, the assets obtained by scanning all ports can cover all assets of several IPs in the public network, reducing the probability of failure to obtain the first enhanced target asset and achieving successful asset acquisition.
[0013] In a preferred example, this application can be further configured as follows:
[0014] The database includes assets corresponding to at least multiple domain names.
[0015] After determining whether the query statement is a domain name, the process also includes:
[0016] If so, then based on the assets corresponding to each of the multiple domain names and the query statement, determine whether there are assets corresponding to the query statement;
[0017] If it does not exist, then based on the query statement, several second IP addresses corresponding to the query statement are obtained through reverse DNS lookup.
[0018] Based on several second IP addresses, the second enhanced target asset is obtained by scanning all ports;
[0019] Use either the first enhanced target asset or the second enhanced target asset as the first target asset.
[0020] By adopting the above technical solution, since the assets corresponding to the domain name can be directly queried in the database when the query statement is a domain name, when the assets corresponding to the domain name in the query statement cannot be found in the database, a new query needs to be performed across the entire network. However, to access a server on the Internet, it is ultimately necessary to use an IP address. Therefore, based on the query statement, several second IP addresses corresponding to the query statement can be obtained through reverse DNS lookup of the domain name. Based on these second IP addresses, the second enhanced target asset can be obtained by scanning all ports, and either the first or second enhanced target asset can be used as the first target asset. Thus, providing a query method for domain names can improve the search engine's adaptability to various query statement formats. Using all ports, rather than just important ports, to scan the entire network to obtain the second enhanced target asset can increase the probability of obtaining it.
[0021] In a preferred example, this application can be further configured as follows:
[0022] The IP address includes a first IP address, and the first target asset includes a first enhanced target asset; or, the IP address includes a second IP address, and the first target asset includes a second enhanced target asset.
[0023] After specifying either the first enhanced target asset or the second enhanced target asset as the first target asset, the method further includes:
[0024] Based on several IP addresses, clustering is used to obtain the IP ranges that are commonly associated with these IP addresses;
[0025] Based on the IP segment and the plurality of IP addresses, a plurality of other IP addresses are obtained, wherein the other IP addresses refer to IP addresses in the IP segment other than the plurality of IP addresses;
[0026] The second target asset was obtained by performing a full port scan on the other IP addresses.
[0027] The first target asset and the second target asset are used as the final target assets.
[0028] By adopting the above technical solution, after obtaining IP segments commonly corresponding to several IP addresses through clustering, to avoid repeatedly obtaining assets corresponding to each of the several IP addresses, several other IP addresses can be obtained from the IP segments. A full port scan is then performed on these other IP addresses to obtain the second target asset. The first and second target assets are then used as the final target assets. After scanning the full ports of several IP addresses to obtain the first target asset, further scanning the full ports of several other IP addresses within the same IP segment to obtain the second target asset can increase the probability that the search engine successfully obtains the asset corresponding to the query statement.
[0029] In a preferred example, this application can be further configured as follows:
[0030] Before obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses, the method further includes:
[0031] From the plurality of first IP addresses, a plurality of first valid IP addresses are determined, wherein the first valid IP address represents the first IP address where asset information exists;
[0032] Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes:
[0033] Based on the aforementioned first valid IP addresses, the first enhanced target asset is obtained by scanning all ports.
[0034] By adopting the above technical solution, resources can be avoided by investing in second IP addresses that cannot acquire assets, thus avoiding resource waste for the search engine when acquiring assets. Furthermore, when the search engine has limited idle resources, avoiding the use of second IP addresses that cannot acquire assets can also improve the search engine's response speed.
[0035] In a preferred example, this application can be further configured as follows:
[0036] The step of obtaining the first enhanced target asset by scanning all ports based on the plurality of first valid IP addresses includes:
[0037] Based on the aforementioned first valid IP addresses, identify several first valid ports corresponding to each first valid IP address from all ports, wherein each valid port represents a port open to each first valid IP address.
[0038] Multiple common domain names are obtained, and for each first valid IP address, under each first valid port, the domain names of the multiple common domain names are collected to obtain several first valid domain names corresponding to the first valid IP address. The first valid domain name represents the subdomain of the common domain name containing asset information.
[0039] Under each valid port, for each first valid IP address, based on several first valid domain names corresponding to the first valid IP address, the first unit target asset corresponding to the first valid IP address is obtained, wherein each first unit target asset represents the total assets obtained from all first valid domain names under each valid IP address;
[0040] All first-unit target assets are designated as the first enhanced target assets.
[0041] By adopting the above technical solution and identifying valid domain names among common domain names, resources can be avoided by investing in domain names from which assets cannot be acquired, thus preventing a waste of search engine resources when acquiring assets. Furthermore, when the search engine has limited available resources, avoiding the use of second IP addresses from acquiring assets can also improve the search engine's response speed.
[0042] In a preferred example, this application can be further configured as follows:
[0043] After determining whether the asset corresponding to the query statement exists, the process also includes:
[0044] Get the current resource ownership status of the search engine;
[0045] Based on the current resource occupancy status, the port scanning method is determined. When the current resource occupancy status is idle, the step of obtaining the first enhanced target asset by scanning all ports based on the several first IP addresses is executed.
[0046] Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes:
[0047] When the current resource occupancy status is busy, the first enhanced target asset is obtained by scanning all ports except for some ports with a utilization rate of 0, based on the several first IP addresses.
[0048] By adopting the above technical solution, and by obtaining the current resource occupancy status of the search engine, the query method for the first enhanced target asset can be determined as a full-port query, or by scanning all ports except for some ports with a usage rate of 0. This can adapt to the busy or idle status of the search engine, improve the flexibility of the search engine query process, and thus improve the stability of the search engine operation.
[0049] In a preferred example, this application can be further configured as follows:
[0050] Before receiving the query statement, the following is also included:
[0051] Get all IP addresses;
[0052] Based on all the IP addresses, the latest common assets corresponding to each IP address are obtained by periodically scanning multiple important ports. The latest common assets represent the sum of the latest assets that each IP address can obtain through the paths of multiple important ports.
[0053] Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes:
[0054] The scanning process for acquiring target assets is given the highest priority, and based on the highest priority, the latest common assets corresponding to each of the several first IP addresses are taken as the first enhanced target assets.
[0055] By adopting the above technical solution, we can scan all IP addresses for common ports among multiple important ports and ports that are prone to causing search engine crashes. This allows us to obtain common assets and assets that are likely to cause search engine crashes during the query process in advance. While ensuring feasibility and avoiding excessive waste of resources, we can further improve the stability of the search engine query process. Prioritizing the task of obtaining target assets to the highest priority can ensure the speed of obtaining target assets.
[0056] Secondly, this application provides an asset acquisition device, which adopts the following technical solution:
[0057] An asset acquisition device, comprising:
[0058] The query statement processing module is used to receive a query statement and, based on the query statement, determine whether the asset corresponding to the query statement exists in the database through matching. If the asset corresponding to the query statement does not exist, the query statement judgment module is triggered.
[0059] The query statement judgment module is used to determine whether the query statement is a domain name. If it is not a domain name, the first IP address acquisition module is triggered.
[0060] The first IP address acquisition module is used to obtain several first IP addresses corresponding to the query statement based on the preset correspondence between the query statement and the IP address.
[0061] The first target asset enhancement module is used to obtain the first enhanced target asset by scanning all ports based on the aforementioned first IP addresses.
[0062] Thirdly, this application provides an electronic device that adopts the following technical solution:
[0063] At least one processor;
[0064] Memory;
[0065] At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: perform the asset acquisition method of any of the above.
[0066] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:
[0067] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the asset acquisition methods described above.
[0068] In summary, this application includes at least one of the following beneficial technical effects:
[0069] 1. When a query statement that is not a domain name cannot be matched with a corresponding asset in the database, it is determined whether the query statement is a domain name. If it is not a domain name, it means that the query statement needs to indirectly obtain the target asset through an IP address. Specifically, based on the preset correspondence between query statements and IP addresses and the query statement itself, several first IP addresses corresponding to the query statement can be obtained. Based on these first IP addresses, the first enhanced target asset can be obtained by scanning all ports. Compared to the impossibility of performing a full network scan on all ports of all IP addresses to obtain the latest assets, performing a full port scan on several IP addresses is equivalent to performing a full port scan on a small number of assets, which is feasible. Moreover, the assets obtained by scanning all ports can cover all assets of several IPs in the public network, reducing the probability of failure to obtain the first enhanced target asset and achieving successful asset acquisition.
[0070] 2. Since the assets corresponding to a domain name can be directly retrieved from the database when the query is a domain name, if the assets corresponding to the domain name cannot be found in the database, a new query needs to be performed across the entire network. However, to access a server on the Internet, it is ultimately necessary to use an IP address. Therefore, based on the query, several second IP addresses corresponding to the query can be obtained through reverse DNS lookup. Based on these second IP addresses, the second enhanced target asset can be obtained by scanning all ports, and either the first or second enhanced target asset can be used as the first target asset. Thus, providing a query method for domain names can improve the search engine's adaptability to various query formats. Using all ports, rather than just important ports, to scan the entire network to obtain the second enhanced target asset can increase the probability of obtaining it.
[0071] 3. After clustering to obtain IP segments commonly corresponding to several IP addresses, to avoid repeatedly acquiring assets corresponding to each of these IP addresses, several other IP addresses can be obtained from the IP segments. A full port scan of these other IP addresses is then performed to obtain the second target asset. The first target asset and the second target asset are then used as the final target asset. After scanning the full ports of several IP addresses to obtain the first target asset, further scanning the full ports of several other IP addresses within the same IP segment to obtain the second target asset can increase the probability that the search engine successfully retrieves the asset corresponding to the query statement. Attached Figure Description
[0072] Figure 1 This is a flowchart illustrating an asset acquisition method provided in an embodiment of this application.
[0073] Figure 2 This is a schematic diagram of an asset acquisition device provided in an embodiment of this application.
[0074] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this application. Detailed Implementation
[0075] The present application will be further described in detail below with reference to the accompanying drawings.
[0076] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.
[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0078] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.
[0079] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.
[0080] This application provides an asset acquisition method executed by an electronic device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application does not impose any limitations on this. Figure 1As shown, the method includes steps S101, S102, S103, and S104, wherein:
[0081] Step S101: Receive the query statement and, based on the query statement, determine whether the asset corresponding to the query statement exists in the database through matching. The database includes at least the assets obtained by scanning multiple important ports.
[0082] Specifically, when a search engine receives a query from a user, it uses keyword matching based on the characters contained in the query to determine whether the corresponding asset exists in the database. If it does not exist, it means that the asset related to the query cannot be obtained from the entire network of assets by scanning multiple important ports. If it does exist, it means that the asset related to the query can be obtained from the entire network by scanning multiple important ports, without the need for a deeper query method. A deeper query method means obtaining the asset directly from the entire network by scanning all ports, and a deeper query method is also known as deep scanning.
[0083] Important ports can include common ports across the entire network, common ports found in search engines, and ports that have been found to have port errors in search engines.
[0084] Step S102: If it does not exist, determine whether the query statement is a domain name.
[0085] Specifically, if the asset corresponding to the query statement does not exist in the database, a pop-up message indicating that the asset could not be obtained will be displayed to the user. At the same time, a scanning command will be issued on the backend. This scanning command can be used to conduct a deep scan across the entire network to obtain the asset corresponding to the query statement.
[0086] It is understandable that query statements can include both domain names and keywords. When the query statement is a keyword, the webpage content corresponding to the keyword cannot be obtained directly. However, when the query statement is a domain name, several subdomains containing valid assets can be obtained through domain name collection, and the webpage content can be obtained directly from the subdomains. The valid assets are the assets corresponding to the query statement. Therefore, it is necessary to determine the form of the query statement in advance.
[0087] Step S103: If not, then according to the preset correspondence between query statement and IP address, obtain several first IP addresses corresponding to the query statement.
[0088] The mapping between query statements and IP addresses can include the mapping between common query statements and IP addresses and the mapping between personalized query statements and IP addresses. The mapping between common query statements and IP addresses can be obtained from historical data of query statements and corresponding IP addresses in search engines. The mapping between personalized query statements and IP addresses can be the mapping between several query statements entered by the user and IP addresses.
[0089] Understandably, keywords cannot directly yield the corresponding webpage content. Generally, one approach is to match keywords with several IP addresses, and then, based on each IP address, identify the webpage content corresponding to each IP address from the entire network assets obtained from multiple important ports. All webpage content corresponding to all IP addresses is then used as the webpage content corresponding to the keyword. However, since multiple important ports cannot provide assets corresponding to the query statement, it is preferable to select several IP addresses to match keywords, and then obtain assets from each IP as the content of this solution. Using IP addresses as a medium to help the query statement obtain assets ensures the stability of the search engine query process.
[0090] Step S104: Based on several first IP addresses, obtain the first enhanced target asset by scanning all ports.
[0091] Specifically, for each first IP address, under each port, domain name collection is performed on all domain names under the HTTP / HTTPS protocol to obtain several subdomains with existing assets; based on all subdomains with existing assets, the web page content corresponding to each subdomain is obtained to obtain the first enhanced target asset.
[0092] Understandably, compared to related technologies that scan common domains under important ports to obtain the latest assets across the entire network, this solution collects all domains under all ports for all first IP addresses. The full port, all domains, and domain collection can all increase the probability of successfully obtaining the first enhanced target asset.
[0093] In this embodiment, when a query statement that is not a domain name cannot be matched with a corresponding asset in the database, it is determined whether the query statement is a domain name. If it is not a domain name, it means that the query statement needs to indirectly obtain the target asset through an IP address. Specifically, based on the preset correspondence between query statements and IP addresses and the query statement, several first IP addresses corresponding to the query statement can be obtained. Based on these several first IP addresses, the first enhanced target asset is obtained by scanning all ports. Compared to the impossibility of performing a full network scan on all ports of all IP addresses to obtain the latest assets, performing a full port scan on several IP addresses is equivalent to performing a full port scan on a small number of assets, which is feasible. Moreover, the assets obtained by scanning all ports can cover all assets of several IPs in the public network, reducing the probability of failure to obtain the first enhanced target asset and achieving successful asset acquisition.
[0094] One possible implementation of this application embodiment, step S102, after determining whether the query statement is a domain name, may further include steps S1021 (not shown in the figure), S1022 (not shown in the figure), S1023 (not shown in the figure), and S1024 (not shown in the figure), wherein:
[0095] Step S1021: If yes, then determine whether there is an asset corresponding to the query statement based on the assets and query statements corresponding to each of the multiple domain names.
[0096] Specifically, if the query is a domain name, then for each domain name in the query, the database is searched to see if any corresponding assets exist. The database can include assets obtained from at least multiple important ports, and all assets in the database can be directly obtained through domain name matching. Depending on the user's needs, the number of domain names or keywords in the query can be one or more. When the number is one, the query targets only that domain name or keyword. When the number is multiple, the search engine can provide different query methods under different logics for multiple keywords or multiple domain names. These different query methods can include AND and OR.
[0097] Understandably, if the domain name in the query statement is not found, it means that the assets related to the domain name in the query statement cannot be obtained from the network-wide assets obtained by scanning multiple important ports. If the domain name is found, it means that the assets related to the query statement can be obtained from the network-wide assets obtained by scanning multiple important ports without the need to use a deeper query method.
[0098] Step S1022: If it does not exist, then according to the query statement, obtain several second IP addresses corresponding to the query statement through reverse DNS lookup.
[0099] Specifically, if no asset corresponds to any of the domain names in the query statement, then for each domain name, the domain name can be used as a resolution request. This request is sent from the user's client to the client's DNS server. The DNS server first sends the domain name resolution request from the root DNS server, which then returns the IP address of the .com / cn domain to the client's local DNS server. The local DNS server then accesses the .com / cn domain server, which returns the IP address of the domain where the search engine resides to the local DNS server. The local DNS server then accesses the domain server where the search engine resides to query the IP address of the www domain server. Through this process, reverse domain name resolution is completed, obtaining several second IP addresses corresponding to the query statement.
[0100] Step S1023: Based on several second IP addresses, obtain the second enhanced target asset by scanning all ports.
[0101] Specifically, for each second IP address, under each port, domain name collection is performed on all domain names under the HTTP / HTTPS protocol to obtain several subdomains with existing assets; based on all subdomains with existing assets, the web page content corresponding to each subdomain is obtained to obtain the second enhanced target asset.
[0102] Step S1024: Select either the first enhanced target asset or the second enhanced target asset as the first target asset.
[0103] In this embodiment, since the assets corresponding to a domain name can be directly queried in the database when the query is a domain name, if the assets corresponding to the domain name cannot be found in the database, a new query needs to be performed across the entire network. However, to access a server on the Internet, it is ultimately necessary to use an IP address. Therefore, based on the query, several second IP addresses corresponding to the query can be obtained through reverse DNS lookup of the domain name. Based on these second IP addresses, the second enhanced target asset can be obtained by scanning all ports, and either the first or second enhanced target asset can be used as the first target asset. Thus, providing a query method for domain names can improve the search engine's adaptability to various query formats. Using all ports, rather than just important ports, to scan the entire network to obtain the second enhanced target asset can increase the probability of obtaining it.
[0104] One possible implementation of this application embodiment, after step S1024, may further include steps S10241 (not shown in the figure), S10242 (not shown in the figure), S10243 (not shown in the figure), and S10244 (not shown in the figure), wherein:
[0105] Step S10241: Based on several IP addresses, obtain the IP ranges that are common to several IP addresses through clustering.
[0106] Specifically, the inventors discovered that several IP addresses share commonalities, namely, the IP range and the corresponding query statement are the same. It can be assumed that the assets corresponding to the IP addresses included in the IP range are all target assets. Therefore, for the same query statement, clustering can be used to find the IP range that several IP addresses commonly correspond to.
[0107] Step S10242: Based on the IP segment and several IP addresses, obtain several other IP addresses, where other IP addresses refer to IP addresses in the IP segment other than the several IP addresses.
[0108] Specifically, since the assets corresponding to several IP addresses have already been acquired, in order to avoid duplicate acquisition, after obtaining the IP ranges that are common to several IP addresses, several other IP addresses that have not yet acquired assets can be filtered out from the IP ranges. Since the difference between the total number of several IP addresses and the total number of IP addresses in the IP range is not certain, there may be one or more other IP addresses.
[0109] Step S10243: Obtain the second target asset by performing a full port scan on several other IP addresses.
[0110] Specifically, for each other IP address, under each port, domain name collection is performed on all domain names under the HTTP / HTTPS protocol to obtain several subdomains with existing assets; based on all subdomains with existing assets, the web page content corresponding to each subdomain is obtained to obtain the second target asset.
[0111] Step S10244: Take the first target asset and the second target asset as the final target asset.
[0112] The process of scanning the entire network through all ports for a number of IP addresses to obtain the first target asset can be considered the first enhancement for obtaining the final target asset. Similarly, the process of scanning the entire network through all ports for a number of other IP addresses to obtain the second target asset can be considered the second enhancement for obtaining the final target asset. These two enhancements significantly increase the probability of obtaining the final target asset.
[0113] In this embodiment, after clustering to obtain IP segments commonly corresponding to several IP addresses, to avoid repeatedly obtaining assets corresponding to each of the several IP addresses, several other IP addresses can be obtained from the IP segments. A full port scan is then performed on these other IP addresses to obtain the second target asset. The first and second target assets are then used as the final target assets. After scanning all ports to obtain the first target asset for several IP addresses, further scanning all ports for several other IP addresses within the IP segments containing the several IP addresses to obtain the second target asset can increase the probability that the search engine successfully obtains the asset corresponding to the query statement.
[0114] One possible implementation of this application embodiment may include, before step S104, determining a plurality of first valid IP addresses from a plurality of first IP addresses, wherein the first valid IP address represents a first IP address where asset information exists.
[0115] Specifically, for each first IP address, determine whether any port is open to the first IP address; if not, it means that the current IP address is an invalid IP address, the success rate of obtaining assets is 0, and there is no need to try to obtain assets; if it is, it means that the current IP address is a valid IP address, the success rate of obtaining assets is not 0, and the current first IP address is determined to be the first valid IP address.
[0116] It is understandable that among all the first IP addresses, there may be some that cannot obtain assets. When the search engine invests resources in the first IP address that cannot obtain assets to acquire assets, it will cause a waste of the search engine's resources. In particular, when the search engine has limited available resources, the first IP address that cannot obtain assets may occupy resources and cause the search engine's response speed to decrease.
[0117] Accordingly, step S104, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, may specifically include:
[0118] Based on several first valid IP addresses, the first enhanced target asset is obtained by scanning all ports.
[0119] In this embodiment, after determining a number of first valid IP addresses from all first IP addresses, the first enhanced target asset is obtained by scanning all ports based on these first valid IP addresses. This avoids wasting resources on first IP addresses from which assets cannot be obtained, thus preventing resource waste on the search engine when acquiring assets. Furthermore, when the search engine has limited idle resources, preventing first IP addresses from consuming resources can improve the search engine's response speed.
[0120] Furthermore, in one possible implementation of this application embodiment, before step S1023, it may further include: determining a plurality of second valid IP addresses from a plurality of second IP addresses, wherein the second valid IP address represents a second IP address where asset information exists.
[0121] Accordingly, step S1023, when executing the process of obtaining the second enhanced target asset by scanning all ports based on several second IP addresses, may specifically include:
[0122] Based on several second valid IP addresses, the second enhanced target asset is obtained by scanning all ports.
[0123] In this embodiment, resources are avoided from being allocated to second IP addresses that cannot acquire assets, which would otherwise waste the search engine's resources when acquiring assets. Furthermore, when the search engine has limited free resources, avoiding the use of second IP addresses that cannot acquire assets can also improve the search engine's response speed.
[0124] One possible implementation of this application embodiment, when performing the process of obtaining the first enhanced target asset by scanning all ports based on several first valid IP addresses, may specifically include steps SA1 to SA4 (not shown in the figure), wherein:
[0125] Step SA1: Based on several first valid IP addresses, identify several first valid ports corresponding to each first valid IP address from all ports, wherein each valid port represents a port open to each first valid IP address.
[0126] Specifically, for each first valid IP address, each port is checked to determine whether return information can be obtained from the port. The return information may include a domain name. If it can be obtained, it means that the current port is open to the current first valid IP address. If it cannot be obtained, it means that the current port is not open to the current first valid IP address.
[0127] It is understandable that each IP address may correspond to different valid ports, and allocating resources to ports that cannot acquire assets for the same IP address would result in a waste of search engine resources when acquiring assets. Therefore, it is advisable to first determine several first valid IP addresses corresponding to each first IP address.
[0128] Step SA2: Obtain multiple common domain names, and for each first valid IP address, collect domain names from multiple common domain names under each first valid port to obtain several first valid domain names corresponding to the first valid IP address. Among them, the first valid domain name represents the subdomain of the common domain name containing asset information.
[0129] It is understandable that when acquiring assets from the entire network, the domain names that can be retrieved from different ports are not the same. Therefore, the effective domain names can be determined in advance to save search engine resources.
[0130] Step SA3: Under each valid port, for each first valid IP address, based on the several first valid domain names corresponding to the first valid IP address, obtain the first unit target asset corresponding to the first valid IP address, where each first unit target asset represents the total assets obtained from all first valid domain names under each valid IP address.
[0131] Specifically, the process of acquiring the first unit of the target asset can be referred to in step S104.
[0132] Step SA4: Use all first-unit target assets as the first enhancement target assets.
[0133] In this embodiment of the application, by determining the valid domain names among common domain names, resources can be avoided for domain names from which assets cannot be acquired, thus avoiding a waste of search engine resources when acquiring assets. Furthermore, when the search engine has limited idle resources, avoiding the use of second IP addresses from which assets cannot be acquired can also improve the search engine's response speed.
[0134] In one possible implementation of this application embodiment, after determining whether an asset corresponding to the query statement exists, step S101 may further include steps SB1 (not shown in the figure) and SB2 (not shown in the figure), wherein:
[0135] Step SB1: Obtain the current resource ownership status of the search engine.
[0136] Specifically, the system obtains the number of first-stage processes of the search engine and determines whether this number is less than a first threshold, where the first-stage process count is the number of currently running processes. If it is less, the search engine's current resource allocation status is determined to be idle. If it is not less, the system determines whether the first-stage process count is less than a second threshold. If it is less, the system determines whether the current resource allocation status is idle and sends information indicating that idle resources are about to be exhausted to the backend administrators. If it is not less, the system determines whether the search engine's current resource allocation status is busy, obtains the number of second-stage processes of the search engine, and determines whether this number is less than a third threshold, where the second-stage process count is the total number of processes in the search engine's buffer queue at the current moment. If it is not less, a search engine crash prediction alert is sent to the administrators; if it is less, no search engine crash prediction alert is sent to the administrators. The first, second, and third thresholds can be set based on actual conditions or determined based on historical data of the search engine, and their values increase sequentially.
[0137] Step SB2: Determine the port scanning method based on the current resource occupancy status. When the current resource occupancy status is idle, execute the step of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses.
[0138] Accordingly, step S104, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, may specifically include:
[0139] When the current resource occupancy status is busy, the first enhanced target asset is obtained by scanning all ports except for some ports with a 0% utilization rate, based on several first IP addresses.
[0140] In this embodiment of the application, by obtaining the current resource occupancy status of the search engine, the query method for the first enhanced target asset is determined to be a full port query, or by scanning all ports except for some ports with a utilization rate of 0. This can adapt to the busy or idle status of the search engine, improve the flexibility of the search engine query process, and thus improve the stability of the search engine operation.
[0141] In one possible implementation of this application embodiment, before receiving the query statement, step S101 may further include steps SC1 (not shown in the figure) and SC2 (not shown in the figure), wherein:
[0142] Step SC1: Obtain all IP addresses.
[0143] Step SC2: Based on all IP addresses, periodically scan multiple important ports to obtain the latest common assets corresponding to each IP address. The latest common assets represent the sum of the latest assets that each IP address can obtain through the paths of multiple important ports.
[0144] Understandably, performing a full port scan on all IP addresses might waste too many resources trying to acquire assets with low query rates, and it's not very feasible. Therefore, it's better to scan all IP addresses for common ports among several important ports and ports that are prone to crashing the search engine. This allows for the early acquisition of common assets and assets that are likely to cause the search engine to crash during the query process. This can further improve the stability of the search engine query process while avoiding excessive waste of resources.
[0145] For periodic scans, the real-time nature of the database can be guaranteed.
[0146] Accordingly, step S104, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, may specifically include:
[0147] The scanning process for acquiring target assets is given the highest priority, and based on the highest priority, the latest common assets corresponding to several first IP addresses are selected as the first enhanced target assets.
[0148] Under the premise of periodic scanning, in order to ensure that the first target asset can be acquired quickly, the task priority of acquiring the target asset can be raised to the highest priority.
[0149] In this embodiment, by scanning all IP addresses for common ports and ports that are prone to causing search engine crashes among multiple important ports, common assets and assets whose query process is prone to causing search engine crashes can be obtained in advance. Under the premise of ensuring feasibility and avoiding excessive waste of resources, the stability of the search engine query process can be further improved. Prioritizing the task of obtaining target assets to the highest priority can ensure the speed of obtaining target assets.
[0150] The above embodiments describe an asset acquisition method from the perspective of process flow. The following embodiments describe an asset acquisition device from the perspective of virtual module or virtual unit. For details, please refer to the following embodiments.
[0151] This application provides an asset acquisition device, such as... Figure 2 As shown, the asset acquisition device may specifically include:
[0152] The query statement processing module 201 is used to receive a query statement and, based on the query statement, determine whether there is an asset in the database corresponding to the query statement through matching. When there is no asset corresponding to the query statement, the query statement judgment module is triggered.
[0153] The query statement judgment module 202 is used to determine whether the query statement is a domain name. If it is not a domain name, the first IP address acquisition module is triggered.
[0154] The first IP address acquisition module 203 is used to obtain several first IP addresses corresponding to the query statement according to the preset correspondence between the query statement and the IP address.
[0155] The first target asset enhancement module 204 is used to obtain the first enhanced target asset by scanning all ports based on the aforementioned first IP addresses.
[0156] In this embodiment, when a query statement that is not a domain name cannot be matched with a corresponding asset in the database, it is determined whether the query statement is a domain name. If it is not a domain name, it means that the query statement needs to indirectly obtain the target asset through an IP address. Specifically, based on the preset correspondence between query statements and IP addresses and the query statement, several first IP addresses corresponding to the query statement can be obtained. Based on these several first IP addresses, the first enhanced target asset is obtained by scanning all ports. Compared to the impossibility of performing a full network scan on all ports of all IP addresses to obtain the latest assets, performing a full port scan on several IP addresses is equivalent to performing a full port scan on a small number of assets, which is feasible. Moreover, the assets obtained by scanning all ports can cover all assets of several IPs in the public network, reducing the probability of failure to obtain the first enhanced target asset and achieving successful asset acquisition.
[0157] One possible implementation of this application embodiment, the asset acquisition device, further includes:
[0158] The first target asset acquisition module is used for:
[0159] If so, then determine whether there is an asset corresponding to the query statement based on the assets and query statements corresponding to each of the multiple domain names;
[0160] If it does not exist, then based on the query statement, several second IP addresses corresponding to the query statement are obtained through reverse DNS lookup.
[0161] Based on several second IP addresses, the second enhanced target asset is obtained by scanning all ports;
[0162] Use either the first enhanced target asset or the second enhanced target asset as the first target asset.
[0163] One possible implementation of this application embodiment, the asset acquisition device, further includes:
[0164] The final target asset acquisition module is used for:
[0165] Based on a number of IP addresses, clustering is used to obtain the IP ranges that the IP addresses commonly correspond to.
[0166] Based on an IP range and several IP addresses, several other IP addresses are obtained, where other IP addresses refer to IP addresses in the IP range other than the specified IP addresses.
[0167] The second target asset was obtained by performing a full port scan on several other IP addresses;
[0168] The first and second target assets will be used as the final target assets.
[0169] One possible implementation of this application embodiment, the asset acquisition device, further includes:
[0170] The valid IP address determination module is used for:
[0171] Several first valid IP addresses are determined from several first IP addresses, wherein the first valid IP address represents the first IP address where asset information exists.
[0172] Correspondingly, the first target asset enhancement module 204, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, is used for:
[0173] Based on several first valid IP addresses, the first enhanced target asset is obtained by scanning all ports.
[0174] In one possible implementation of this application, the effective IP address determination module, when performing the operation of obtaining the first enhanced target asset by scanning all ports based on several first effective IP addresses, is used to:
[0175] Based on several first valid IP addresses, identify several first valid ports corresponding to each first valid IP address from all ports, wherein each valid port represents a port open to each first valid IP address.
[0176] Obtain multiple common domain names, and for each first valid IP address, collect domain names from multiple common domain names under each first valid port to obtain several first valid domain names corresponding to the first valid IP address. Among them, the first valid domain name represents the subdomain of the common domain name containing asset information.
[0177] Under each valid port, for each first valid IP address, based on several first valid domain names corresponding to the first valid IP address, the first unit target asset corresponding to the first valid IP address is obtained, wherein each first unit target asset represents the total assets obtained from all first valid domain names under each valid IP address;
[0178] All first-unit target assets are designated as the first enhanced target assets.
[0179] One possible implementation of this application embodiment, the asset acquisition device, further includes:
[0180] The current resource occupancy status reference module is used for:
[0181] Get the current resource ownership status of the search engine;
[0182] Based on the current resource occupancy status, the port scanning method is determined. When the current resource occupancy status is idle, the step of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses is executed.
[0183] Correspondingly, the first target asset enhancement module 204, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, is used for:
[0184] When the current resource occupancy status is busy, the first target asset for enhancement is obtained by scanning all ports except for some ports with a utilization rate of 0, based on several IP addresses.
[0185] One possible implementation of this application embodiment, the asset acquisition device, further includes:
[0186] The highest priority is assigned to the module for:
[0187] Get all IP addresses;
[0188] Based on all IP addresses, the latest common assets corresponding to each IP address are obtained by periodically scanning multiple important ports. The latest common assets represent the sum of the latest assets that each IP address can obtain through the paths of multiple important ports.
[0189] Correspondingly, the first target asset enhancement module 204, when executing the process of obtaining the first enhanced target asset by scanning all ports based on several first IP addresses, is used for:
[0190] The scanning process for acquiring target assets is given the highest priority, and based on the highest priority, the latest common assets corresponding to several first IP addresses are selected as the first enhanced target assets.
[0191] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the asset acquisition device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0192] This application provides an electronic device, such as... Figure 3 As shown, Figure 3 The illustrated electronic device includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.
[0193] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0194] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0195] The memory 303 may be a ROM (Read-Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM (Electrically Erasable Programmable Read-Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0196] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.
[0197] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0198] This application provides a computer-readable storage medium storing a computer program. When the program is run on a computer, it enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with related technologies, in this application embodiment, when a query statement that is not a domain name cannot be matched with a corresponding asset in the database, it is determined whether the query statement is a domain name. If it is not a domain name, it means that the query statement needs to indirectly obtain the target asset through an IP address. Specifically, based on a preset correspondence between query statements and IP addresses and the query statement, several first IP addresses corresponding to the query statement can be obtained. Based on these several first IP addresses, the first enhanced target asset is obtained by scanning all ports. Compared to the impossibility of performing a full network scan on all ports of all IP addresses to obtain the latest assets, performing a full port scan on several IP addresses is equivalent to performing a full port scan on a small number of assets, which is feasible. Moreover, the assets obtained by scanning all ports can cover all assets of several IPs in the public network, reducing the probability of failure to obtain the first enhanced target asset and achieving successful asset acquisition.
[0199] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0200] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An asset acquisition method characterized by comprising: include: Upon receiving a query statement, the system determines, through matching, whether the corresponding asset exists in the database. The database includes assets obtained by scanning multiple important ports. If it does not exist, then determine whether the query statement is a domain name; If not, then based on the preset correspondence between query statements and IP addresses, several first IP addresses corresponding to the query statement are obtained; Based on the aforementioned first IP addresses, the first enhanced target asset is obtained by scanning all ports; The database also includes assets corresponding to at least multiple domain names, and after determining whether the query statement is a domain name, it also includes: If so, then based on the assets corresponding to each of the multiple domain names and the query statement, determine whether there are assets corresponding to the query statement; If it does not exist, then based on the query statement, several second IP addresses corresponding to the query statement are obtained through reverse DNS lookup. From a plurality of second IP addresses, determine a plurality of second valid IP addresses, wherein the second valid IP addresses represent second IP addresses that contain asset information; Based on several second IP addresses, the second enhanced target asset is obtained by scanning all ports; The first enhanced target asset or the second enhanced target asset is used as the first target asset; The IP address includes a first IP address, and the first target asset includes a first enhanced target asset; or, the IP address includes a second IP address, and the first target asset includes a second enhanced target asset. After specifying either the first enhanced target asset or the second enhanced target asset as the first target asset, the method further includes: Based on several IP addresses, clustering is used to obtain the IP ranges that are commonly associated with these IP addresses; Based on the IP segment and the plurality of IP addresses, a plurality of other IP addresses are obtained, wherein the other IP addresses refer to IP addresses in the IP segment other than the plurality of IP addresses; The second target asset was obtained by performing a full port scan on the other IP addresses. The first target asset and the second target asset are taken as the final target assets; Before obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses, the method further includes: From the plurality of first IP addresses, a plurality of first valid IP addresses are determined, wherein the first valid IP address represents the first IP address where asset information exists; Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes: Based on the aforementioned first valid IP addresses, the first enhanced target asset is obtained by scanning all ports; The step of obtaining the first enhanced target asset by scanning all ports based on the plurality of first valid IP addresses includes: Based on the aforementioned first valid IP addresses, identify several first valid ports corresponding to each first valid IP address from all ports, wherein each valid port represents a port open to each first valid IP address. Multiple common domain names are obtained, and for each first valid IP address, under each first valid port, the domain names of the multiple common domain names are collected to obtain several first valid domain names corresponding to the first valid IP address. The first valid domain name represents the subdomain of the common domain name containing asset information. Under each valid port, for each first valid IP address, based on several first valid domain names corresponding to the first valid IP address, the first unit target asset corresponding to the first valid IP address is obtained, wherein each first unit target asset represents the total assets obtained from all first valid domain names under each valid IP address; All first-unit target assets are designated as the first enhanced target assets.
2. The asset acquisition method according to claim 1, characterized in that, After determining whether the asset corresponding to the query statement exists, the process also includes: Get the current resource ownership status of the search engine; Based on the current resource occupancy status, the port scanning method is determined. When the current resource occupancy status is idle, the step of obtaining the first enhanced target asset by scanning all ports based on the several first IP addresses is executed. Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes: When the current resource occupancy status is busy, the first enhanced target asset is obtained by scanning all ports except for some ports with a utilization rate of 0, based on the several first IP addresses.
3. The asset acquisition method of claim 1, wherein, Before receiving the query statement, the following is also included: Get all IP addresses; Based on all the IP addresses, the latest common assets corresponding to each IP address are obtained by periodically scanning multiple important ports. The latest common assets represent the sum of the latest assets that each IP address can obtain through the paths of multiple important ports. Accordingly, obtaining the first enhanced target asset by scanning all ports based on the plurality of first IP addresses includes: The scanning process for acquiring target assets is given the highest priority, and based on the highest priority, the latest common assets corresponding to each of the several first IP addresses are taken as the first enhanced target assets.
4. An asset acquisition apparatus, characterized by comprising: Performing the asset acquisition method as described in any one of claims 1-3 includes, The query statement processing module is used to receive a query statement and, based on the query statement, determine whether the asset corresponding to the query statement exists in the database through matching. If the asset corresponding to the query statement does not exist, the query statement judgment module is triggered. The query statement judgment module is used to determine whether the query statement is a domain name. If it is not a domain name, the first IP address acquisition module is triggered. The first IP address acquisition module is used to obtain several first IP addresses corresponding to the query statement based on the preset correspondence between the query statement and the IP address. The first target asset enhancement module is used to obtain the first enhanced target asset by scanning all ports based on the aforementioned first IP addresses.
5. An electronic device, characterized in that, include: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, said at least one application being configured to: perform the asset acquisition method according to any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the asset acquisition method according to any one of claims 1 to 3.