Network resource data detection method and device, computer equipment and storage medium

Abstract

The invention relates to a network resource data detection method and device, computer equipment and a storage medium. The method comprises the following steps: generating a plurality of address access requests based on an address monitoring list; performing proxy on the plurality of address access requests according to a preset proxy service to obtain a plurality of proxy service links; requesting an address corresponding to each address access request according to the plurality of proxy service links to obtain request response data; and performing page crawling on the request response data according to a preset crawler process to obtain page information data and a page screenshot, and transmitting the page information data and the page screenshot to a content auditing platform. By means of the method, through proxy of the address access request, the nodes can be obtained from the recorded node pool, namely the preset node pool, and the link is constructed, so that the stable link is constructed, the stability of crawling the content of the webpage corresponding to the address is improved, and the scanning success rate is guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of network data acquisition, and in particular to a network resource data detection method, device, computer equipment and storage medium. Background Art

[0002] When conducting web content testing, Python programming language and Selenium components have become widely used tools. They can simulate user behavior and automatically operate web pages to obtain the content in the web pages. However, the special nature of the Dark Web makes this task complicated. Since the Dark Web is not accessed through regular search engines, users cannot directly open the page like visiting ordinary websites. Accessing the Dark Web requires establishing a secure connection through the Tor network or similar anonymous networks through multiple relay nodes (such as bridges, entry nodes, intermediate nodes, and exit nodes). Although this access method can protect the anonymity of users, it also poses challenges to access speed and access restrictions, especially when frequent and batch testing of Dark Web content is required. How to efficiently and stably test the content of the Dark Web has become a current technical problem.

[0003] In the related technology, in the process of dark web content dialing, common technical solutions usually rely on the construction of relay links in the Tor network. In this process, Selenium is used in combination with the Python programming language to simulate browser operations and capture page data. The key to this method is to break through conventional network restrictions through the Tor proxy, bypass the blocking of IP, and access it by configuring appropriate Tor bridges or nodes. Selenium simulates user behavior, performs operations such as page loading, content capture, and screenshots to obtain the required web page content.

[0004] However, in the related technologies, Tor network and Selenium combined with Python for dark web dialing test provide certain solutions, but there are also obvious shortcomings. First, Tor's relay link leads to slow access speed, affecting the efficiency of frequent access; second, the change of nodes may lead to unstable access, affecting the continuity and accuracy of the task. In addition, some dark web sites will monitor IP and block abnormal access, which increases the difficulty of dialing. Finally, with the complexity of dark web content, the single Selenium combined with Tor method is difficult to efficiently support large-scale automated crawling, and the maintenance cost is high. Summary of the invention

[0005] Based on this, it is necessary to provide a network resource data detection method, device, computer equipment, computer-readable storage medium and computer program product that can ensure link connectivity, improve the stability of crawled page content and maintain the scanning success rate in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for detecting network resource data. The method comprises:

[0007] generating a plurality of address access requests based on the address monitoring list;

[0008] Proxify multiple address access requests according to the preset proxy service to obtain multiple proxy service links;

[0009] Requesting the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0010] The request response data is crawled according to the preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

[0011] In one embodiment, proxying multiple address access requests according to a preset proxy service to obtain multiple proxy service links includes:

[0012] Perform proxy screening on each address access request according to the preset node proxy and the preset node pool, and obtain the filtered node and the first port;

[0013] Proxify the filtered node based on the preset forward proxy to obtain the forward proxy node and the second port;

[0014] A plurality of proxy service links are generated based on the filtered node, the first port, the forward proxy node, and the second port of each address access request.

[0015] In one embodiment, performing proxy screening on each address access request according to a preset node proxy and a preset node pool to obtain a filtered node and a first port includes:

[0016] Perform node detection on the egress pool according to preset configuration rules to obtain available node queues and unavailable node queues;

[0017] In the case of proxying each address access request, node screening is performed according to the preset node proxy, entry pool, intermediate pool and available node queue to obtain the screened entry node, screened intermediate node and screened available exit node;

[0018] When multiple post-screening entry nodes are obtained, the entry ports of all post-screening entry nodes are set to the first port.

[0019] In one embodiment, the post-screening node includes a post-screening entry node; proxying the post-screening node based on a preset forward proxy to obtain a forward proxy node and a second port includes:

[0020] According to the preset forward proxy, the filtered entry node and the first port are configured to obtain a forward proxy node;

[0021] Set the inbound port of the forward proxy node to the second port.

[0022] In one embodiment, before performing node detection on the egress pool according to a preset configuration rule to obtain an available node queue and an unavailable node queue, the method further includes:

[0023] Generate monitoring access requests;

[0024] Proxify each monitoring access request according to the preset node proxy and the preset node pool to obtain the monitoring node;

[0025] Determining a monitoring access link based on a monitoring node;

[0026] Perform address access according to the monitoring access link and the monitoring access request to obtain monitoring response data;

[0027] According to the preset node division algorithm and monitoring response data, the node detection is performed on the export pool to obtain the available node queue and the unavailable node queue.

[0028] In one embodiment, node detection is performed on the egress pool according to a preset node partitioning algorithm and monitoring response data to obtain an available node queue and an unavailable node queue, including:

[0029] When each egress node is detected, an egress weight set is determined based on multiple egress response times and preset weight mapping relationships;

[0030] Performing data processing on multiple export response times to obtain an export response set;

[0031] Perform weighted calculation according to the export weight set, the export response set and the preset node partitioning algorithm to obtain a response weighted value;

[0032] When the response weighted value corresponding to each egress node is obtained, the egress pool is divided based on the preset response threshold and the response weighted value corresponding to the egress node to obtain an available node queue and an unavailable node queue.

[0033] In one embodiment, the method further comprises:

[0034] Perform response detection on the egress pool according to the preset timed monitoring process to obtain the node response detection result;

[0035] The available node queue and the unavailable node queue are divided into queues according to the node response results to obtain the available queue after detection and the unavailable queue after detection.

[0036] In a second aspect, the present application also provides a network resource data detection device. The device comprises:

[0037] A request generation module, used to generate multiple address access requests based on the address monitoring list and the preset crawler process;

[0038] A proxy service module, used to proxy multiple address access requests according to a preset proxy service to obtain multiple proxy service links;

[0039] A data acquisition module, used to request the address corresponding to each address access request according to multiple proxy service links, and obtain request response data;

[0040] The page crawling module is used to crawl the request response data according to the preset crawler process, obtain page information data and page screenshots, and transmit the page information data and the said page screenshots to the content review platform.

[0041] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0042] generating a plurality of address access requests based on the address monitoring list;

[0043] Proxify multiple address access requests according to the preset proxy service to obtain multiple proxy service links;

[0044] Requesting the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0045] The request response data is crawled according to the preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0047] generating a plurality of address access requests based on the address monitoring list;

[0048] Proxify multiple address access requests according to the preset proxy service to obtain multiple proxy service links;

[0049] Requesting the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0050] The request response data is crawled according to the preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

[0051] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] generating a plurality of address access requests based on the address monitoring list;

[0053] Proxify multiple address access requests according to the preset proxy service to obtain multiple proxy service links;

[0054] Requesting the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0055] The request response data is crawled according to the preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

[0056] The above-mentioned network resource data detection method, device, computer equipment, storage medium and computer program product, in the process of content dialing test on the web page address in the address monitoring list, generates an address access request for the address that needs to be accessed, and then starts a multi-node preset proxy service to proxy multiple address access requests, and obtains a link established by the entry node, intermediate node and exit node obtained from the node pool. Each address access request is output through the corresponding link, and then a request response result is obtained, and the request response result includes a response log of the exit node. At the same time, a preset crawler process is used to crawl the page of the access address, and the crawled page information and screenshots are transmitted to the content review platform; by proxying the address access request, nodes can be obtained from the recorded node pool, that is, the preset node pool, and links can be constructed to achieve the construction of a stable link, thereby improving the stability of content crawling of the web page corresponding to the address, that is, ensuring the scanning success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a flow chart of a network resource data detection method in an embodiment;

[0058] Figure 2 A flowchart of the proxy configuration steps in one embodiment;

[0059] Figure 3 A structural block diagram of an agent configuration in one embodiment;

[0060] Figure 4 is a flow chart of an egress node detection step in one embodiment;

[0061] Figure 5 A schematic diagram of a specific process of a network resource data detection method in an embodiment;

[0062] Figure 6 is a structural block diagram of a network resource data detection device in an embodiment;

[0063] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] In one embodiment, Figure 1 As shown, a network resource data detection method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0066] Step 102: Generate multiple address access requests based on the address monitoring list.

[0067] Exemplarily, the operator records the network addresses corresponding to the web pages that need to be monitored in the monitoring address library. Then, when monitoring, the operator selects some network addresses from the monitoring address library and puts them into the address monitoring list. Then, a request, namely an address access request, is generated for each network address in the address monitoring list.

[0068] Step 104: proxy multiple address access requests according to a preset proxy service to obtain multiple proxy service links.

[0069] The preset proxy service includes a preset node proxy and a preset forward proxy.

[0070] Exemplarily, after receiving multiple address access requests, a preset node proxy (which may be, but is not limited to, an ip-proxy proxy service) is started, and nodes in the entry node, intermediate node, and exit node are obtained from the recorded node pool through the proxy service, a link is established, and the port of the link is exposed. A preset forward proxy (which may be, but is not limited to, an nginx forward proxy) is used to proxy the node of the above proxy service and expose the port. Through the above proxy, multiple proxy service links are then obtained.

[0071] Step 106: Request the address corresponding to each address access request according to the multiple proxy service links to obtain request response data.

[0072] Exemplarily, after the link is established, a request is made to the target network address through the link, and then a response fed back by the network address, namely, request response data, is obtained.

[0073] Step 108, crawl the request response data according to the preset crawler process, obtain page information data and page screenshots, and transmit the page information data and the page screenshots to the content review platform.

[0074] Exemplarily, after receiving a response to the target network address feedback, a pre-written crawler process, i.e., a preset crawler process, is used to crawl the page resources. In one embodiment, the crawler process is used and a browser driver (e.g., selenium.webdriver) and other components are imported to crawl the page resources to obtain page information data and page screenshots.

[0075] The page information data and page screenshots are then transmitted to the content review platform for machine and manual review.

[0076] In the above-mentioned network resource data detection method, in the process of content dialing test on the web page address in the address monitoring list, an address access request is generated for the address that needs to be accessed, and then a multi-node preset proxy service is started to proxy multiple address access requests, and a link established by the entry node, intermediate node and exit node obtained from the node pool is obtained. Each address access request is output through the corresponding link, and then a request response result is obtained, and the request response result includes a response log of the exit node. At the same time, a preset crawler process is used to crawl the page of the access address, and the crawled page information and the screenshot are transmitted to the content review platform; by proxying the address access request, nodes can be obtained from the recorded node pool, that is, the preset node pool, and links can be constructed to achieve the construction of a stable link, thereby improving the stability of content crawling of the web page corresponding to the address, that is, ensuring the scanning success rate.

[0077] In an exemplary embodiment, Figure 2 As shown, multiple address access requests are proxied according to the preset proxy service to obtain multiple proxy service links, including:

[0078] Step 202: Perform proxy screening on each address access request according to the preset node proxy and the preset node pool to obtain the screened node and the first port.

[0079] Exemplarily, all address access requests are input into the proxy service node, the ip-proxy proxy service is started, and then for each address access request, the entry node, intermediate node and exit node are screened from the preset node pool, and the entry node, intermediate node and exit node are used to build an access link, and the ports of multiple entry nodes, i.e., the first port, are exposed at the same time. For example, two ports are exposed and set to 9106 and 9107, and multiple proxy services monitor the exposed ports.

[0080] Specifically: the crawler process client is started, and the crawler process code is configured to use the ip-proxy proxy service.

[0081] Step 204: proxy the screened nodes based on the preset forward proxy to obtain the forward proxy node and the second port.

[0082] Among them, the preset forward proxy is nginx forward proxy.

[0083] For example, nginx is configured through the crawler process client so that nginx uniformly forwards request traffic to different nodes. A unified proxy address is set in the nginx configuration file, all backend proxy nodes, i.e., forward proxy nodes, are specified, and a listening port (e.g., 9998) is configured, i.e., the second port is used for the unified proxy.

[0084] Step 206: Generate multiple proxy service links based on the filtered node, the first port, the forward proxy node, and the second port of each address access request.

[0085] Exemplarily, after the proxy service is completed, a proxy service link is constructed using the second port, the forward proxy node, and the access link.

[0086] Nginx receives the address access request sent by the client through the second port, selects the appropriate backend node through the load balancing strategy, and then requests the target address through the access link to obtain the request response result.

[0087] In an exemplary embodiment, performing proxy screening on each address access request according to a preset node proxy and a preset node pool to obtain a filtered node and a first port includes:

[0088] Perform node detection on the exit pool according to preset configuration rules to obtain available node queues and unavailable node queues; perform node screening according to preset node proxy, entry pool and intermediate pool when proxying each address access request to obtain screened entry nodes, screened intermediate nodes and screened available exit nodes; in the case of obtaining multiple screened entry nodes, set the entry ports of all screened entry nodes to the first port.

[0089] The preset node pool includes an entry pool, an intermediate pool and an exit pool, and the exit pool includes an available node queue.

[0090] Exemplarily, response data corresponding to the responses stored in the request log are collected, and random weighted statistics are performed on each node in the egress pool to divide the nodes into an available node queue and an unavailable node queue.

[0091] All external address access requests are first input through the proxy service node to start the IP-Proxy proxy service. The proxy service will select suitable entry nodes, intermediate nodes and exit nodes from the available node queues in the preset entry pool, intermediate pool and exit pool according to the target address of each request, and build a complete access link. The specific node selection process is as follows: Figure 3 Specifically, the entry node is responsible for receiving and processing the initial request from the client and forwarding the request to the intermediate node. The intermediate node acts as a transit station for traffic and further forwards the request to the exit node. The exit node is responsible for the actual data request and returns the response, completing the entire traffic flow process.

[0092] To improve flexibility and high availability, the proxy service will select a suitable node from the available node queue to forward traffic based on the traffic load. At the same time, the ports (i.e., the first port) of multiple entry nodes are exposed to the client, allowing the client to connect to the proxy service through different entry nodes. After receiving the request, these entry nodes will select a suitable node combination based on the load balancing strategy (such as polling, least connection, etc.) to build a complete access link to ensure that the request can be efficiently and stably delivered to the target server.

[0093] In an exemplary embodiment, proxying the filtered node based on a preset forward proxy to obtain the forward proxy node and the second port includes:

[0094] According to the preset forward proxy, the filtered entry node and the first port are configured to obtain a forward proxy node; and the entry port of the forward proxy node is set as the second port.

[0095] Among them, the post-screening node includes a post-screening entry node.

[0096] For example, after starting the IP-Proxy service and building a complete access link, set the forward proxy configuration in the Nginx configuration file to specify the address of the proxy server. Nginx itself listens to a port as a unified entrance for external exposure.

[0097] In an exemplary embodiment, before performing node detection on the egress pool according to a preset configuration rule to obtain an available node queue and an unavailable node queue, the method further includes:

[0098] Generate monitoring access requests; proxy each monitoring access request according to a preset node proxy and a preset node pool to obtain monitoring nodes; determine a monitoring access link based on the monitoring nodes; perform address access according to the monitoring access link and the monitoring access requests to obtain monitoring response data; perform node detection on the egress pool according to a preset node partitioning algorithm and the monitoring response data to obtain an available node queue and an unavailable node queue.

[0099] Wherein, the preset node partitioning algorithm is a weighted average algorithm.

[0100] Exemplarily, all monitoring access requests are input through a proxy service node, and the IP-Proxy proxy service is started. The proxy service will screen out suitable ingress nodes, intermediate nodes, and egress nodes from the available node queues in the preset ingress pool, intermediate pool, and egress pool according to the target address of each request, and construct a complete access link. Specifically, the ingress node is responsible for receiving and processing the initial request from the client and forwarding the request to the intermediate node. The intermediate node, as a traffic transfer station, further forwards the request to the egress node, and the egress node is responsible for the actual data request and returns a response, completing the entire traffic flow process.

[0101] At the same time, record the response data of the egress node, and the response data includes the response time. Then, use the response data, the weighted average algorithm, and the node partitioning threshold to partition the nodes in the egress pool to obtain an available node queue and an unavailable node queue.

[0102] In an exemplary embodiment, as Figure 4 shown, performing node detection on the egress pool according to a preset node partitioning algorithm and the monitoring response data to obtain an available node queue and an unavailable node queue includes:

[0103] Step 402, in the case of detecting each egress node, determine an egress weight set based on multiple egress response times and a preset weight mapping relationship.

[0104] Exemplarily, before performing node detection, set the weights corresponding to the response times, specifically: when the response time is less than 50 ms, i.e., t < 50 ms, the weight is 10, marked as ɑ; when the response time is greater than 50 ms and less than 200 ms, i.e., 50 ms < t < 200 ms, the weight is 6, marked as β; when the response time is greater than 200 ms and less than 500 ms, i.e., 200 ms < t < 500 ms, the weight is 4, marked as θ; when the response time is greater than 500 ms, the weight is 0, marked as ɡ.

[0105] Multiple response data are obtained in the request log, the response data includes the export response time, and then the time set t is obtained, and the weight set f corresponding to the set t is determined, and then the weight value fn is determined according to the preset weight relationship.

[0106] For example, if node 1 is used for 10 requests, the response data corresponding to the 10 requests will be generated. The response data includes the response time. The specific response data is:

[0107] Web1:{Point_IP:[],Point:x1,responseList:[],PingTimes:10,Type:oion},

[0108] Web2:{Point_IP:[],Point:x1,responseList:[],PingTimes:220,Type:oion},

[0109] Web3:{Point_IP:[],Point:x1,responseList:[],PingTimes:110,Type:oion},

[0110] Web4:{Point_IP:[],Point:x1,responseList:[],PingTimes:10,Type:oion},

[0111] Web5:{Point_IP:[],Point:x1,responseList:[],PingTimes:40,Type:oion},

[0112] Web6:{Point_IP:[],Point:x1,responseList:[],PingTimes:500,Type:oion},

[0113] Web7:{Point_IP:[],Point:x1,responseList:[],PingTimes:2000,Type:oion},

[0114] Web8:{Point_IP:[],Poin:x1,responseList:[],PingTimes:60,Type:oion},

[0115] Web9:{Point_IP:[],Point:x1,responseList:[],PingTimes:990,Type:oion}.

[0116] The response time set t of node 1 is obtained: {10, 220, 110, 10, 40, 500, 2000, 60, 990}, and the weight set f corresponding to the set t is determined: {ɑ, θ, β, ɑ, ɑ, ɡ, ɡ, ɑ, ɡ} and the corresponding weight value set fn: {10, 4, 6, 10, 10, 0, 0, 10, 0}.

[0117] Step 404: perform data processing on multiple egress response times to obtain an egress response set.

[0118] Exemplarily, after obtaining set t, the data in set t are multiplied by a coefficient of 0.01, and then the data in the set are subtracted from 1 to obtain set Xn.

[0119] For example: multiply the data in the set t: {10,220,110,10,40,500,2000,60,990} by the coefficient 0.01 to obtain {0.01,0.22,0.11,0.01,0.04,0.5,2,0.06,0.99}, and then subtract 1 from the data in the set to obtain the set Xn: {0.99,0.78,0.89,0.99,0.96,0.5,-1,0.94,0.01}.

[0120] Step 406, performing weighted calculation according to the outlet weight set, the outlet response set and the preset node partitioning algorithm to obtain a response weighted value.

[0121] Exemplarily, the above set Xn and set fn are brought into a preset node partitioning algorithm, and the algorithm is as follows:

[0122]

[0123] Then the weighted response time of the node is calculated.

[0124] For example, Xn: {0.99, 0.78, 0.89, 0.99, 0.96, 0.5, -1, 0.94, 0.01} and fn: {10, 4, 6, 10, 10, 0, 0, 10, 0} are input into the algorithm, and the weighted response time of node 1 is calculated to be 0.945.

[0125] Step 408: When the response weighted value corresponding to each egress node is obtained, the egress pool is divided based on the preset response threshold and the response weighted value corresponding to the egress node to obtain an available node queue and an unavailable node queue.

[0126] Exemplarily, the node partitioning threshold is set to 0.85, and is compared with the weighted response time of the node to determine whether the node enters the available node queue or the unavailable node queue.

[0127] For example, the weighted response time of node 1 is 0.945, so node 1 enters the available node queue.

[0128] In an exemplary embodiment, the method further comprises:

[0129] According to the preset timed monitoring process, the egress pool is subjected to response detection to obtain the node response detection result; according to the node response result, the available node queue and the unavailable node queue are divided into queues to obtain the available queue after detection and the unavailable queue after detection.

[0130] Exemplarily, the crawler client is provided with a node timing detection thread, so when the node monitoring time is reached, based on the above Figure 4 The egress node detection step shown re-monitors the nodes in the available node queue and the unavailable node queue, thereby obtaining a post-detection available queue and a post-detection unavailable queue.

[0131] In an exemplary embodiment, a network resource data detection method is provided, the method comprising the following steps:

[0132] generating a plurality of address access requests based on the address monitoring list;

[0133] Generate monitoring access requests;

[0134] Proxify each monitoring access request according to the preset node proxy and the preset node pool to obtain the monitoring node;

[0135] Determining a monitoring access link based on a monitoring node;

[0136] Perform address access according to the monitoring access link and the monitoring access request to obtain monitoring response data;

[0137] When each egress node is detected, an egress weight set is determined based on multiple egress response times and preset weight mapping relationships;

[0138] Performing data processing on multiple export response times to obtain an export response set;

[0139] Perform weighted calculation according to the export weight set, the export response set and the preset node partitioning algorithm to obtain a response weighted value;

[0140] When the response weighted value corresponding to each egress node is obtained, the egress pool is divided based on the preset response threshold and the response weighted value corresponding to the egress node to obtain an available node queue and an unavailable node queue;

[0141] Perform response detection on the egress pool according to the preset timed monitoring process to obtain the node response detection result;

[0142] According to the node response results, the available node queue and the unavailable node queue are divided into queues to obtain the available queue after detection and the unavailable queue after detection;

[0143] Perform node detection on the egress pool according to preset configuration rules to obtain available node queues and unavailable node queues;

[0144] In the case of proxying each address access request, node screening is performed according to the preset node proxy, entry pool, intermediate pool and available node queue to obtain the screened entry node, screened intermediate node and screened available exit node;

[0145] When a plurality of post-screening entry nodes are obtained, the entry ports of all post-screening entry nodes are set as the first port;

[0146] According to the preset forward proxy, the filtered entry node and the first port are configured to obtain a forward proxy node;

[0147] Setting the inbound port of the forward proxy node to the second port;

[0148] generating a plurality of proxy service links based on the filtered node, the first port, the forward proxy node, and the second port of each address access request;

[0149] Requesting the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0150] The request response data is crawled according to the preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

[0151] The specific flow chart of the above steps is as follows Figure 5 shown.

[0152] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0153] Based on the same inventive concept, the embodiment of the present application also provides a network resource data detection device for implementing the network resource data detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more network resource data detection device embodiments provided below can refer to the limitations of the network resource data detection method above, and will not be repeated here.

[0154] In one embodiment, Figure 6 As shown, a network resource data detection device is provided, including: a request generation module 602, a proxy service module 604, a data acquisition module 606 and a page crawling module 608, wherein:

[0155] A request generation module 602, configured to generate a plurality of address access requests based on the address monitoring list;

[0156] The proxy service module 604 is used to proxy multiple address access requests according to a preset proxy service to obtain multiple proxy service links;

[0157] A data acquisition module 606 is used to request the address corresponding to each address access request according to multiple proxy service links to obtain request response data;

[0158] The page crawling module 608 is used to crawl the request response data according to the preset crawler process, obtain page information data and page screenshots, and transmit the page information data and the page screenshots to the content review platform.

[0159] In an exemplary embodiment, the proxy service module 604 is also used to perform proxy filtering on each address access request according to a preset node proxy and a preset node pool to obtain a filtered node and a first port; to proxy the filtered node based on a preset forward proxy to obtain a forward proxy node and a second port; and to generate multiple proxy service links based on the filtered node, the first port, the forward proxy node and the second port of each address access request.

[0160] In an exemplary embodiment, the proxy service module 604 is also used to perform node detection on the exit pool according to preset configuration rules to obtain an available node queue and an unavailable node queue; when proxying each address access request, perform node screening according to the preset node proxy, entry pool, intermediate pool and available node queue to obtain screened entry nodes, screened intermediate nodes and screened available exit nodes; when multiple screened entry nodes are obtained, set the entry ports of all screened entry nodes to the first port.

[0161] In an exemplary embodiment, the proxy service module 604 is further used to configure the filtered entry node and the first port according to the preset forward proxy to obtain the forward proxy node; and set the entry port of the forward proxy node to the second port.

[0162] In an exemplary embodiment, the network resource data detection device is also used to generate a monitoring access request; proxy each monitoring access request according to a preset node proxy and a preset node pool to obtain a monitoring node; determine a monitoring access link based on the monitoring node; perform address access according to the monitoring access link and the monitoring access request to obtain monitoring response data; perform node detection on the export pool according to a preset node partitioning algorithm and the monitoring response data to obtain an available node queue and an unavailable node queue.

[0163] In an exemplary embodiment, the network resource data detection device is also used to determine an export weight set based on multiple export response times and a preset weight mapping relationship when detecting each export node; perform data processing on multiple export response times to obtain an export response set; perform weighted calculation based on the export weight set, the export response set and a preset node partitioning algorithm to obtain a response weighted value; when obtaining the response weighted value corresponding to each export node, divide the export pool based on the preset response threshold and the response weighted value corresponding to the export node to obtain an available node queue and an unavailable node queue.

[0164] In an exemplary embodiment, the network resource data detection device is also used to perform response detection on the export pool according to a preset timed monitoring process to obtain a node response detection result; and to divide the available node queue and the unavailable node queue into queues according to the node response results to obtain an available queue after detection and an unavailable queue after detection.

[0165] Each module in the above network resource data detection device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0166] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store request log data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a network resource data detection method is implemented.

[0167] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0169] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0170] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0172] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0173] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A network resource data detection method, characterized in that: The method comprises: generating a plurality of address access requests based on the address monitoring list; Proxify the plurality of address access requests according to a preset proxy service to obtain a plurality of proxy service links; Requesting the address corresponding to each address access request according to the plurality of proxy service links to obtain request response data; The request response data is crawled according to a preset crawler process to obtain page information data and page screenshots, and the page information data and the page screenshots are transmitted to the content review platform.

2. The network resource data detection method according to claim 1, characterized in that: The preset proxy service includes a preset node proxy and a preset forward proxy; The proxying of the plurality of address access requests according to the preset proxy service to obtain a plurality of proxy service links includes: Perform proxy screening on each of the address access requests according to the preset node proxy and the preset node pool to obtain the screened node and the first port; Proxify the screened node based on a preset forward proxy to obtain a forward proxy node and a second port; A plurality of proxy service links are generated based on the screened node, the first port, the forward proxy node, and the second port for each of the address access requests.

3. The network resource data detection method according to claim 2, characterized in that: The preset node pool includes an entry pool, an intermediate pool and an exit pool, and the exit pool includes an available node queue; performing proxy screening on each address access request according to the preset node proxy and the preset node pool to obtain the screened node and the first port includes: Perform node detection on the egress pool according to preset configuration rules to obtain available node queues and unavailable node queues; In the case of proxying each of the address access requests, performing node screening according to the preset node proxy, the entry pool, the intermediate pool and the available node queue to obtain a screened entry node, a screened intermediate node and a screened available exit node; When a plurality of the post-screening entry nodes are obtained, the entry ports of all the post-screening entry nodes are set as the first port.

4. The network resource data detection method according to claim 2, characterized in that: The post-screening node includes a post-screening entry node; The step of proxying the screened node based on a preset forward proxy to obtain a forward proxy node and a second port includes: According to the preset forward proxy, the filtered entry node and the first port are configured to obtain a forward proxy node; The inbound port of the forward proxy node is set to the second port.

5. The network resource data detection method according to claim 1, characterized in that: Before performing node detection on the egress pool according to the preset configuration rule to obtain an available node queue and an unavailable node queue, the method further includes: Generate monitoring access requests; Proxify each monitoring access request according to the preset node proxy and the preset node pool to obtain a monitoring node; Determining a monitoring access link based on the monitoring node; Performing address access according to the monitoring access link and the monitoring access request to obtain monitoring response data; Node detection is performed on the egress pool according to a preset node partitioning algorithm and the monitoring response data to obtain an available node queue and an unavailable node queue.

6. The network resource data detection method according to claim 5, characterized in that: The monitoring response data includes multiple egress response times of each egress node; the node detection of the egress pool is performed according to the preset node partitioning algorithm and the monitoring response data to obtain an available node queue and an unavailable node queue, including: When each of the egress nodes is detected, an egress weight set is determined based on a plurality of egress response times and a preset weight mapping relationship; Performing data processing on a plurality of the egress response times to obtain an egress response set; Perform weighted calculation according to the outlet weight set, the outlet response set and the preset node partitioning algorithm to obtain a response weighted value; When the response weighted value corresponding to each of the egress nodes is obtained, the egress pool is divided based on a preset response threshold and the response weighted value corresponding to the egress node to obtain an available node queue and an unavailable node queue.

7. The network resource data detection method according to claim 1, characterized in that: The method further comprises: Perform response detection on the egress pool according to a preset timing monitoring process to obtain a node response detection result; The available node queue and the unavailable node queue are divided into queues according to the node response results to obtain a post-detection available queue and a post-detection unavailable queue.

8. A network resource data detection device, characterized in that: The device comprises: A request generation module, used for generating multiple address access requests based on the address monitoring list; A proxy service module, used to proxy the plurality of address access requests according to a preset proxy service to obtain a plurality of proxy service links; A data acquisition module, used to request the address corresponding to each address access request according to the plurality of proxy service links, and obtain request response data; The page crawling module is used to crawl the request response data according to a preset crawler process, obtain page information data and page screenshots, and transmit the page information data and the page screenshots to the content review platform.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images