Information crawling method and device, electronic equipment and storage medium

Abstract

An information crawling method and device, electronic equipment and storage medium, the method comprises: opening a target uniform resource locator (URL) network resource in a browser to enter a target page corresponding to the target URL network resource (101); locating a document object model (DOM) element where target valuable information is located in the target page to obtain the target DOM element (102); obtaining positioning path information of the target DOM element (103); loading the target URL network resource into a new tab window (104); extracting the target valuable information from the new tab window according to the positioning path information, and uniformly storing the target valuable information (105). Thus, a single-machine crawler based on a pure browser environment can be realized in a short time. For a developer, the development cycle is short and the operation is simple, thereby effectively reducing the development threshold, configuration management and maintenance cost, so that low-cost, convenient and efficient lightweight information crawling can be realized.

Description

Technical Field

[0001] This application relates to the field of computers, specifically to an information crawling method, apparatus, electronic device, and storage medium. Background Technology

[0002] Web crawling is a process of automatically analyzing, collecting, parsing, and storing large amounts of valuable information on the internet. Existing web crawling systems are mainly divided into two categories based on their system architecture: standalone and distributed. These systems are primarily based on popular Python and Java web crawling frameworks (such as Scrapy and Nutch) to analyze and retrieve target information.

[0003] Existing web crawling frameworks have complex and cumbersome interfaces, resulting in the following drawbacks for small-scale or temporary crawling tasks: First, they have long development cycles and high maintenance costs. For example, when implementing crawling tasks based on existing single-machine and distributed crawling frameworks, it's necessary to consider not only how to implement Python and Java code but also the configuration and management of servers and corresponding databases. Therefore, for temporary crawling needs, existing crawling frameworks have excessively long development cycles, high learning costs, and high maintenance costs. Second, they struggle to crawl asynchronous JavaScript and Extensible Markup Language (AJAX) information and dynamically generated value information from JavaScript code. Existing crawling frameworks struggle to locate target value information using AJAX asynchronous loading and dynamically generated JavaScript code, requiring the use of application automation testing frameworks to simulate the real webpage browsing process to extract target value information. This necessitates installing browser testing frameworks and corresponding browser drivers, adding extra cost and overhead. Third, they are easily restricted by anti-crawler mechanisms and login verification. For example, the browser identifiers of existing crawling frameworks are too simple, making them easily detectable by anti-crawler mechanisms. Summary of the Invention

[0004] This application provides an information crawling method and related products, which can achieve low-cost, convenient and efficient lightweight information crawling.

[0005] In a first aspect, an embodiment of this application provides an information crawling method, comprising:

[0006] Open the target Uniform Resource Locator (URL) network resource in the browser to enter the target page corresponding to the target URL network resource;

[0007] Locate the Document Object Model (DOM) element containing the target value information on the target page to obtain the target DOM element;

[0008] Obtain the location path information of the target DOM element;

[0009] Load the target URL network resource into a new tab window based on the location path information;

[0010] The target value information is extracted from the new tab window and stored uniformly.

[0011] Secondly, embodiments of this application provide an information crawling device, the device comprising: an activation unit, a positioning unit, an acquisition unit, a loading unit, an extraction unit, and a storage unit, wherein,

[0012] The opening unit is used to open the target Uniform Resource Locator (URL) network resource in a browser and enter the target page corresponding to the target URL network resource;

[0013] The positioning unit is used to locate the Document Object Model (DOM) element containing the target value information in the target page, and obtain the target DOM element.

[0014] The acquisition unit is used to acquire the location path information of the target DOM element;

[0015] A loading unit is used to load the target URL network resource into a new tab window based on the location path information;

[0016] Extraction unit, used to extract the target value information in the new label window;

[0017] A storage unit is used to uniformly store the target value information.

[0018] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of embodiments of this application.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of embodiments of this application.

[0020] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package. Attached Figure Description

[0021] The accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0024] Figure 1B A schematic diagram of another electronic device provided in the embodiments of this application;

[0025] Figure 1C This is a flowchart illustrating an information crawling method disclosed in an embodiment of this application;

[0026] Figure 2 This is a flowchart illustrating another information crawling method disclosed in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram of the structure of another electronic device disclosed in an embodiment of this application;

[0028] Figure 4A This is a schematic diagram of the structure of an information crawling device disclosed in an embodiment of this application;

[0029] Figure 4B This is one of the embodiments disclosed in this application. Figure 4A The described variant structure of the information crawling device. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0031] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] The electronic devices involved in the embodiments of this application may include various handheld devices, vehicle-mounted devices, wearable devices (smartwatches, wireless headsets), computing devices or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile station (MS), terminal device, etc. For ease of description, the devices mentioned above are collectively referred to as electronic devices. Electronic devices may also be servers.

[0034] To facilitate a better understanding of the technical solutions described in this application, the technical terms involved in the embodiments of this application are explained below:

[0035] Python is an object-oriented, cross-platform computer programming language.

[0036] Java is an object-oriented, cross-platform computer programming language.

[0037] JavaScript is an object-oriented web programming language.

[0038] Scrapy is an open-source web crawling framework written in Python.

[0039] Nutch is an open-source search engine written in Java.

[0040] User-agent is an HTTP header field used to identify the browser, browser operating system, encryption level, and browser rendering engine.

[0041] Cookie is a Hypertext Transfer Protocol (HTTP) header field used to identify legitimate users.

[0042] The embodiments of this application will be described in detail below.

[0043] Please see Figure 1A , Figure 1A This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 100 may include a control circuit, which may include a storage and processing circuit 110. The storage and processing circuit 110 may be a memory, such as a hard disk drive memory, a non-volatile memory (e.g., flash memory or other electronically programmable read-only memory used to form a solid-state drive), or a volatile memory (e.g., static or dynamic random access memory), etc., and this embodiment of the application is not limited thereto. The processing circuit in the storage and processing circuit 110 can be used to control the operation of the electronic device 100. The processing circuit may be implemented based on one or more microprocessors, microcontrollers, baseband processors, power management units, audio codec chips, application-specific integrated circuits, display driver integrated circuits, etc.

[0044] The storage and processing circuitry 110 can be used to run software in the electronic device 100, such as internet browsing applications, Voice over Internet Protocol (VoIP) telephone calling applications, email applications, media playback applications, operating system functions, etc. This software can be used to perform various control operations, such as image acquisition based on a camera, ambient light measurement based on an ambient light sensor, proximity sensor measurement based on a proximity sensor, information display functions based on status indicators such as LED status lights, touch event detection based on a touch sensor, functions associated with displaying information on multiple (e.g., layered) displays, operations associated with performing wireless communication functions, operations associated with collecting and generating audio signals, control operations associated with collecting and processing button press event data, and other functions in the electronic device 100, etc., which are not limited in the embodiments of this application.

[0045] The electronic device 100 may further include an input-output circuit 150. The input-output circuit 150 enables the electronic device 100 to input and output data, allowing the electronic device 100 to receive data from external devices and also allowing the electronic device 100 to output data from the electronic device 100 to external devices. The input-output circuit 150 may further include a sensor 170. The sensor 170 may include an ambient light sensor, a light- and capacitance-based proximity sensor, a touch sensor (e.g., a light-based touch sensor and / or a capacitive touch sensor, wherein the touch sensor may be part of a touch display screen or used independently as a touch sensor structure), an accelerometer, a gravity sensor, and other sensors, etc.

[0046] The input-output circuit 150 may also include one or more displays, such as display 130. Display 130 may include one or more of the following: liquid crystal display, organic light-emitting diode display, electronic ink display, plasma display, and displays using other display technologies. Display 130 may include a touch sensor array (i.e., display 130 may be a touch screen). The touch sensor may be a capacitive touch sensor formed by an array of transparent touch sensor electrodes (e.g., indium tin oxide (ITO) electrodes), or it may be a touch sensor formed using other touch technologies, such as acoustic touch, pressure-sensitive touch, resistive touch, optical touch, etc., which are not limited in the embodiments of this application.

[0047] Audio component 140 can be used to provide audio input and output functions for electronic device 100. Audio component 140 in electronic device 100 may include speakers, microphones, buzzers, tone generators, and other components for generating and detecting sound.

[0048] Communication circuit 120 can be used to provide electronic device 100 with the ability to communicate with external devices. Communication circuit 120 may include analog and digital input-output interface circuitry, and wireless communication circuitry based on radio frequency signals and / or optical signals. The wireless communication circuitry in communication circuit 120 may include radio frequency transceiver circuitry, power amplifier circuitry, low-noise amplifier, switches, filters, and antennas. For example, the wireless communication circuitry in communication circuit 120 may include circuitry for supporting near-field communication (NFC) by transmitting and receiving near-field coupled electromagnetic signals. For example, communication circuit 120 may include a near-field communication antenna and a near-field communication transceiver. Communication circuit 120 may also include cellular telephone transceivers and antennas, wireless local area network transceiver circuitry and antennas, etc.

[0049] The electronic device 100 may further include a battery, power management circuitry, and other input-output units 160. Input-output units 160 may include buttons, joysticks, click wheels, scroll wheels, touchpads, keypads, keyboards, cameras, LEDs, and other status indicators.

[0050] Users can input commands through the input-output circuit 150 to control the operation of the electronic device 100, and can use the output data of the input-output circuit 150 to receive status information and other outputs from the electronic device 100.

[0051] In related technologies, web crawling frameworks can be categorized into standalone and distributed crawling frameworks. Scrapy, a standalone crawling framework implemented in Python, primarily consists of five modules: the Scrapy engine, task scheduler, downloader, crawler, and pipeline. The Scrapy engine is responsible for sending crawling commands to each module and coordinating communication and data transfer between them. The task scheduler manages the unified scheduling and queueing of Uniform Resource Locators (URLs) sent by the Scrapy engine. The downloader sends URL requests to the URL network resources and retrieves the URL responses. The crawler parses the response content, extracts the necessary valuable information, and passes it to the pipeline for unified analysis, filtering, and storage. Nutch, a distributed search engine and crawling framework implemented in Java, relies primarily on a distributed infrastructure to achieve distributed crawling and data storage of massive amounts of information. It mainly consists of a generator, task scheduler, downloader, parser, and storage module. The generator primarily queries the database for target value information and the task scheduler dynamically distributes search tasks to the distributed system infrastructure cluster to complete the search for and indexing of target value information. The downloader and parser are responsible for establishing URL network requests and extracting information fields from the URL network responses. Finally, the storage unit centrally stores the target value information.

[0052] The existing web crawling frameworks mentioned above have good crawling capabilities for tasks involving massive amounts of information. However, for lightweight information crawling tasks, the existing web crawling frameworks have long development cycles, high maintenance costs, difficulty in crawling AJAX information and value information dynamically generated by JavaScript code, and are also easily restricted by anti-crawler mechanisms.

[0053] Based on this, please refer to Figure 1B , Figure 1BA schematic diagram of another electronic device is provided, wherein the electronic device includes an information crawling framework for implementing the information crawling method involved in the embodiments of this application, wherein the information crawling framework may include a browser 100, a browser console 110, a network resource loader 120, a network resource parser 130, and a memory 140, wherein...

[0054] The browser 100 is used to open the target Uniform Resource Locator (URL) network resource and enter the target page corresponding to the target URL network resource.

[0055] The console 110 is used to open a new tab window and load the URL network resource in the new tab window;

[0056] The network resource parser 130 is used to locate the document object model (DOM) element where the target value information is located in the target page, obtain the target DOM element, and acquire the location path information of the target DOM element;

[0057] The network resource loader 120 is used to load the target value information through a URL network resource;

[0058] The network resource parser 130 is also used to extract the target value information in the new tab window based on the location path information;

[0059] The memory 140 is used to uniformly store the target value information.

[0060] The aforementioned information crawling framework requires no installation of Java or Python runtime environments or application automation testing framework dependencies. It also requires no configuration of any distributed system infrastructure servers or databases. It only needs to leverage the browser's built-in functionality to locate the target information, enabling a single-machine crawler based on a pure browser environment in a short time. For web developers, this results in a short development cycle and simple operation, effectively reducing development barriers, configuration management, and maintenance costs. Since most operating systems come with built-in browser applications, this solution has a cross-platform advantage, capable of handling penetration testing, security testing, and other temporary and targeted crawling projects on different platforms, demonstrating excellent cross-platform compatibility. Furthermore, this solution uses a real browser to initiate normal browsing behavior, providing strong resistance to anti-crawler measures.

[0061] Please see Figure 1C , Figure 1C This is a flowchart illustrating an information crawling method provided in an embodiment of this application. The information crawling method described in this embodiment is applied to, for example... Figure 1A or Figure 1BThe electronic device shown, the information crawling method includes:

[0062] 101. Open the target Uniform Resource Locator (URL) network resource in the browser and enter the target page corresponding to the target URL network resource.

[0063] This means that the target URL network resource can be opened in a browser. The target Uniform Resource Locator (URL) is used to identify the location and access method of the network resource.

[0064] The target page is the browser page corresponding to the target URL network resource. You can access the target page by opening the target URL network resource through a browser.

[0065] Optionally, step 101 above, when opening the target Uniform Resource Locator (URL) network resource in a browser, may also include the following steps:

[0066] If the target URL network resource requires a login account, obtain the login account information corresponding to the URL network resource;

[0067] The login account information is verified. If the verification is successful, the operation of entering the target page corresponding to the target URL network resource is executed.

[0068] Considering that some online resources require user account login, for target URLs that require account login, the login account information corresponding to the URL can be obtained. Specifically, when logging into the account, the login account information can be obtained through user input, such as the username, password, and verification code entered by the user through a browser on an electronic device. Optionally, upon first login, the login account information can be recorded and saved so that subsequent data scraping can directly retrieve the saved login account information without requiring the user to re-enter it.

[0069] As can be seen, the embodiments of this application initiate normal browsing behavior based on a real browser and carry normal user account information. Therefore, existing anti-crawler technologies based on login function restrictions are difficult to restrict, thereby improving anti-crawler capabilities.

[0070] 102. Locate the Document Object Model (DOM) element containing the target value information on the target page to obtain the target DOM element.

[0071] In this embodiment, the target DOM element containing the target value information can be located through a browser. This allows for the location of the target value information solely through a browser, eliminating the need for an application-based automated testing framework to simulate real webpage browsing, thus saving costs and simplifying the operation.

[0072] Optionally, in step 102 above, locating the Document Object Model (DOM) element containing the target value information on the target page and obtaining the target DOM element may include the following steps:

[0073] The target DOM element containing the target value information is located using the browser's page element inspection function.

[0074] In specific implementation, the browser has a page element inspection function. Electronic devices can use the browser's page element inspection function to locate the target value information in the target DOM element, thus obtaining an accurate target DOM element location result.

[0075] 103. Obtain the location path information of the target DOM element.

[0076] The location path information may include cascading style sheets (CSS) selectors or extensible markup language (XPath) paths.

[0077] In practice, electronic devices can locate the DOM node where the target information is located and obtain the CSS selector or XPath path of that node element. By obtaining the location path information through the browser, the location path information corresponding to the target value information can be located in a short time, thereby improving the efficiency of information crawling.

[0078] Optionally, in step 103 above, obtaining the location path information of the target DOM element may include the following steps:

[0079] 21. Locate the DOM node containing the target value information;

[0080] 22. Obtain the positioning path information corresponding to the positioning path of the first node element under the DOM node.

[0081] In this context, a DOM node refers to each component in an XML document being a node, the entire document being a document node, and each XML tag being an element node.

[0082] In practice, the electronic device can first locate the DOM node where the target value information is located, then obtain the first node element under the DOM node, and obtain the CSS selector or XPath path for locating the first node element. In this way, accurate location path information can be obtained.

[0083] Optionally, after step 103 above, the following steps may also be included:

[0084] 31. Verify the validity of the location path information by using the browser's console.

[0085] 32. If so, perform the operation of loading the target URL network resource into a new tab window;

[0086] 33. If not, adjust the location path information.

[0087] In this embodiment of the application, the browser console can be opened, and then the validity of the CSS selector or XPath path can be verified. If the CSS selector or XPath path is valid, the process of loading the target URL network resource into a new tab window can continue, thereby extracting the target value information. If the CSS selector or XPath path is invalid, the CSS selector or XPath path can be adjusted.

[0088] Optionally, in step 31 above, verifying the validity of the location path information through the console may include the following steps:

[0089] Enter the location path corresponding to the location path information in the console. If the target DOM element can be successfully located, the location path corresponding to the location path information is determined to be valid.

[0090] One method is to open the browser's console and enter the CSS selector or XPath path. If the target DOM element can be successfully located, the CSS selector or XPath path is valid. If the target DOM element cannot be successfully located, the CSS selector or XPath path is invalid.

[0091] Optionally, adjusting the positioning path information in step 33 above may include the following steps:

[0092] Obtain the positioning path information corresponding to the positioning path of the second node element under the DOM node, wherein the second node element and the first node element respectively correspond to different child nodes under the DOM node;

[0093] Adjust the positioning path information to the positioning path information corresponding to the positioning path of the second node element.

[0094] Specifically, the electronic device can obtain the CSS selector or XPath path of the second node element under the DOM node, obtain the adjusted CSS selector or XPath path, and input the adjusted CSS selector or XPath path into the console to determine whether the adjusted CSS selector or XPath path is valid. In this way, by adjusting the positioning path information, it can be ensured that the positioning path information corresponding to the target value information is located.

[0095] 104. Load the target URL network resource into a new tab window.

[0096] In this method, electronic devices can open a new tab window through the browser's console, and then load the target URL network resource into the new tab window, thereby extracting the target value information in the new tab window.

[0097] 105. Extract the target value information from the new tab window based on the location path information, and store the target value information uniformly.

[0098] The target value information can include AJAX information and value information generated by JavaScript code. In practice, the electronic device can extract the target value information from the new tab window based on the location path information, and then store the target value information in the memory. Thus, the crawling of AJAX information and value information generated by JavaScript code can be achieved solely through a browser, without the need to install Java or Python runtime environments or application automation testing framework dependencies, or configure any distributed system infrastructure servers or databases. It only requires leveraging the browser's own functions to locate the target value information, enabling the crawling of target value information in a short time. This results in a short development cycle, simple operation, and reduced development threshold and maintenance costs.

[0099] Furthermore, since most existing operating systems come with browser applications, this solution has the advantage of cross-platform compatibility, and can meet the needs of penetration testing, security testing, and other temporary and targeted crawling projects on different platforms. Therefore, this solution has good cross-platform compatibility.

[0100] Optionally, in step 105 above, extracting the target value information in the new label window based on the location path information may include the following steps:

[0101] 51. Inject crawler code into the new tab window via the console;

[0102] 52. Execute the crawler code and extract the target value information based on the location path information.

[0103] The crawler code mentioned above can be JavaScript code, which is an object-oriented web programming language.

[0104] In practice, crawler code can be injected into a new tab window via the console, and the crawler code can be executed in the new tab window. The target value information can be extracted based on CSS selectors or XPath paths. In this way, value information dynamically generated by JavaScript code can be extracted, achieving better dynamic information crawling capabilities.

[0105] Optionally, the crawler code is JavaScript code. In step 52 above, extracting the target value information based on the location path information may include the following steps:

[0106] The browser parses and renders the target value information.

[0107] The target value information is downloaded based on the location path information. The target value information includes AJAX information and value information generated by the JavaScript code. The AJAX information is asynchronous JavaScript and Extensible Markup Language (XML) information.

[0108] During the execution of JavaScript code, the browser can parse and render the target value information and then download it according to the location path information. Thus, the browser can parse and render the target value information without installing Java or Python runtime environments, application automatic testing framework dependencies, or configuring any distributed system infrastructure servers and databases.

[0109] As can be seen, the information crawling method described in this application involves opening a target Uniform Resource Locator (URL) network resource in a browser to access the target page corresponding to the target URL network resource; locating the Document Object Model (DOM) element containing the target value information on the target page to obtain the target DOM element; acquiring the location path information of the target DOM element; loading the target URL network resource into a new tab window; extracting the target value information in the new tab window based on the location path information; and uniformly storing the target value information. This eliminates the need to install Java or Python runtime environments and application automation testing framework dependencies, and eliminates the need to configure any distributed system infrastructure servers and databases. It only requires leveraging the browser's own functionality to locate the target value information, enabling a single-machine crawler based on a pure browser environment in a short time. For web developers, this results in a short development cycle and simple operation, effectively reducing the development threshold, configuration management, and maintenance costs, thus achieving low-cost, convenient, and efficient lightweight information crawling.

[0110] Consistent with the above, please refer to Figure 2 , Figure 2 This is a flowchart illustrating another information crawling method provided in this application embodiment. The information crawling method described in this embodiment is applied to, for example... Figure 1A or Figure 1B The method for the electronic device shown may include the following steps:

[0111] Open the target Uniform Resource Locator (URL) in a browser. Determine if the target URL requires account login. If login is required, obtain the login account information. Verify the login account information. If successful, navigate to the target page. Locate the target DOM element containing the target value information using the browser's element inspection function. If login is not required, directly locate the target DOM element containing the target value information using the browser's element inspection function. Obtain the location path information of the target DOM element. Verify the validity of the location path using the browser's console. If valid, load the target URL into a new tab. If not, adjust the location path information. Open the new tab using the console. Load the target URL into the new tab. Inject crawler code into the new tab using the console. Execute the crawler code and extract the target value information based on the location path information. Store the target value information uniformly.

[0112] For a detailed description of the above steps, please refer to Figure 1C The information scraping methods shown will not be elaborated upon here.

[0113] As can be seen, the information crawling method described in this application involves opening a target URL network resource in a browser, determining whether the target URL network resource requires an account login, and if so, obtaining the login account information corresponding to the URL network resource; verifying the login account information, and if successful, entering the target page corresponding to the target URL network resource; locating the target DOM element containing the target value information using the browser's page element inspection function; and if the target URL network resource does not require an account login, directly locating the target DOM element containing the target value information using the browser's page element inspection function; obtaining the location path information of the target DOM element; and controlling the browser... The system verifies the validity of the location path information. If valid, it loads the target URL network resource into a new tab window. If not, it adjusts the location path information, opens a new tab window via the console, loads the target URL network resource into the new tab window, injects crawler code into the new tab window via the console, executes the crawler code, and extracts target value information based on the location path information. The target value information is then stored uniformly. This enables low-cost, convenient, and efficient lightweight information crawling. Furthermore, this solution initiates normal browsing behavior based on a real browser and carries normal user account information. Therefore, existing anti-crawler technologies based on login function restrictions are difficult to restrict, thus improving anti-crawler capabilities.

[0114] The following is the apparatus for implementing the above information crawling method:

[0115] Consistent with the above, please refer to Figure 3 , Figure 3 This application provides an embodiment of an electronic device, comprising: a processor and a memory; and one or more programs, said programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the following steps:

[0116] Open the target Uniform Resource Locator (URL) network resource in the browser to enter the target page corresponding to the target URL network resource;

[0117] Locate the Document Object Model (DOM) element containing the target value information on the target page to obtain the target DOM element;

[0118] Obtain the location path information of the target DOM element;

[0119] Load the target URL network resource into a new tab window;

[0120] Based on the location path information, the target value information is extracted in the new tab window and stored uniformly.

[0121] As can be seen, the electronic device described in this application opens the target Uniform Resource Locator (URL) network resource in a browser, enters the target page corresponding to the target URL network resource; locates the Document Object Model (DOM) element containing the target value information on the target page, obtains the target DOM element; acquires the location path information of the target DOM element; loads the target URL network resource into a new tab window; extracts the target value information in the new tab window based on the location path information, and stores the target value information uniformly. In this way, there is no need to install Java or Python runtime environments and application automatic testing framework dependencies, and no need to configure any distributed system infrastructure servers and databases. It only needs to locate the target value information based on the browser's own functions. A single-machine crawler based on a pure browser environment can be realized in a short time. For web developers, the development cycle is short and the operation is simple, thereby effectively reducing the development threshold, configuration management and maintenance costs, and thus realizing low-cost, convenient and efficient lightweight information crawling.

[0122] In one possible example, regarding locating the Document Object Model (DOM) element containing the target value information on the target page and obtaining the target DOM element, the procedure includes instructions for performing the following steps:

[0123] The target DOM element containing the target value information is located using the browser's page element inspection function.

[0124] In one possible example, regarding obtaining the location path information of the target DOM element, the program includes instructions for performing the following steps:

[0125] Locate the DOM node containing the target value information;

[0126] Obtain the positioning path information corresponding to the positioning path of the first node element under the DOM node.

[0127] In one possible example, after obtaining the location path information of the target DOM element, the program further includes instructions for performing the following steps:

[0128] Verify the validity of the location path information by using the browser's console.

[0129] If so, perform the operation of loading the target URL network resource into a new tab window;

[0130] If not, adjust the location path information.

[0131] In one possible example, regarding the verification of the validity of the location path corresponding to the location path information via the console, the program includes instructions for performing the following steps:

[0132] Enter the location path corresponding to the location path information in the console. If the target DOM element can be successfully located, the location path corresponding to the location path information is determined to be valid.

[0133] In one possible example, regarding the adjustment of the location path information, the program includes instructions for performing the following steps:

[0134] Obtain the positioning path information corresponding to the positioning path of the second node element under the DOM node, wherein the second node element and the first node element respectively correspond to different child nodes under the DOM node;

[0135] Adjust the positioning path information to the positioning path information corresponding to the positioning path of the second node element.

[0136] In one possible example, before loading the target URL web resource into the new tab window, the program further includes instructions for performing the following steps:

[0137] Open the new tab window via the console;

[0138] In retrieving the target value information from the new tab window based on the location path information, the program includes instructions for performing the following steps:

[0139] Inject crawler code into the new tab window via the console;

[0140] The crawler code is executed, and the target value information is extracted based on the location path information.

[0141] In one possible example, the crawler code is JavaScript code, and in terms of extracting the target value information based on the location path information, the program includes instructions for performing the following steps:

[0142] The browser parses and renders the target value information.

[0143] Download target value information based on the location path information. The target value information includes AJAX information and value information generated by the JavaScript code. The AJAX information is asynchronous JavaScript and extensible markup language XML information.

[0144] In one possible example, the program also includes instructions for performing the following steps:

[0145] If the target URL network resource requires a login account, obtain the login account information corresponding to the URL network resource;

[0146] The login account information is verified. If the verification is successful, the operation of entering the target page corresponding to the target URL network resource is executed.

[0147] Please see Figure 4A , Figure 4A This is a schematic diagram of the structure of an information crawling device provided in this embodiment. This information crawling device is applied to, for example... Figure 1A As shown or Figure 1B The electronic device shown includes an information crawling device comprising: an activation unit 401, a positioning unit 402, an acquisition unit 403, a loading unit 404, an extraction unit 405, and a storage unit 406.

[0148] The opening unit 401 is used to open the target Uniform Resource Locator (URL) network resource in a browser and enter the target page corresponding to the target URL network resource.

[0149] The positioning unit 402 is used to locate the Document Object Model (DOM) element where the target value information is located in the target page, and obtain the target DOM element.

[0150] The acquisition unit 403 is used to acquire the location path information of the target DOM element;

[0151] The loading unit 404 is used to load the target URL network resource into a new tab window;

[0152] The extraction unit 405 is used to extract the target value information in the new label window according to the positioning path information;

[0153] The storage unit 406 is used to uniformly store the target value information.

[0154] As can be seen, the information crawling device described in this application embodiment is applied to electronic devices. It opens the target Uniform Resource Locator (URL) network resource in a browser, enters the target page corresponding to the target URL network resource; locates the Document Object Model (DOM) element containing the target value information on the target page, and obtains the target DOM element; obtains the location path information of the target DOM element; loads the target URL network resource into a new tab window; extracts the target value information in the new tab window according to the location path information, and stores the target value information uniformly. In this way, there is no need to install Java or Python runtime environments and application automatic testing framework dependencies, and no need to configure any distributed system infrastructure servers and databases. It only needs to rely on the browser's own functions to locate the target value information. It can realize a single-machine crawler based on a pure browser environment in a short time. For web developers, the development cycle is short and the operation is simple, thereby effectively reducing the development threshold, configuration management and maintenance costs, and thus realizing low-cost, convenient and efficient lightweight information crawling.

[0155] In one possible example, in locating the Document Object Model (DOM) element containing the target value information on the target page and obtaining the target DOM element, the locating unit 402 is specifically used for:

[0156] The target DOM element containing the target value information is located using the browser's page element inspection function.

[0157] In one possible example, the acquisition unit 403 is specifically used for obtaining the location path information of the target DOM element:

[0158] Locate the DOM node containing the target value information;

[0159] Obtain the positioning path information corresponding to the positioning path of the first node element under the DOM node.

[0160] In one possible example, such as Figure 4B , Figure 4B for Figure 4A The described variant structure of the information crawling device, which is similar to Figure 4A In comparison, it may also include: a verification unit 407 and an adjustment unit 408, wherein,

[0161] The opening unit 401 is also used to open the browser's console;

[0162] The verification unit 407 is used to verify whether the location path corresponding to the location path information is valid through the console.

[0163] If so, the loading unit 404 performs the operation of loading the target URL network resource into a new tab window;

[0164] The adjustment unit 408 is used to adjust the positioning path information if the positioning path corresponding to the positioning path information is invalid.

[0165] In one possible example, regarding the verification of the validity of the location path corresponding to the location path information via the console, the verification unit 407 is specifically used for:

[0166] Enter the location path corresponding to the location path information in the console. If the target DOM element can be successfully located, the location path corresponding to the location path information is determined to be valid.

[0167] In one possible example, regarding the adjustment of the positioning path information, the adjustment unit 408 is specifically used for:

[0168] Obtain the positioning path information corresponding to the positioning path of the second node element under the DOM node, wherein the second node element and the first node element respectively correspond to different child nodes under the DOM node;

[0169] Adjust the positioning path information to the positioning path information corresponding to the positioning path of the second node element.

[0170] In one possible example, the loading unit loads the target URL network resource into the new tab window before...

[0171] The opening unit 401 is also used to open the new tab window via the console;

[0172] In extracting the target value information in the new label window based on the location path information, the extraction unit 405 is specifically used for:

[0173] Inject crawler code into the new tab window via the console;

[0174] The crawler code is executed, and the target value information is extracted based on the location path information.

[0175] In one possible example, the crawler code is JavaScript code, and in the process of extracting the target value information based on the location path information, the extraction unit 405 is specifically used for:

[0176] The browser parses and renders the target value information.

[0177] Download target value information based on the location path information. The target value information includes AJAX information and value information generated by the JavaScript code. The AJAX information is asynchronous JavaScript and extensible markup language XML information.

[0178] In one possible example, the acquisition unit 403 is further configured to acquire the login account information corresponding to the target URL network resource if the target URL network resource requires a login account;

[0179] The opening unit is also used to verify the login account information. If the verification is successful, the operation of entering the target page corresponding to the target URL network resource is executed.

[0180] It is understood that the functions of each program module of the information crawling device in this embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, and will not be repeated here.

[0181] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the information crawling methods described in the above method embodiments.

[0182] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the information crawling methods described in the above method embodiments.

[0183] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0184] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0188] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0189] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include a flash drive, ROM, RAM, disk, or optical disk, etc.

[0190] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An information crawling method, characterized in that, The method includes: Open the target Uniform Resource Locator (URL) web resource in your browser to access the target page corresponding to the target URL web resource; The document object model (DOM) element containing the target value information in the target page is located to obtain the target DOM element, so that the target value information can be located through the browser without installing an application automatic testing framework to simulate real web page browsing to locate the target value information; Obtain the location path information of the target DOM element; Load the target URL network resource into a new tab window; The target value information is extracted from the new tab window based on the location path information. Specifically, this involves: injecting crawler code into the new tab window via the console; executing the crawler code and extracting the target value information based on the location path information; and storing the target value information uniformly.

2. The method according to claim 1, characterized in that, The process of locating the Document Object Model (DOM) element containing the target value information on the target page, and obtaining the target DOM element, includes: The target DOM element containing the target value information is located using the browser's page element inspection function.

3. The method according to claim 2, characterized in that, The step of obtaining the location path information of the target DOM element includes: Locate the DOM node containing the target value information; Obtain the positioning path information corresponding to the positioning path of the first node element under the DOM node.

4. The method according to claim 3, characterized in that, After obtaining the location path information of the target DOM element, the method further includes: Verify the validity of the location path information by using the browser's console. If so, perform the operation of loading the target URL network resource into a new tab window; If not, adjust the location path information.

5. The method according to claim 4, characterized in that, The step of verifying the validity of the location path information via the console includes: Enter the location path corresponding to the location path information in the console. If the target DOM element can be successfully located, the location path corresponding to the location path information is determined to be valid.

6. The method according to claim 4, characterized in that, The adjustment of the positioning path information includes: Obtain the positioning path information corresponding to the positioning path of the second node element under the DOM node, wherein the second node element and the first node element respectively correspond to different child nodes under the DOM node; Adjust the positioning path information to the positioning path information corresponding to the positioning path of the second node element.

7. The method according to any one of claims 1-6, characterized in that, Before loading the target URL network resource into the new tab window, the method further includes: Open the new tab window via the console.

8. The method according to claim 7, characterized in that, The crawler code is JavaScript code, and the step of extracting the target value information based on the location path information includes: The browser parses and renders the target value information. Download target value information based on the location path information. The target value information includes AJAX information and value information generated by the JavaScript code. The AJAX information is asynchronous JavaScript and Extensible Markup Language (XML) information.

9. The method according to claim 1, characterized in that, The method further includes: If the target URL network resource requires a login account, obtain the login account information corresponding to the URL network resource; The login account information is verified. If the verification is successful, the operation of entering the target page corresponding to the target URL network resource is executed.

10. An information crawling device, characterized in that, The device includes: an opening unit, a positioning unit, an acquisition unit, a loading unit, an extraction unit, and a storage unit, wherein, The opening unit is used to open the target Uniform Resource Locator (URL) network resource in the browser and enter the target page corresponding to the target URL network resource. The positioning unit is used to locate the Document Object Model (DOM) element where the target value information is located on the target page, and obtain the target DOM element so as to locate the target value information through the browser. It does not require the installation of an application automatic testing framework to simulate real web page browsing in order to locate the target value information. The acquisition unit is used to acquire the location path information of the target DOM element; The loading unit is used to load the target URL network resource into a new tab window; The extraction unit is used to extract the target value information in the new tab window according to the location path information, specifically by: injecting crawler code into the new tab window through the console; executing the crawler code; and extracting the target value information according to the location path information. The storage unit is used to uniformly store the target value information.

11. The apparatus according to claim 10, characterized in that, In locating the Document Object Model (DOM) element containing the target value information on the target page and obtaining the target DOM element, the locating unit is specifically used for: The target DOM element containing the target value information is located using the browser's page element inspection function.

12. The apparatus according to claim 11, characterized in that, In obtaining the location path information of the target DOM element, the obtaining unit is specifically used for: Locate the DOM node containing the target value information; Obtain the positioning path information corresponding to the positioning path of the first node element under the DOM node.

13. The apparatus according to claim 12, characterized in that, The device further includes a verification unit and an adjustment unit, wherein... The opening unit is also used to open the browser's console; The verification unit is used to verify whether the location path corresponding to the location path information is valid through the console. If so, the loading unit performs the operation of loading the target URL network resource into a new tab window; The adjustment unit is used to adjust the positioning path information if the positioning path corresponding to the positioning path information is invalid.

14. The apparatus according to claim 13, characterized in that, In verifying the validity of the location path information via the console, the verification unit is specifically used for: Enter the location path corresponding to the location path information in the console. If the target DOM element can be successfully located, the location path corresponding to the location path information is determined to be valid.

15. The apparatus according to claim 13, characterized in that, Regarding the adjustment of the positioning path information, the adjustment unit is specifically used for: Obtain the positioning path information corresponding to the positioning path of the second node element under the DOM node, wherein the second node element and the first node element respectively correspond to different child nodes under the DOM node; Adjust the positioning path information to the positioning path information corresponding to the positioning path of the second node element.

16. The apparatus according to any one of claims 10-15, characterized in that, The loading unit loads the target URL network resource into the new tab window before... The opening unit is also used to open the new tab window via the console.

17. An electronic device, characterized in that, This includes the browser, the browser console, the network resource loader, the network resource resolver, and storage, among which... The browser is used to open the target Uniform Resource Locator (URL) network resource and enter the target page corresponding to the target URL network resource. The console is used to open a new tab window and load the URL network resource in the new tab window; The network resource parser is used to locate the Document Object Model (DOM) element containing the target value information on the target page, obtain the target DOM element, and locate the target value information through the browser without installing an application's automatic testing framework to simulate real webpage browsing; and obtain the location path information of the target DOM element. The network resource loader is used to load the target value information through a URL network resource; The network resource parser is further configured to extract the target value information in the new tab window based on the location path information, specifically by: injecting crawler code into the new tab window via the console; executing the crawler code; and extracting the target value information based on the location path information. The memory is used to uniformly store the target value information.

18. An electronic device, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-9.

19. A computer-readable storage medium, characterized in that, A computer program is stored, wherein the computer program causes the computer to perform the method as described in any one of claims 1-9.

20. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the method as described in any one of claims 1-9.

Images