Web page element search method, device and computing equipment

Abstract

The present invention discloses a web page element search method, device, and computing equipment. The web page element search method includes the following steps: in response to a user's request to search for a web page element on a Chrome browser, sending the domain name and URL of the current site to a server, then obtaining a template returned by the server, wherein the template records a target field and an extraction rule for the target field; when the extraction rule for the target field is absolute positioning, extracting the target field in the current page using one of an Xpath selector, a CSS selector, and an ID selector; when the extraction rule for the target field is relative positioning, extracting a target container in the current page, and extracting the target field in the target container; and using the extracted target field as a search result for the web page element. The present invention also discloses a corresponding computing device and apparatus.

Description

Technical Field

[0001] The present invention relates to the field of crawlers, and in particular to a method and device for finding web page elements and a computing device. Background Art

[0002] Scraping data from Internet websites is a common requirement. Usually, a program or script is written to obtain the response of the target website, and then the required element fields are parsed from the response according to the own needs. When there are many fields, many target websites, and the target elements appear and disappear according to the data changes, the difficulty and workload of parsing the element fields will increase accordingly.

[0003] One way to find web page elements is manual parsing. This method requires programmers to customize a set of scraping programs for each target website, obtain the response of the target website through network requests, and then parse the required fields one by one from the HTML of the response. Before parsing the fields, developers need to first analyze the web page structure of the target website, find the selector for each field, and then locate the required field through the selector. The disadvantage of this solution is that developers need to analyze the web page structure and write suitable selectors for multiple pages, which is difficult, and it is not easy to test whether the selector is applicable to other pages of the same type. The reason is that with the development of front-end technologies, the front-end and back-end separation technology is becoming more and more common, and many page elements do not have fixed IDs, no fixed CSS, and some data is dynamic data, resulting in great difficulty in writing a good selector. Especially when obtaining a certain item in the table data, it is very easy to get the wrong data. For example, a table shows three fields, as shown in Table 1:

[0004] Table 1 Table with three fields

[0005] NAME Zhang San AGE 20 SEX male

[0006] Suppose the target element we want to obtain is the element with the value of AGE being 20. Then it is easy to obtain the data through the index of the Table row. In this case, if the AGE field of a certain page is not displayed, and the table data is as shown in Table 2:

[0007] Table 2 Table without the AGE field displayed

[0008] NAME Zhang San SEX male

[0009] In this case, using the table index is very likely to cause misalignment, and the final result will be parsed as "male", which becomes incorrect data.

[0010] Writing the right selector requires numerous attempts, which is extremely time-consuming. After writing a selector, you also need to test whether it works on other pages of the same type. For example, in Table 1 above, after completing a selector to retrieve age, you need to test whether the age data can be retrieved on another page, which is also inconvenient.

[0011] A second method for finding web page elements is the Portia project's visual acquisition. The Portia project uses a visual solution to embed the target website within an iframe. Using visual operations, it retrieves the target web page's elements, achieving rapid recording. It then crawls the target website using Scrapy, ultimately parsing the crawled response using the acquired web elements. This approach offers easy-to-use visual operations, with selectors automatically generated by the program, reducing developer workload. However, its disadvantages include limited support for front-end / back-end separated pages and limited flexibility for specialized pages. Portia uses an iframe to integrate the target website into its user interface. However, due to the security limitations of iframes, Portia uses a back-end proxy to assist in rendering the page, which prevents Portia from properly parsing and running on separated pages. With the popularity of Vue and React frameworks in modern websites, more and more websites are adopting the latest front-end / back-end separation technologies and the React framework. However, the Portia project's visual acquisition does not support React, limiting its widespread applicability.

[0012] A third method for finding web page elements is to use the XPath Helper Chrome plugin. XPath Helper implements a Chrome browser plugin that displays the XPath of the target object and highlights the target by hovering the mouse over a specified location. However, the XPath Helper Chrome plugin is relatively limited in functionality, only displaying the XPath of the target at the mouse cursor. The retrieved XPath only serves as a reference for the spider, and often fails to accurately retrieve dynamically laid out pages. The main reasons for this are as follows: some websites are inherently dynamic, displaying different layouts depending on the data. In modern websites, with the advent of frameworks like Vue and React, asynchronous loading techniques are ubiquitous, and JavaScript can easily add and delete a significant amount of content. This can easily cause XPath samples to become invalid, requiring only resampling and modifying the spider code. For example, in some cases, the XPaths collected by XPath Helper contain randomly generated patterns, which can change after a website is updated or released, rendering the XPath invalid. In addition, the XPath Helper Chrome plug-in has the following problems: it cannot be persisted, has no memory function, the selected content is lost after the page is refreshed, multiple fields cannot be selected at the same time, and manual copying is required, "crowded elements" cannot be selected, and the element pointed to by the mouse is the top-level element.

[0013] In summary, the above existing crawler technologies have problems such as difficulty in developing web page element plug-ins and poor adaptability to dynamic pages. Summary of the Invention

[0014] To this end, the present invention provides a web page element search method and apparatus and a computing device, in an effort to solve or at least alleviate at least one of the above problems.

[0015] According to one aspect of the present invention, a web page element search method is provided, comprising the steps of: in response to a user's request to search for web page elements on a Chrome browser, sending the domain name and URL of a current site to a server, obtaining a template returned by the server that matches the domain name and URL of the current site, the template recording a target field and a target field extraction rule; when the target field extraction rule is absolute positioning, extracting the target field in the current page using one of an XPath selector, a CSS selector, and an ID selector; when the target field extraction rule is relative positioning, extracting a target container in the current page, extracting the target field in the target container, the target container representing an element whose relative position to a preset node satisfies a preset condition; and using the extracted target field as a web page element search result.

[0016] Optionally, in the web page element search method according to the present invention, the step of extracting the target field in the current page using one of the Xpath selector, the CSS selector and the ID selector includes: extracting the target field in the current page using the Xpath selector; when the extraction of the target field in the current page using the Xpath selector fails, extracting the target field in the current page using the CSS selector; and when the extraction of the target field in the current page using the CSS selector fails, extracting the target field in the current page using the ID selector.

[0017] Optionally, in the web page element search method according to the present invention, the step of extracting the target container in the current page includes: extracting the target container in the current page according to the position of the target container recorded in the template relative to the preset node.

[0018] Optionally, in the web page element search method according to the present invention, the web page element search method further includes the steps of: in response to a user's request to record a template on a Chrome browser, saving the field name and field type newly added by the user; in response to a user's request to select a positioning method on a Chrome browser page, determining the positioning method selected by the user, the positioning method including absolute positioning and relative positioning; in response to a user's request to collect samples on a Chrome browser page, saving the element selected by the user on the current page as a sample; positioning the sample selected by the user according to the positioning method selected by the user, and using the process of positioning the sample selected by the user as an extraction rule for the target field; and in response to a user's request to save a template on a Chrome browser, saving the recorded template, and sending the saved template and the URL of the template to the server.

[0019] Optionally, in the web page element search method according to the present invention, the web page element search method further comprises: using a rule set by a user for cleaning the web page elements as a search expression of a target field.

[0020] Optionally, in the web page element search method according to the present invention, the template also records the search expression of the target field, and the step of using the extracted target field as the search result of the web page element also includes: cleaning the found target field according to the search expression of the target field in the template; and using the cleaned field as the final search result of the web page element.

[0021] According to another aspect of the present invention, a web page element search device is provided, comprising: a template acquisition unit, adapted to respond to a user's request to search for web page elements on a Chrome browser, send the domain name and URL of a current site to a server, and obtain a template returned by the server that matches the domain name and URL of the current site, wherein the template records a target field and an extraction rule for the target field; an absolute positioning extraction unit, adapted to extract a target field in the current page using one of an Xpath selector, a CSS selector, and an ID selector when the extraction rule for the target field is absolute positioning; a relative positioning extraction unit, adapted to extract a target container in the current page and extract a target field in the target container when the extraction rule for the target field is relative positioning, wherein the target container represents an element whose relative position to a preset node satisfies a preset condition; and a display unit, adapted to use the extracted target field as a search result for the web page element.

[0022] Optionally, in the web page element search device according to the present invention, the absolute positioning extraction unit also includes: an Xpath extraction subunit, suitable for extracting the target field in the current page using the Xpath selector; a CSS extraction subunit, suitable for extracting the target field in the current page using the CSS selector when the extraction of the target field in the current page using the Xpath selector fails; and an ID extraction subunit, suitable for extracting the target field in the current page using the ID selector when the extraction of the target field in the current page using the CSS selector fails.

[0023] Optionally, in the web page element search device according to the present invention, the web page element search device also includes: a new field unit, suitable for saving the field name and field type added by the user in response to the user's request to record the template on the Chrome browser; a positioning selection unit, suitable for determining the positioning method selected by the user in response to the user's request to select the positioning method on the Chrome browser page, the positioning method including absolute positioning and relative positioning; a sample collection unit, suitable for saving the element selected by the user on the current page as a sample in response to the user's request to collect the sample on the Chrome browser page; a sample positioning unit, suitable for positioning the sample selected by the user according to the positioning method selected by the user, and using the process of positioning the sample selected by the user as the extraction rule of the target field; and a template saving unit, suitable for saving the recorded template in response to the user's request to save the template on the Chrome browser, and sending the saved template and the template's URL rule to the server.

[0024] Optionally, in the web page element search device according to the present invention, the web page element search device further comprises: a cleaning setting unit adapted to use a rule set by a user for cleaning web page elements as a search expression of a target field.

[0025] Optionally, in the web page element search device according to the present invention, the template also records the search expression of the target field, and the display unit also includes: a cleaning sub-unit, suitable for cleaning the found target field according to the search expression of the target field in the template; and a display sub-unit, suitable for using the cleaned field as the final search result of the web page element.

[0026] According to another aspect of the present invention, a web page element search system is also provided, including: a client, suitable for executing the web page element search method of the present invention; and a server, connected to the client, suitable for receiving and saving templates sent by the client, and also suitable for receiving the domain name and URL of the current site sent by the client, performing template matching according to the template list, domain name and URL of the current site, and sending the matched template to the client.

[0027] According to another aspect of the present invention, a computing device is provided, comprising: at least one processor and a memory storing program instructions; when the program instructions are read and executed by the processor, the computing device executes the web page element search method as described above.

[0028] According to yet another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the web page element search method as described above.

[0029] The web page element search method, apparatus, and computing device according to the present invention can achieve at least one of the following beneficial effects:

[0030] The versatility of the template is improved by using the field template method (i.e., extracting target elements according to the target field and target field extraction rules recorded in the template). The versatility is further improved by using the URL regular matching template, so that the crawler can extract the required elements through the field template corresponding to the maintained site and URL;

[0031] The template recording process is visual, no software code editing is required, and the recording speed is fast;

[0032] By collecting web page elements based on the Chrome browser plug-in, the availability, ease of use, and testability of the web page element search method are improved, and it is compatible with all web pages;

[0033] Automatically generate search paths by taking into account both absolute and relative positioning, and regularly extract multiple extraction schemes to increase the success rate of extraction;

[0034] Visual operation: click the required element on the page with the mouse to automatically generate a selector, reducing development workload.

[0035] In the form of Chrome plug-in, the plug-in JS code runs in a sandbox, does not affect the target website, and is compatible with all websites;

[0036] Highlight the selected elements, what you see is what you get, high ease of use;

[0037] The selector of the selected element on the page can be persisted. When the page is refreshed, the page element is echoed according to the generated selector tag, which is convenient for testing whether the selector is applicable to other identical pages. After the page is refreshed, the sample data can still be echoed. The sample data can still be echoed after changing the page of the same type but different data.

[0038] Fault-tolerant processing mechanism, the selectors used include ID selector, CSS selector and XPath selector. As long as one selector can extract the target field, it means the extraction is successful, which improves the success rate of element extraction;

[0039] The relative positioning search function can automatically generate an XPath for dynamic data through a container with a relatively unchanged relative position and a relatively unchanged keyword to process dynamic table data;

[0040] Target element type specification, supporting text, numbers, and URLs, etc., customizing regular expression extraction rules, refining target rules, and more accurate extraction results;

[0041] It supports URL matching. Multiple page field templates can be recorded for multiple pages of a site. When extracting data, the corresponding field template is found according to the URL for parsing, which improves the reusability and versatility of the template. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other objects, features, and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0043] Figure 1 A schematic diagram of a web page element search system 100 according to an embodiment of the present invention is shown;

[0044] Figure 2 shows a schematic diagram of a computing device 200 according to one embodiment of the present invention;

[0045] Figure 3 A flow chart of a web page element search method 300 according to an embodiment of the present invention is shown;

[0046] Figure 4 A screenshot of a browser page showing a target field found by absolute positioning according to one embodiment of the present invention is shown;

[0047] Figure 5 A flowchart of a recording template according to an embodiment of the present invention is shown;

[0048] Figure 6 A schematic diagram of a web page element search device 400 according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0050] In view of the problems that the existing web page element search method has, such as the difficulty of plug-in development and poor adaptability to dynamic pages, the present invention provides a web page element search method, which can reduce the difficulty of plug-in development and expand the scope of application of plug-ins.

[0051] Figure 1 FIG. 1 is a schematic diagram of a web page element search system 100 according to an embodiment of the present invention.

[0052] like Figure 1 As shown, the web page element search system 100 includes a server 102 and several clients 101. The server 102 is connected to the several clients 101 in communication, for example, through a wired or wireless network connection.

[0053] In an embodiment of the present invention, client 101 is adapted to execute a web page element search method. Server 102 may be an application on a server. Server 102 stores a template list for each site, where each template list records all templates for a corresponding site. Server 102 is adapted to receive and save templates sent by client 101, and further adapted to receive the domain name and URL of the current site sent by client 101. Server 102 then searches the template list for the current site stored on the server for a template that matches the domain name and URL sent by client 101, and sends the matching template to the corresponding client 101.

[0054] The web page element search method 300 of the present invention will be described in detail below.

[0055] In one embodiment, the client 101 of the present invention can be implemented as a computing device, so that the web page element search method of the present invention can be executed on the computing device. The computing device can be any device with storage and computing capabilities, and can be implemented as, for example, a server, workstation, etc., or as a personal computer such as a desktop computer or laptop computer, or as a terminal device such as a mobile phone, tablet computer, smart wearable device, or Internet of Things device, but is not limited thereto.

[0056] Figure 2 FIG2 shows a schematic diagram of a computing device 200 according to an embodiment of the present invention. It should be noted that, Figure 2 The computing device 200 shown is only an example. In practice, the computing device used to implement the web page element search method of the present invention can be any type of device, and its hardware configuration can be different from the one in the embodiment of the present invention. Figure 2 The computing device 200 shown is the same as Figure 2 The computing device 200 shown is different. In practice, the computing device used to implement the automatic verification method of the present invention can be Figure 2 The hardware components of the computing device 200 shown are added or deleted, and the present invention does not limit the specific hardware configuration of the computing device.

[0057] like Figure 2 As shown, in a basic configuration 202, computing device 200 typically includes system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between processor 204 and system memory 206.

[0058] Depending on the desired configuration, the processor 204 can be any type of processor, including but not limited to: a microprocessor (UP), a microcontroller (UC), a digital signal processing unit (DSP), or any combination thereof. The processor 204 can include one or more levels of cache, such as a first-level cache 210 and a second-level cache 212, a processor core 214, and registers 216. An example processor core 214 can include an arithmetic logic unit (ALU), a floating-point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 218 can be used with the processor 204, or in some implementations, the memory controller 218 can be an internal part of the processor 204.

[0059] Depending on the desired configuration, system memory 206 can be any type of memory, including, but not limited to, volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 can include an operating system 220, one or more applications 222, and program data 224. Applications 222 are essentially multiple program instructions that instruct processor 204 to perform corresponding operations. In some embodiments, applications 222 can be arranged to operate on the operating system so that processor 204 can utilize program data 224.

[0060] The computing device 200 may also include a storage interface bus 234. The storage interface bus 234 enables communication from storage devices 232 (e.g., removable storage 236 and non-removable storage 238) via the bus / interface controller 230 to the basic configuration 202. At least a portion of the operating system 220, applications 222, and data 224 may be stored on the removable storage 236 and / or the non-removable storage 238 and loaded into the system memory 206 via the storage interface bus 234 when the computing device 200 is powered on or when an application 222 is to be executed, and executed by the one or more processors 204.

[0061] The computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus / interface controller 230. Example output devices 242 include a graphics processing unit 248 and an audio processing unit 250. These can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 252. Example peripheral interfaces 244 may include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communication with external devices such as input devices (e.g., a keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., a printer, scanner, etc.) via one or more I / O ports 258. Example communication devices 246 may include a network controller 260, which can be arranged to facilitate communication with one or more other computing devices 262 via a network communication link via one or more communication ports 264.

[0062] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. A "modulated data signal" can be a signal in which one or more of its data sets or changes thereto can be carried out in the signal in a manner that encodes information. As non-limiting examples, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium as used herein can include both storage media and communication media.

[0063] In the computing device 200 according to the present invention, the application 222 includes multiple program instructions for executing the web page element search method 300. These program instructions can instruct the processor 204 to execute the web page element search method 300 of the present invention so as to enable the computing device 200 to execute the web page element search method 300 of the present invention.

[0064] Figure 3 FIG. 3 is a flow chart of a web page element search method 300 according to an embodiment of the present invention. The method 300 is executed in a computing device (eg, the aforementioned computing device 200). Figure 3 As shown, the method 300 begins at step S310 .

[0065] According to an embodiment of the present invention, the web page element search method 300 provides the user with a web page element search function in the form of a browser plug-in, and the Chrome plug-in-based method can be compatible with all web pages. It should be noted that the browser here should be the Chrome browser.

[0066] S310 responds to the user's request to find web page elements on the Chrome browser, sends the domain name and URL of the current site to the server, and then obtains the template matched and returned by the server based on the template list, domain name and URL of the current site. The template records the target field and the extraction rules of the target field.

[0067] In step S310, the user clicks the web page element search plug-in on the Chrome browser of client 101. In the pop-up dialog box, when the user clicks the Find Web Page Elements button, step S310 is initiated and the domain name and URL of the current site (the URL can be used to search for corresponding templates) are sent to server 102. A site can have multiple templates, and different templates are applicable to different web pages of the same site (website). Server 102 stores a site table, a template list, and templates. The site table records the sites to be visited, and the template list records each template and its number. The template record contains the target field and the extraction rules for the target field, for example, extraction rules for different types of fields such as price, URL, email address, and phone number. A site can have multiple templates, and a template can have multiple target fields and corresponding extraction rules. Server 102 matches the templates in the template list of the current site based on the domain name and URL of the current site sent by client 101, and feeds the matched template back to client 101. In the next step, method 300 searches and extracts web page elements according to the target field and the extraction rules recorded in the template. If the extraction rule of the target field in the returned template is absolute positioning, then step S320 is executed next, and absolute positioning is applicable to static pages; if the extraction rule of the target field in the returned template is relative positioning, then step S330 is executed next, and relative positioning is applicable to dynamic pages.

[0068] The method of retrieving the template from the server 102 provides high versatility of the template. In most cases, there is no need to develop new crawler codes, and the data crawling task can be completed by using only a general public crawler.

[0069] In absolute positioning, the position of the target field refers to the target field relative to the browser's DOM. For example, if the target field is a title, the title is in the second section.

[0070] In step S320, the target field is extracted from the current page using the Xpath selector, the CSS selector, and the ID selector. Here, the three selectors are coordinated and the priority of each selector is reasonably set to ensure accurate extraction of the required target elements.

[0071] In one embodiment, step S320 may include steps S321 to S323.

[0072] In step S321 , the target field is extracted from the current page using the Xpath selector, that is, the target field is matched in the current page according to the Xpath of the target field recorded in the template.

[0073] In step S322, when the target field fails to be extracted in the current page using the Xpath selector, the target field is extracted in the current page using the CSS selector, that is, when the target field cannot be matched in the current page using the Xpath selector, the target field is matched in the current page using the CSS selector.

[0074] In step S323, when the CSS selector fails to extract the target field in the current page, the ID selector is used to extract the target field in the current page, that is, when the CSS selector cannot match the target field in the current page, the ID selector is used to match the target field in the current page.

[0075] In relative positioning, the position of the target field is the position of the target field relative to a certain element on the DOM.

[0076] In step S330 , when the extraction rule of the target field is relative positioning, a target container is extracted from the current page, and a target field is extracted from the target container. The target container represents an element whose relative position to a preset node meets a preset condition.

[0077] Taking price as an example, the web page element that the user wants to find is the price value. For example, if "price 100 yuan" appears on the page, "100" is the specific value of the price to be found. For dynamic web pages, the content of the web page changes. If absolute positioning is still used, for example, positioning to the second section, the element extracted may not be the element the user needs. Assuming that the price in the dynamic web page is immediately after the title, in this case, it is necessary to use relative positioning to find the target element. First, locate the title, and then locate the price based on its position relative to the title. Specifically: set the title as the base node, and set the relative position between the price and the title. This relative position is the target container. When searching for the price, first find the target container using the absolute position search method, and then search for the target element in the target container based on the position of the price relative to the title.

[0078] In step S340, the target fields extracted in steps S320 and S330 are used as search results for web page elements, and the search results are presented to the user.

[0079] In one embodiment, on the current page, the target fields extracted in step S320 and step S330 are marked with colored wireframes so that the user can intuitively see the extraction results.

[0080] Figure 4 FIG. 1 shows a screenshot of a browser page that uses absolute positioning to find a target field according to an embodiment of the present invention. Figure 4 As shown, the target field is price, i.e., "price" in the figure. The target field extraction rule is absolute positioning. On the current page, the matching price elements are outlined with a colored frame. If relative positioning is used, after finding the target container, the target container can be outlined with a colored frame, and the target elements within it can be outlined with a different colored frame. The frame can be solid or dashed.

[0081] It should be noted that the template in method 300 should have been stored in the server 102 before the web page element search. If the user finds that the server 102 does not return a usable template when searching for web page elements, the user needs to record a template and then send the template to the server 102 for saving.

[0082] Figure 5 FIG. 1 shows a flow chart of a recording template according to an embodiment of the present invention. Figure 5 As shown, in one embodiment, method 300 also includes a method for recording a template, and the method for recording a template includes the following steps S350 to S390.

[0083] In step S350 , in response to the user's request to record a template on the Chrome browser, the field name and field type newly added by the user are saved.

[0084] In step S350, when the user clicks the record template button in the web page element search plug-in, the template recording process begins. The user needs to set the newly added field name and field type, click the save button, and then proceed to the next step.

[0085] Step S360 responds to the user's positioning mode selection request on the Chrome browser page and determines the positioning mode selected by the user, which includes absolute positioning and relative positioning. After the user selects the positioning mode of the target field, the next step is entered.

[0086] Step S370 responds to the user's sample collection request on the Chrome browser page, saves the element selected by the user on the current page as a sample, and then proceeds to the next step.

[0087] In step S370, the user needs to collect samples and use the mouse to follow during sample collection. The user places the mouse cursor on the current web page. During the movement of the mouse, the area where the cursor is located is highlighted. For example, if the element in the current web page is text, then the entire line of text where the cursor is located is highlighted.

[0088] In one embodiment, the current web page is a static page. The user can click the mouse to select a field therein, and then click the confirmation button. The selected field is saved as a sample, and the Xpath of the sample is automatically generated. This Xpath will be used as the location of the sample.

[0089] In one embodiment, when the target element to be collected cannot be selected using the mouse, the "add margin" function can be used. Adding margins can change the margin size of the source website, creating a gap between the "crowded elements" so that the "crowded" elements can be selected using the mouse. After the selection is completed, the "restore margins" function can be used to restore the style of the source website.

[0090] In one embodiment, if the current web page is dynamic, such as one containing dynamic data or a dynamic table, a field on the page that does not change with the dynamic data can be found. This field is used as a reference node, and nodes whose relative positions relative to this reference node meet preset conditions are used as containers. Absolute positioning is first used to locate this reference node. A keyword is then entered into the container, and the target element is automatically found using the keyword, a corresponding XPath is generated, and the target element is highlighted.

[0091] During the sample collection process described above, users only need to use their mouse to collect samples. The entire process is visually intuitive, and multiple selectors can be used without analyzing the website's code structure, significantly reducing development effort and improving efficiency. By collecting samples in multiple ways, a fallback solution can be used if a selector fails, improving the accuracy of target element capture.

[0092] Step S380: locate the sample selected by the user according to the positioning method selected by the user, and use the process of locating the sample selected by the user as the extraction rule of the target field.

[0093] In one embodiment, when a template is recorded using relative positioning, the sample needs to be positioned after the user selects the sample. The process of generating the Xpath of the sample value relative to the keyword node is shown in steps S381 to S386.

[0094] In step S381, the position of the keyword is determined based on the container and the keyword: first use text() through the descendant selector of the parent container to find the keyword position (for example, which API to use, how to generate the "price" field). If the keyword position is not found, use string(.) to try to find the keyword. After finding the keyword, find the nearest keyword position as the keyword node based on the depth of the keyword. For example, the user wants to find the price value in the current page, the keyword is "price", and then proceed to the next step.

[0095] In step S382, the keyword node is searched for a sample value. For example, in "price 100 yuan", 100 is the sample value. If a sample value is found, the current query method is used to generate an XPath. For example, if a sample value is found under the keyword node, the XPath of the sample relative to the keyword is:

[0096] / self::*[contains(text(),'sampleValue')][1]

[0097] Then proceed to the next step.

[0098] In step S383, if the sample value is not found in step S382, the sibling node is searched backward from the keyword node. If it is found, the current query method is used as the basis, and the Xpath of the sample value relative to the sibling node is generated. This is used as the Xpath of the sample value relative to the keyword node, and then the next step is entered.

[0099] In step S384, if the sample value is not found in step S383, the sibling node is searched forward from the keyword node. If it is found, the current query method is used as the basis, and the Xpath of the sample value relative to the sibling node is generated. This is used as the Xpath of the sample value relative to the keyword node, and then the next step is entered.

[0100] In step S385, if the sample value is not found in step S384, the descendant nodes are queried from the keyword node downward. If the sample value is found, the current query method is used as the basis, and the Xpath of the sample value relative to the descendant node is generated. This is used as the Xpath of the sample value relative to the keyword node, and then the next step is entered.

[0101] In step S386, the final position of the sample value is obtained by combining the container position, the keyword, and the Xpath of the sample value relative to the keyword node. Based on this position, the user can directly preview the extracted data through the interface.

[0102] The relative positioning method combines absolute positioning with keyword search, solving the problem that some dynamic data is difficult to capture.

[0103] In step S390 , in response to the user's request to save the template on the Chrome browser, the recorded template is saved, and the saved template and the template's URL regular expression are sent to the server 102 .

[0104] In one embodiment, the method 300 further includes step S3100.

[0105] In step S3100, the rule for cleaning web page elements set by the user is used as the search expression of the target field, and the search expression is recorded in the recorded template.

[0106] During template recording, after locating the sample position, if the previewed sample data contains too many characters, for example, the user wants to extract the element "100", but the previewed sample data is "100 yuan", in this case, the user can set a price regularization to clean the located data, such as extracting the price value from the extracted string.

[0107] Correspondingly, step S340 may also clean the found target field according to the search expression of the target field in the template, and use the cleaned field as the final search result of the web page element.

[0108] In one embodiment, when recording a template, for a table type without keywords, the entire table element can be selected as a sample, and then the required field values ​​can be extracted through a custom regular expression.

[0109] In one embodiment, after the template is recorded, the page is refreshed to verify the template. If the marked element is consistent with the target element that the user is looking for, it means that the selector is valid, and then it can be tested and verified on other pages of the same type.

[0110] The web page element search method of the present application can be used for general crawlers in background applications, accessing corresponding sites according to the sites in the database, and then automatically completing the collection of required fields according to the template.

[0111] The embodiment of the present invention further provides a web page element search device 400, which can execute the processing steps of the web page element search method 300 described above. Figure 6 The web page element search device 400 is described below.

[0112] like Figure 6As shown, the web page element search device 400 includes a template acquisition unit 410 , an absolute positioning extraction unit 420 , a relative positioning extraction unit 430 and a display unit 440 .

[0113] In response to the user's request to search for web page elements on the Chrome browser, the template acquisition unit 410 sends the domain name and URL of the current site to the server, and obtains the template returned by the server that matches the domain name and URL of the current site. The template records the target field and the extraction rules of the target field.

[0114] The absolute positioning extraction unit 420 is adapted to extract the target field in the current page using one of an Xpath selector, a CSS selector, and an ID selector when the extraction rule of the target field is absolute positioning.

[0115] The relative positioning extraction unit 430 is adapted to extract a target container in the current page and a target field in the target container when the extraction rule of the target field is relative positioning. The target container represents an element whose relative position to a preset node satisfies a preset condition.

[0116] The display unit 440 is adapted to use the extracted target field as a search result of a web page element.

[0117] In one embodiment, the absolute position extraction unit 420 further includes:

[0118] Xpath extraction subunit, suitable for extracting target fields in the current page using Xpath selector;

[0119] a CSS extraction subunit adapted to extract the target field from the current page using a CSS selector when the extraction of the target field from the current page using an Xpath selector fails; and

[0120] The ID extraction subunit is adapted to extract the target field in the current page by using the ID selector when the target field in the current page fails to be extracted by using the CSS selector.

[0121] In one embodiment, the web page element search device 400 further includes:

[0122] A new field unit is adapted to save the field name and field type added by the user in response to a user's request to record a template on a Chrome browser;

[0123] A positioning selection unit, adapted to respond to a positioning mode selection request from a user on a Chrome browser page and determine a positioning mode selected by the user, the positioning modes including absolute positioning and relative positioning;

[0124] A sample collection unit, adapted to respond to a sample collection request from a user on a Chrome browser page and save an element selected by the user on the current page as a sample;

[0125] a sample locating unit adapted to locate the sample selected by the user according to the locating method selected by the user, and use the process of locating the sample selected by the user as an extraction rule for the target field; and

[0126] The template saving unit is adapted to save the recorded template in response to a user's request to save the template on the Chrome browser, and send the saved template and the template's URL regular expression to the server.

[0127] In one embodiment, the web page element search device 400 further includes:

[0128] The cleaning setting unit is adapted to use the rules set by the user for cleaning the web page elements as the search expression of the target field.

[0129] In one embodiment, the template further records a search expression for the target field, and the presentation unit further includes:

[0130] a cleaning subunit adapted to clean the found target field according to the search expression of the target field in the template; and

[0131] The display subunit is suitable for using the cleaned fields as the final search results of the web page elements.

[0132] The web page element search device 400 of the embodiment of the present invention has the same principle as the web page element search method 300 described above, and can achieve the technical effect of the web page element search method 300, which will not be described in detail here.

[0133] An embodiment of the present invention further provides a readable storage medium storing program instructions. When the program instructions are read and executed by a computing device, the computing device executes the web page element search method 300 described above.

[0134] The various techniques described herein may be implemented in conjunction with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may be implemented in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes an apparatus for practicing the present invention.

[0135] When program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; and the processor is configured to execute the automatic verification method of the present invention according to the instructions in the program code stored in the memory.

[0136] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media store information such as computer-readable instructions, data structures, program modules, or other data. Communication media typically embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery medium. Combinations of any of the above are also included within the scope of readable media.

[0137] In the description provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems may also be used in conjunction with the examples of the present invention. Based on the above description, it is apparent that the structure required for constructing such systems is contemplated. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​may be utilized to implement the present invention described herein, and the description of specific languages ​​described above is provided for the purpose of disclosing preferred embodiments of the present invention.

[0138] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0139] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0140] Those skilled in the art will appreciate that the modules, units, or components of the devices in the examples disclosed herein may be arranged in the device described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or further divided into multiple submodules.

[0141] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0142] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0143] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions described. Thus, a processor having the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. Furthermore, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0144] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and are not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0145] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative and not restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A method for searching for web page elements, the method comprising the steps of: In response to a user's request to search for web page elements on a Chrome browser, the domain name and URL of the current site are sent to a server, and a template returned by the server that matches the domain name and URL of the current site is obtained. The template records a target field and an extraction rule for the target field. The template recording process includes: in response to a user's request to record a template on a Chrome browser, saving the field name and field type added by the user; in response to a user's request to select a positioning method on a Chrome browser page, determining the positioning method selected by the user, the positioning method including absolute positioning and relative positioning; if the page is a static page, absolute positioning is selected; if the page is a dynamic page, dynamic positioning is selected; in response to a user's request to collect samples on a Chrome browser page, saving the element selected by the user on the current page as a sample; positioning the sample selected by the user according to the positioning method selected by the user; and using the process of positioning the sample selected by the user as an extraction rule for the target field, wherein, When recording a template, if the target element to be captured cannot be selected with the mouse, use the Add Margin function to add margins to change the margin size of the source website, creating space between crowded elements so that the crowded elements can be selected with the mouse. After the selection is completed, restore the margins to restore the source website style. When the extraction rule of the target field is absolute positioning, extract the target field in the current page using one of an Xpath selector, a CSS selector, and an ID selector; When the extraction rule of the target field is relative positioning, a target container is extracted from the current page, and a target field is extracted from the target container, where the target container represents an element whose relative position to a preset node satisfies a preset condition; as well as The extracted target field is used as the search result of the web page element.

2. The method according to claim 1, wherein The step of extracting the target field from the current page using one of the Xpath selector, the CSS selector, and the ID selector includes: Use Xpath selector to extract target fields in the current page; When the target field fails to be extracted from the current page using the XPath selector, the target field is extracted from the current page using the CSS selector; and When the target field fails to be extracted from the current page using the CSS selector, the target field is extracted from the current page using the ID selector.

3. The method according to claim 1, wherein The step of extracting the target container from the current page includes: The target container is extracted from the current page according to the position of the target container recorded in the template relative to the preset node.

4. The method according to any one of claims 1 to 3, wherein The method further comprises the steps of: In response to a user's request to record a template on a Chrome browser, the user's newly added field name and field type are saved; In response to a positioning mode selection request from a user on a Chrome browser page, determining a positioning mode selected by the user, the positioning mode including absolute positioning and relative positioning; In response to a sample collection request from a user on a Chrome browser page, the user saves the element selected on the current page as a sample; Positioning the sample selected by the user according to the positioning method selected by the user, and using the process of positioning the sample selected by the user as the extraction rule of the target field; and In response to the user's request to save the template on the Chrome browser, the recorded template is saved, and the saved template and the template's URL are sent to the server.

5. The method according to claim 4, wherein: The method further comprises the steps of: The rules set by the user for cleaning the web page elements are used as the search expression of the target field.

6. The method according to claim 5, wherein: The template further records a search expression for a target field, and the step of using the extracted target field as a search result for the web page element further includes: Cleaning the found target field according to the search expression of the target field in the template; and The cleaned fields are used as the final search results of the web page elements.

7. A web page element search device, comprising: A template acquisition unit adapted to respond to a user's request to search for web page elements on a Chrome browser, send the domain name and URL of the current site to a server, and obtain a template returned by the server that matches the domain name and URL of the current site, wherein the template records a target field and an extraction rule for the target field. The template recording process includes: responding to the user's request to record the template on the Chrome browser, saving the field name and field type added by the user; responding to the user's request to select a positioning method on the Chrome browser page, determining the positioning method selected by the user, wherein the positioning methods include absolute positioning and relative positioning; responding to the user's request to collect a sample on the Chrome browser page, saving the element selected by the user on the current page as a sample, locating the sample selected by the user according to the positioning method selected by the user, and using the process of locating the sample selected by the user as the extraction rule for the target field. When recording the template, if the target element to be collected cannot be selected using a mouse, a margin adding function is used to add margins to change the margin size of the source website, thereby creating a gap between the crowded elements so that the crowded elements can be selected using a mouse, and restoring the margins after the selection is completed to restore the style of the source website; an absolute positioning extraction unit, adapted to extract the target field in the current page using one of an Xpath selector, a CSS selector, and an ID selector when the extraction rule of the target field is absolute positioning; a relative positioning extraction unit adapted to extract a target container from the current page and a target field from the target container when the extraction rule of the target field is relative positioning, wherein the target container represents an element whose relative position to a preset node satisfies a preset condition; and The display unit is adapted to use the extracted target field as a search result of the web page element.

8. The apparatus of claim 7, further comprising: A new field unit is adapted to save the field name and field type added by the user in response to a user's request to record a template on a Chrome browser; a positioning selection unit, adapted to determine a positioning mode selected by the user in response to a positioning mode selection request on a Chrome browser page, wherein the positioning mode includes absolute positioning and relative positioning; A sample collection unit, adapted to respond to a sample collection request from a user on a Chrome browser page and save an element selected by the user on the current page as a sample; a sample locating unit adapted to locate the sample selected by the user according to the locating method selected by the user, and use the process of locating the sample selected by the user as an extraction rule for the target field; as well as The template saving unit is adapted to save the recorded template in response to a user's request to save the template on the Chrome browser, and to send the saved template and the template's URL to the server.

9. A computing device comprising: at least one processor and memory storing program instructions; When the program instructions are read and executed by the processor, the computing device is caused to execute the web page element search method according to any one of claims 1 to 6.

10. A readable storage medium storing program instructions, wherein when the program instructions are read and executed by a computing device, the computing device executes the web page element search method according to any one of claims 1 to 6.

Images