Method and apparatus for page data collection

Abstract

This invention discloses a method and apparatus for page data acquisition, relating to the field of computer technology. One specific implementation of the method includes: rendering the target page to be acquired, and selecting candidate regions on the rendered target page according to a target field to generate a page screenshot; based on the page screenshot and the document object model structure, identifying semantic information of the candidate regions through a first intelligent agent; based on the semantic information, determining the target module where the target field is located through a second intelligent agent; determining whether the content of the target module meets the acquisition conditions through a third intelligent agent; and, if the acquisition conditions are met, generating an acquisition path configuration file corresponding to the target field. This implementation can support data recognition of pages with lengthy content or complex webpage structures, as well as data acquisition from website pages that have not been previously adapted for recognition, and can perform real-time updates of the acquisition path configuration file, thereby improving the efficiency, flexibility, and accuracy of page data acquisition.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for acquiring page data. Background Technology

[0002] With the explosive growth of online information, more and more scenarios require web crawlers to acquire relevant information to assist strategic decision-making, enrich material databases, or assist in training large models. Currently, the mainstream technical solutions for page data collection are mainly of two types: one is to convert page content into Markdown (a lightweight markup language for plain text) format using a small language model (such as the ReaderLM series models) and then identify and collect the required data; the other is to manually analyze the document object model (DOM) structure of the target webpage, locate the required data, generate a data collection path configuration file, and then let the program complete the automated data collection.

[0003] However, small language models are only suitable for data recognition and collection on pages with short content and simple structure. They perform poorly on pages with long content or complex structure, and cannot accurately identify and collect data. The method of manually configuring the data collection path and then collecting data cannot support website pages that have not been adapted for recognition. It lacks flexibility and generalization, and there is also the problem of untimely updates, which greatly affects the efficiency of page data collection. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for page data acquisition, which can render a target page to display candidate areas of selected target fields, and generate a collection path configuration file corresponding to the target field to be acquired by combining multiple intelligent agents, so as to collect data of the target field based on the collection path configuration file. This realizes the generation of collection path configuration files corresponding to target fields by fusing visual features and semantic features, and can support data recognition of pages with long content or complex web page structures, as well as data acquisition of website pages that have not been adapted for recognition. Moreover, it can perform real-time updates of the collection path configuration file, thereby improving the efficiency, flexibility and accuracy of page data acquisition.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for page data acquisition is provided, comprising: The target page to be collected is rendered, and a candidate area including the target field is selected on the rendered target page according to the target field to be collected, and a screenshot of the page including the candidate area is generated. Based on the page screenshot and the document object model structure of the target page, the first intelligent agent identifies the semantic information of the candidate region; Based on the semantic information, the target module where the target field is located is determined by the second intelligent agent; The third-party intelligent agent determines whether the content of the target module meets the preset collection conditions. If the content of the target module meets the preset collection conditions, a collection path configuration file corresponding to the target field is generated according to the target module, so as to collect data of the target field based on the collection path configuration file.

[0006] Optionally, before rendering the target page to be collected, the method further includes: performing similarity matching between the target page and a preset collection path configuration library, wherein the collection path configuration library stores the mapping relationship between historical pages and their corresponding target fields and collection path configuration files; and determining that the similarity matching fails.

[0007] Optionally, the method further includes: if the similarity match is successful, extracting the data of the target field from the target page based on the matched collection path configuration file; after generating the collection path configuration file corresponding to the target field according to the target module, the method further includes: storing the mapping relationship between the target field and the corresponding collection path configuration file into the collection path configuration library.

[0008] Optionally, the target page is matched with a preset collection path configuration library for similarity, including: calculating the comprehensive similarity between the page structure and style features of the target page and the page structure and style features of historical pages in the collection path configuration library for similarity matching; wherein, if the comprehensive similarity exceeds a preset threshold, the match is determined to be successful.

[0009] Optionally, based on the page screenshot and the document object model structure of the target page, the semantic information of the candidate region is identified by a first intelligent agent, including: inputting the page screenshot and the document object model structure of the target page into the first intelligent agent, so as to identify the location information and semantic description of the candidate region by the first intelligent agent, and generating the semantic information of the candidate region based on the location information and semantic description of the candidate region.

[0010] Optionally, based on the semantic information, determining the target module where the target field is located by the second intelligent agent includes: inputting the semantic information into the second intelligent agent, so that the second intelligent agent generates module positioning information based on the location information of the candidate region, the semantic description of the candidate region and the target field, and determines the target module where the target field is located based on the module positioning information, wherein the module positioning information indicates the node where the target field is located in the document object model structure.

[0011] Optionally, determining whether the content of the target module meets the preset collection conditions through a third intelligent agent includes: inputting the node content of the document object model corresponding to the module location information into the third intelligent agent, so as to determine whether the content of the target module meets the preset collection conditions through the third intelligent agent; wherein, if the content of the target module is only related to the target field, it is determined that the preset collection conditions are met.

[0012] Optionally, selecting a candidate region including the target field on the rendered target page according to the target field to be collected includes: selecting a candidate region including the target field on the rendered target page using the node where the target field is located in the document object model structure of the target page as the root node; the method further includes: if the content of the target module does not meet the preset collection conditions, iteratively identifying the node of the document object model corresponding to the target module as the new root node until the collection conditions are met.

[0013] Optionally, iterative identification is performed using the target module as the new root node, including: taking the node of the document object model corresponding to the target module as the new root node; and repeating the steps of candidate region selection, semantic information identification, target module determination, and collection condition judgment under the new root node.

[0014] Optionally, the method further includes: during the iterative identification process, in response to the fact that the depth of the document object model subtree corresponding to the new root node is less than a preset number of layers, directly inputting the web page fragment corresponding to the new root node into the fourth agent to obtain a candidate collection path configuration file; verifying whether the content corresponding to the candidate collection path configuration file meets the collection conditions through the third agent; if the content corresponding to the candidate collection path configuration file meets the preset collection conditions, collecting target field data according to the candidate collection path configuration file; if the content corresponding to the candidate collection path configuration file does not meet the preset collection conditions, continuing to perform the iterative identification.

[0015] According to another aspect of the present invention, a page data acquisition apparatus is provided, comprising: The candidate region selection module is used to render the target page to be collected, and select candidate regions including the target fields on the rendered target page according to the target fields to be collected, and generate a page screenshot including the candidate regions. The semantic information recognition module is used to identify the semantic information of the candidate region through a first intelligent agent based on the page screenshot and the document object model structure of the target page; The target module determination module is used to determine the target module where the target field is located by a second intelligent agent based on the semantic information. The data acquisition condition judgment module is used to determine, through a third intelligent agent, whether the content of the target module meets the preset data acquisition conditions; The data acquisition path configuration module is used to generate a data acquisition path configuration file corresponding to the target field based on the target module when the content of the target module meets the preset data acquisition conditions, so as to collect target field data based on the data acquisition path configuration file.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the page data acquisition method provided in the embodiments of the present invention.

[0017] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the page data acquisition method provided in the embodiments of the present invention.

[0018] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the page data acquisition method provided in the embodiments of the present invention.

[0019] One embodiment of the above invention has the following advantages or beneficial effects: by rendering the target page to be collected, and selecting candidate areas including the target fields on the rendered target page according to the target fields to be collected, a page screenshot including the candidate areas is generated; based on the page screenshot and the document object model structure of the target page, the semantic information of the candidate areas is identified by a first intelligent agent; based on the semantic information, the target module where the target field is located is determined by a second intelligent agent; a third intelligent agent determines whether the content of the target module meets the preset collection conditions; if the content of the target module meets the preset collection conditions, a collection path configuration file corresponding to the target field is generated according to the target module, and the data collection of the target field is carried out based on the collection path configuration file. By rendering the target page, the candidate areas of the selected target fields can be displayed, and the collection path configuration file corresponding to the target fields to be collected is generated by combining multiple intelligent agents, and the data collection of the target fields is carried out based on the collection path configuration file. This realizes the fusion of visual features and semantic features to generate the collection path configuration file corresponding to the target fields, which can support the data recognition of pages with long content or complex web page structures and the data collection of website pages that have not been adapted for recognition, and can perform real-time updates of the collection path configuration file, thereby improving the efficiency, flexibility and accuracy of page data collection.

[0020] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0021] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram illustrating the main steps of a page data acquisition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a page data acquisition process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the main modules of a page data acquisition device according to an embodiment of the present invention; Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0022] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0024] To address the technical problems existing in the prior art, this invention provides a method and apparatus for page data acquisition. By rendering the target page, candidate regions of selected target fields can be displayed. Multiple intelligent agents are combined to generate acquisition path configuration files (e.g., Extensible Markup Language (XPath), used to locate specific nodes or node sets in an XML document tree structure) corresponding to the target fields to be acquired. Data acquisition of target fields is performed based on the acquisition path configuration file. This achieves the fusion of visual and semantic features to generate acquisition path configuration files corresponding to target fields. It can support data recognition of pages with lengthy content or complex webpage structures, as well as data acquisition from website pages that have not been adapted for recognition. Furthermore, the acquisition path configuration file can be updated in real time, thereby improving the efficiency, flexibility, and accuracy of page data acquisition.

[0025] Figure 1 This is a schematic diagram illustrating the main steps of a page data acquisition method according to an embodiment of the present invention. Figure 1 As shown, the page data acquisition method of this embodiment mainly includes the following steps S101 to S105.

[0026] Step S101: Render the target page to be collected, and select candidate areas including the target fields on the rendered target page according to the target fields to be collected, generating a screenshot of the page including the candidate areas. The target page to be collected is, for example, an HTML (Hypertext Markup Language) page determined by a given page URL. Rendering the target page can be done on the browser client, selecting a commonly used rendering method. The target fields to be collected are, for example, pre-defined according to business needs and included in the page data collection request.

[0027] According to one embodiment of the present invention, selecting candidate regions including the target field on the rendered target page based on the target field to be collected can specifically include: using the node where the target field is located in the document object model (DOM) structure of the target page as the root node, selecting candidate regions including the target field on the rendered target page. Assuming the target field to be collected is an article title field, the node where this target field is located in the document object model (DOM) structure of the target page is the DOM container where the article title is located. When selecting candidate regions including the target field on the rendered target page based on the target field to be collected, the DOM container where the article title is located can be used as the root node, selecting candidate regions including the target field on the rendered target page. The selected candidate regions can be one or more. After selecting the candidate regions, taking a screenshot of the rendered target page will yield a screenshot of the page including the candidate regions.

[0028] According to one embodiment of the present invention, before rendering the target page to be collected, the method for collecting page data may further include: performing similarity matching between the target page and a preset collection path configuration library, wherein the collection path configuration library stores the mapping relationship between historical pages and their corresponding target fields and collection path configuration files; and determining that the similarity matching fails. Furthermore, the method for collecting page data may further include: if the similarity matching is successful, extracting data of the target fields from the target page based on the matched collection path configuration file. In embodiments of the present invention, a collection path configuration library can be pre-established to store the mapping relationship between historical pages and their corresponding target fields and collection path configuration files. Each time data collection is performed on a target page, the collection path configuration library can be searched for a directly usable collection path configuration file, and the target field data can be collected based on the directly usable collection path configuration file. In this way, a usable collection path configuration file can be quickly matched, thereby improving data collection efficiency.

[0029] According to one embodiment of the present invention, similarity matching between a target page and a preset collection path configuration library may specifically include: calculating the comprehensive similarity between the page structure and style features of the target page and the page structure and style features of historical pages in the collection path configuration library for similarity matching; wherein, if the comprehensive similarity exceeds a preset threshold, a successful match is determined. Specifically, when performing similarity matching between the target page and the collection path configuration library, the HTML structure and style features of the target page may be compared with the HTML structure and style features of historical pages in the collection path configuration library. If the similarity exceeds a set threshold (e.g., 0.6), a successful match is determined, and the collection path configuration file corresponding to the target field of the successfully matched historical page can be directly used to collect data of the target field from the target page. If a match is determined to be unsuccessful, steps S101 to S105 can be executed to generate a collection path configuration file for the target field corresponding to the target page.

[0030] Furthermore, after generating the collection path configuration file corresponding to the target field based on the target module, the process can also include storing the mapping relationship between the target field and the corresponding collection path configuration file in the collection path configuration library. This enriches the collection path configuration library, allowing the matching collection path configuration file to be directly found in the library the next time the target field is extracted. This enables real-time updates of the collection path configuration file, thereby improving the efficiency, flexibility, and accuracy of page data collection.

[0031] Step S102: Based on the page screenshot and the document object model structure of the target page, the semantic information of the candidate regions is identified by a first intelligent agent. In embodiments of the present invention, the first intelligent agent is, for example, a large language model, which can perform image-text recognition to identify the semantic information of the candidate regions.

[0032] According to one embodiment of the present invention, based on a page screenshot and the document object model (DOM) structure of the target page, the semantic information of candidate regions is identified by a first intelligent agent. Specifically, this may include: inputting the page screenshot and the DOM structure of the target page into the first intelligent agent, so that the first intelligent agent identifies the location information and semantic description of the candidate regions, and generates semantic information of the candidate regions based on the location information and semantic description. Specifically, prompt words can be generated based on the page screenshot and the DOM structure of the target page using a pre-set prompt word template. Then, the first intelligent agent can identify the location information and semantic description of each candidate region based on the prompt words, thereby generating the semantic information of each candidate region. Since the context of the first intelligent agent includes the page screenshot and the DOM structure of the target page, it can accurately determine which DOM tree node the selected candidate region is, and return the content included in each candidate region, such as article title, body text, author, etc.

[0033] Step S103: Based on semantic information, the target module containing the target field is determined by the second intelligent agent. In embodiments of the present invention, the second intelligent agent is, for example, a large language model, which can perform semantic recognition to determine whether the selected candidate region is correct and whether the selected candidate region includes the target field (e.g., article title) based on the semantic information of the candidate region.

[0034] According to one embodiment of the present invention, determining the target module where a target field is located through a second intelligent agent based on semantic information may specifically include: inputting semantic information into the second intelligent agent, which then generates module location information based on the location information of candidate regions, the semantic description of candidate regions, and the target field, and determines the target module where the target field is located based on the module location information. The module location information indicates the node where the target field is located in the document object model structure. Specifically, prompt words can be generated using a pre-set prompt word template based on the location information of candidate regions, the semantic description of candidate regions, and the target field to be collected (e.g., article title). Then, the second intelligent agent can generate module location information indicating the node where the target field is located in the document object model structure based on the prompt words, thereby determining the target module where the target field is located.

[0035] Step S104: The third agent determines whether the content of the target module meets the preset collection conditions. In an embodiment of the present invention, the third agent is, for example, a large language model, which can determine whether the target field can be collected from the target module based on its content. The collection condition, for example, is that the content of the target module only includes data of the target field and excludes data of other fields. This improves the accuracy of data collection.

[0036] According to one embodiment of the present invention, determining whether the content of a target module meets preset collection conditions through a third intelligent agent may specifically include: inputting the node content of the document object model corresponding to the module location information into the third intelligent agent, so as to determine whether the content of the target module meets the preset collection conditions; wherein, if the content of the target module is only related to the target field, it is determined that the preset collection conditions are met. The content of the target module is, for example, the content of a node in the DOM tree corresponding to the target module, which can be obtained from the DOM tree based on the module location information of the target module.

[0037] In practice, prompts are generated using pre-set prompt templates based on the target module's location information and the Document Object Model (DOM) tree of the target page. A third-party intelligent agent then determines whether the target module's content meets the preset collection conditions. For example, if the target field is the article title, but the target module's content includes the author, publication time, etc., it indicates the selected area for the target module is too large, and the preset collection conditions are not met. If the target module's content only contains the article title and no other fields, then the target module meets the collection conditions.

[0038] Step S105: If the content of the target module meets the preset acquisition conditions, a acquisition path configuration file corresponding to the target field is generated based on the target module, and data acquisition of the target field is performed based on the acquisition path configuration file. When the content of the target module meets the preset acquisition conditions, the acquisition path is determined based on the location information of the candidate area corresponding to the target module, module positioning information, etc., to generate the acquisition path configuration file corresponding to the target field.

[0039] According to one embodiment of the present invention, selecting candidate regions including the target field on the rendered target page based on the target field to be collected can specifically include: using the node where the target field is located in the Document Object Model (DOM) structure of the target page as the root node, selecting candidate regions including the target field on the rendered target page. Specifically, using the node where the target field is located in the DOM structure of the target page as the root node, node searching is performed layer by layer until all nodes including the target field are selected, so as to select candidate regions including the target field on the rendered target page.

[0040] According to an embodiment of the present invention, the method for collecting page data may further include: when the content of the target module does not meet the preset collection conditions, iterative identification is performed using the node of the document object model corresponding to the target module as the new root node until the collection conditions are met. When the content of the target module includes other fields besides the target field, iterative identification is performed using the node of the document object model (DOM) corresponding to the target module as the new root node until a target module containing only the target field is selected.

[0041] According to one embodiment of the present invention, iterative identification using the target module as the new root node may specifically include: taking the node of the document object model corresponding to the target module as the new root node; and repeating the steps of candidate region selection, semantic information identification, target module determination, and collection condition judgment under the new root node. Specifically, taking the node of the document object model corresponding to the target module as the new root node, and then selecting candidate regions including the target field on the rendered target page according to the target field to be collected, generating a page screenshot including the candidate regions; based on the page screenshot and the document object model structure of the target page, identifying the semantic information of the candidate regions through a first intelligent agent; based on the semantic information, determining the target module where the target field is located through a second intelligent agent; judging whether the content of the target module meets the preset collection conditions through a third intelligent agent; until it is determined that the content of the target module meets the preset collection conditions.

[0042] According to an embodiment of the present invention, the method for page data acquisition may further include: during iterative identification, in response to the fact that the depth of the DOM subtree corresponding to the new root node is less than a preset number of layers, directly inputting the web page fragment corresponding to the new root node into a fourth agent to obtain a candidate acquisition path configuration file; verifying whether the content corresponding to the candidate acquisition path configuration file meets the acquisition conditions through a third agent; if the content corresponding to the candidate acquisition path configuration file meets the preset acquisition conditions, acquiring target field data according to the candidate acquisition path configuration file; if the content corresponding to the candidate acquisition path configuration file does not meet the preset acquisition conditions, continuing iterative identification. The preset number of layers is, for example, 5 layers. During the process of updating the root node and performing iterative identification, if the depth of the DOM subtree from the root node to the last-level node is less than 5 layers, it indicates that the structure of the web page fragment corresponding to the new root node is simple. In this case, the target field can be directly identified through the large model, and the candidate acquisition path configuration file can be directly obtained by directly inputting the HTML web page fragment corresponding to the new root node into the fourth agent. This improves the efficiency of generating the acquisition path configuration file.

[0043] After obtaining the candidate acquisition path configuration file, a third-party intelligent agent verifies whether the content corresponding to the candidate acquisition path configuration file meets the acquisition conditions. If the content corresponding to the candidate acquisition path configuration file meets the preset acquisition conditions, target field data is acquired according to the candidate acquisition path configuration file. If the content corresponding to the candidate acquisition path configuration file does not meet the preset acquisition conditions, iterative recognition continues.

[0044] Figure 2 This is a schematic diagram of a page data acquisition process according to an embodiment of the present invention. Figure 2 As shown, in one embodiment of the present invention, when collecting page data, the target page to be collected is first obtained, and the target page is matched with a preset collection path configuration library for similarity. If the match is successful, the data of the target field is extracted from the target page based on the matched collection path configuration file; otherwise, the target page to be collected is rendered, and candidate areas are selected on the rendered target page according to the target field to generate a page screenshot including the candidate areas.

[0045] Subsequently, based on the page screenshot and the document object model structure of the target page, the first agent identifies the semantic information of the candidate regions; based on the semantic information, the second agent determines the target module where the target field is located; and the third agent determines whether the content of the target module meets the preset collection conditions. If the preset collection conditions are met, a collection path configuration file corresponding to the target field is generated according to the target module, data collection of the target field is performed based on the collection path configuration file, and the mapping relationship between the target field and the corresponding collection path configuration file is stored in the collection path configuration library; otherwise, the node of the document object model corresponding to the target module is used as the new root node, and iterative identification is performed, repeating the steps of candidate region selection, semantic information identification, target module determination, and collection condition judgment until the content of the target module meets the preset collection conditions.

[0046] Figure 3 This is a schematic diagram of the main modules of a page data acquisition device according to an embodiment of the present invention. Figure 3 As shown, the page data acquisition device 300 of this embodiment mainly includes a candidate region selection module 301, a semantic information recognition module 302, a target module determination module 303, an acquisition condition judgment module 304, and an acquisition path configuration module 305.

[0047] The candidate region selection module 301 is used to render the target page to be collected, and select candidate regions including the target fields on the rendered target page according to the target fields to be collected, and generate a screenshot of the page including the candidate regions. The semantic information recognition module 302 is used to identify the semantic information of candidate regions based on the page screenshot and the document object model structure of the target page through the first intelligent agent; The target module determination module 303 is used to determine the target module where the target field is located by a second intelligent agent based on semantic information. The data acquisition condition judgment module 304 is used to determine whether the content of the target module meets the preset data acquisition conditions through a third intelligent agent; The data acquisition path configuration module 305 is used to generate a data acquisition path configuration file corresponding to the target field based on the target module when the content of the target module meets the preset data acquisition conditions, so as to collect target field data based on the data acquisition path configuration file.

[0048] According to an embodiment of the present invention, the page data acquisition device 300 further includes a configuration library matching module (not shown in the figure), which is used to: match the target page with a preset acquisition path configuration library for similarity before rendering the target page to be acquired, wherein the acquisition path configuration library stores the mapping relationship between historical pages and their corresponding target fields and acquisition path configuration files; and determine that the similarity matching fails.

[0049] According to an embodiment of the present invention, the page data acquisition device 300 further includes a data extraction module (not shown in the figure), used for: extracting data of target fields from the target page based on the matched acquisition path configuration file when the similarity matching is successful; and a configuration library update module (not shown in the figure), used for: storing the mapping relationship between the target field and the corresponding acquisition path configuration file into the acquisition path configuration library after the target module generates the acquisition path configuration file corresponding to the target field.

[0050] According to one embodiment of the present invention, the configuration library matching module (not shown in the figure) can also be used to: calculate the page structure and style features of the target page and the comprehensive similarity between them and the page structure and style features of historical pages in the acquisition path configuration library to perform similarity matching; wherein, if the comprehensive similarity exceeds a preset threshold, it is determined that the match is successful.

[0051] According to an embodiment of the present invention, the semantic information recognition module 302 can be specifically used to: input a screenshot of the page and the document object model structure of the target page into a first intelligent agent, so as to identify the location information and semantic description of the candidate region through the first intelligent agent, and generate the semantic information of the candidate region based on the location information and semantic description of the candidate region.

[0052] According to an embodiment of the present invention, the target module determination module 303 may be specifically used to: input semantic information into a second intelligent agent, so that the second intelligent agent generates module positioning information based on the location information of the candidate region, the semantic description of the candidate region and the target field, and determines the target module where the target field is located based on the module positioning information, wherein the module positioning information indicates the node where the target field is located in the document object model structure.

[0053] According to an embodiment of the present invention, the acquisition condition judgment module 304 can be specifically used to: input the node content of the document object model corresponding to the module positioning information into a third intelligent agent, so as to determine whether the content of the target module meets the preset acquisition conditions; wherein, when the content of the target module is only related to the target field, it is determined that the preset acquisition conditions are met.

[0054] According to one embodiment of the present invention, the candidate region selection module 301 can be specifically used to: select candidate regions including the target field on the rendered target page, using the node where the target field to be collected is located in the document object model structure of the target page as the root node. Furthermore, the page data acquisition device 300 also includes an iterative recognition module (not shown in the figure), used to: when the content of the target module does not meet the preset acquisition conditions, perform iterative recognition using the node of the document object model corresponding to the target module as the new root node until the acquisition conditions are met.

[0055] According to an embodiment of the present invention, the iterative identification module (not shown in the figure) can be specifically used to: take the node of the document object model corresponding to the target module as the new root node; and repeatedly execute the candidate region selection module, the semantic information identification module, the target module determination module, and the collection condition judgment module under the new root node.

[0056] According to one embodiment of the present invention, the iterative identification module (not shown in the figure) can also be used to: during the iterative identification process, in response to the fact that the depth of the document object model subtree corresponding to the new root node is less than a preset number of layers, directly input the web page fragment corresponding to the new root node to the fourth intelligent agent to obtain a candidate acquisition path configuration file; verify whether the content corresponding to the candidate acquisition path configuration file meets the acquisition conditions through the third intelligent agent; if the content corresponding to the candidate acquisition path configuration file meets the preset acquisition conditions, perform target field data acquisition according to the candidate acquisition path configuration file; if the content corresponding to the candidate acquisition path configuration file does not meet the preset acquisition conditions, continue to perform iterative identification.

[0057] According to the technical solution of the present invention, the target page to be collected is rendered, and candidate areas including the target fields are selected on the rendered target page according to the target fields to be collected, generating a page screenshot including the candidate areas; based on the page screenshot and the document object model structure of the target page, the semantic information of the candidate areas is identified by a first intelligent agent; based on the semantic information, the target module where the target field is located is determined by a second intelligent agent; a third intelligent agent determines whether the content of the target module meets the preset collection conditions; if the content of the target module meets the preset collection conditions, a collection path configuration file corresponding to the target field is generated according to the target module, and the data of the target field is collected based on the collection path configuration file. By rendering the target page, the candidate areas of the selected target fields can be displayed, and the collection path configuration file corresponding to the target fields to be collected is generated by combining multiple intelligent agents, and the data of the target fields is collected based on the collection path configuration file. This realizes the fusion of visual features and semantic features to generate the collection path configuration file corresponding to the target fields, which can support the data recognition of pages with long content or complex web page structures and the data collection of website pages that have not been adapted for recognition, and can perform real-time updates of the collection path configuration file, thereby improving the efficiency, flexibility and accuracy of page data collection.

[0058] Figure 4 An exemplary system architecture 400 is shown, to which the page data acquisition method or apparatus of embodiments of the present invention can be applied.

[0059] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0060] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0061] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0062] Server 405 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 401, 402, and 403 (this is just an example). The backend management server can process received page data collection requests and other data by performing tasks such as candidate region selection, semantic information recognition, target module determination, collection condition judgment, and collection path configuration, and then feed back the processing results (such as the generated collection path configuration file corresponding to the target field—this is just an example) to the terminal device.

[0063] It should be noted that the page data acquisition method provided in this embodiment of the invention is generally executed by server 405, and correspondingly, the page data acquisition device is generally located in server 405.

[0064] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0065] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing terminal devices or servers of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0066] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0067] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0068] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0069] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0071] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a candidate region selection module, a semantic information recognition module, a target module determination module, a data acquisition condition judgment module, and a data acquisition path configuration module. The names of these units or modules do not necessarily limit the specific unit or module itself. For example, the semantic information recognition module can also be described as "a module for recognizing the semantic information of candidate regions through a first intelligent agent based on a page screenshot and the document object model structure of the target page."

[0072] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: rendering a target page to be collected, and selecting candidate regions including the target fields on the rendered target page according to the target fields to be collected, generating a page screenshot including the candidate regions; identifying semantic information of the candidate regions through a first intelligent agent based on the page screenshot and the document object model structure of the target page; determining the target module where the target field is located through a second intelligent agent based on the semantic information; determining whether the content of the target module meets preset collection conditions through a third intelligent agent; and, if the content of the target module meets the preset collection conditions, generating a collection path configuration file corresponding to the target field according to the target module, so as to collect data of the target field based on the collection path configuration file.

[0073] According to the technical solution of the present invention, the target page to be collected is rendered, and candidate areas including the target fields are selected on the rendered target page according to the target fields to be collected, generating a page screenshot including the candidate areas; based on the page screenshot and the document object model structure of the target page, the semantic information of the candidate areas is identified by a first intelligent agent; based on the semantic information, the target module where the target field is located is determined by a second intelligent agent; a third intelligent agent determines whether the content of the target module meets the preset collection conditions; if the content of the target module meets the preset collection conditions, a collection path configuration file corresponding to the target field is generated according to the target module, and the data of the target field is collected based on the collection path configuration file. By rendering the target page, the candidate areas of the selected target fields can be displayed, and the collection path configuration file corresponding to the target fields to be collected is generated by combining multiple intelligent agents, and the data of the target fields is collected based on the collection path configuration file. This realizes the fusion of visual features and semantic features to generate the collection path configuration file corresponding to the target fields, which can support the data recognition of pages with long content or complex web page structures and the data collection of website pages that have not been adapted for recognition, and can perform real-time updates of the collection path configuration file, thereby improving the efficiency, flexibility and accuracy of page data collection.

[0074] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for collecting page data, characterized in that, include: The target page to be collected is rendered, and a candidate area including the target field is selected on the rendered target page according to the target field to be collected, and a screenshot of the page including the candidate area is generated. Based on the page screenshot and the document object model structure of the target page, the first intelligent agent identifies the semantic information of the candidate region; Based on the semantic information, the target module where the target field is located is determined by the second intelligent agent; The third-party intelligent agent determines whether the content of the target module meets the preset collection conditions; If the content of the target module meets the preset collection conditions, a collection path configuration file corresponding to the target field is generated according to the target module, so as to collect target field data based on the collection path configuration file.

2. The method according to claim 1, characterized in that, Before rendering the target page to be collected, it also includes: The target page is matched with a preset collection path configuration library for similarity. The collection path configuration library stores the mapping relationship between historical pages and their corresponding target fields and collection path configuration files. And, it was determined that the similarity match failed.

3. The method according to claim 2, characterized in that, The method further includes: If a similarity match is successful, the data of the target field is extracted from the target page based on the matched collection path configuration file; After generating the acquisition path configuration file corresponding to the target field based on the target module, the process also includes: The mapping relationship between the target field and the corresponding acquisition path configuration file is stored in the acquisition path configuration library.

4. The method according to claim 2 or 3, characterized in that, The target page is matched with a preset collection path configuration library based on similarity, including: Calculate the comprehensive similarity between the page structure and style features of the target page and the page structure and style features of historical pages in the collection path configuration library for similarity matching; If the overall similarity exceeds a preset threshold, the match is considered successful.

5. The method according to claim 1, characterized in that, Based on the page screenshot and the document object model structure of the target page, the first intelligent agent identifies the semantic information of the candidate regions, including: The screenshot of the page and the document object model structure of the target page are input into the first agent, so that the first agent can identify the location information and semantic description of the candidate region, and generate the semantic information of the candidate region based on the location information and semantic description of the candidate region.

6. The method according to claim 5, characterized in that, Based on the semantic information, the second agent determines the target module where the target field is located, including: The semantic information is input into the second agent, which generates module location information based on the location information of the candidate region, the semantic description of the candidate region, and the target field. The second agent then determines the target module where the target field is located based on the module location information, which indicates the node where the target field is located in the document object model structure.

7. The method according to claim 6, characterized in that, The third-party intelligent agent determines whether the content of the target module meets the preset acquisition conditions, including: The node content of the document object model corresponding to the module location information is input into the third intelligent agent, so that the third intelligent agent can determine whether the content of the target module meets the preset collection conditions. Specifically, if the content of the target module is only related to the target field, it is determined that the preset collection conditions are met.

8. The method according to claim 1 or 7, characterized in that, Based on the target field to be collected, select a candidate region on the rendered target page that includes the target field, including: Using the node where the target field to be collected is located in the document object model structure of the target page as the root node, select the candidate area including the target field on the rendered target page; The method further includes: If the content of the target module does not meet the preset collection conditions, the node of the document object model corresponding to the target module is used as the new root node for iterative identification until the collection conditions are met.

9. The method according to claim 8, characterized in that, Iterative identification is performed using the target module as the new root node, including: The node of the document object model corresponding to the target module is taken as the new root node; Under the new root node, repeat the steps of candidate region selection, semantic information recognition, target module determination, and collection condition judgment.

10. The method according to claim 9, characterized in that, The method further includes: During the iterative identification process, in response to the fact that the depth of the document object model subtree corresponding to the new root node is less than the preset number of layers, the web page fragment corresponding to the new root node is directly input into the fourth intelligent agent to obtain the candidate collection path configuration file. The third intelligent agent verifies whether the content corresponding to the candidate acquisition path configuration file meets the acquisition conditions. If the content corresponding to the candidate acquisition path configuration file meets the preset acquisition conditions, target field data is acquired according to the candidate acquisition path configuration file. If the content corresponding to the candidate acquisition path configuration file does not meet the preset acquisition conditions, the iterative identification continues.

11. A device for acquiring page data, characterized in that, include: The candidate region selection module is used to render the target page to be collected, and select candidate regions including the target fields on the rendered target page according to the target fields to be collected, and generate a page screenshot including the candidate regions. The semantic information recognition module is used to identify the semantic information of the candidate region through a first intelligent agent based on the page screenshot and the document object model structure of the target page; The target module determination module is used to determine the target module where the target field is located by a second intelligent agent based on the semantic information. The data acquisition condition judgment module is used to determine, through a third intelligent agent, whether the content of the target module meets the preset data acquisition conditions; The data acquisition path configuration module is used to generate a data acquisition path configuration file corresponding to the target field based on the target module when the content of the target module meets the preset data acquisition conditions, so as to collect target field data based on the data acquisition path configuration file.

12. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images