Data acquisition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data acquisition method and device, electronic equipment and a storage medium, and relates to the technical field of the Internet, and the method comprises the steps: obtaining a plurality of second webpage addresses related to a first webpage address through crawling based on the pre-configured first webpage address; selecting a target webpage address meeting a preset condition from the plurality of second webpage addresses; obtaining initial webpage data in a webpage corresponding to the target webpage address based on the target webpage address; according to the method, the initial webpage data and the prompt text are input into the generative language model, and the initial webpage data are analyzed based on the prompt text through the generative language model to obtain the target webpage data, so that when the task requirement and the website architecture change, the required data can be obtained, and the universality of data acquisition is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and more specifically, to a data acquisition method, device, electronic device, and storage medium. Background Art

[0002] As an automated data capture tool, web crawlers play an important role in information retrieval, data collection, analysis and other fields. As the amount of network data explodes, the number and types of web pages continue to increase, and the content of web pages becomes more diverse and complex, the shortcomings of traditional crawler technology are gradually exposed.

[0003] In the related art, specific data in a web page is usually extracted by relying on a pre-designed solution.

[0004] However, changes in task requirements and website architecture require redesigning solutions to extract specific data from web pages, resulting in poor versatility in data acquisition. Summary of the Invention

[0005] In view of the above problems, the present application proposes a data acquisition method, device, electronic device and storage medium, which can obtain the required data when the task requirements and website architecture change, thereby improving the versatility of data acquisition.

[0006] In a first aspect, an embodiment of the present application provides a data acquisition method, comprising:

[0007] Based on a pre-configured first web page address, multiple second web page addresses related to the first web page address are crawled; a target web page address that meets preset conditions is selected from the multiple second web page addresses; initial web page data is obtained based on the target web page address; the initial web page data and prompt text are input into a generative language model, so that the initial web page data is analyzed by the generative language model based on the prompt text to obtain target web page data, and the prompt text is used to prompt the generative language model to analyze the initial web page data.

[0008] In a possible implementation, crawling and obtaining a plurality of second web page addresses related to a preconfigured first web page address based on the preconfigured first web page address includes:

[0009] A pre-configured first web page address is used as a first-level web page address, and multiple p-th level web page addresses are crawled from the web page corresponding to the first-level web page address, wherein the p-th level web page address is obtained by performing address resolution based on the content of the web page corresponding to the p-1-th level web page address, 2≤p≤M, and p and M are integers; the multiple p-th level web page addresses are used as multiple second web page addresses.

[0010] In a possible implementation, the method further includes:

[0011] Obtaining a web page structure tree, the web page structure tree being used to represent each target web page address and N-level parent web page addresses mapped to each target web page address, wherein the target web page address is obtained by performing address resolution based on the content of the web page corresponding to the first-level parent web page address, and the q-1-th-level parent web page address is obtained by performing address resolution based on the content of the web page corresponding to the q-th-level parent web page address, where 2≤q≤N<M, and q and N are integers; determining the N-level parent web page addresses corresponding to the target web page address based on the web page structure tree; inputting the initial web page data and the prompt text into a generative language model so that the generative language model analyzes the initial web page data based on the prompt text to obtain the target web page data, including: inputting the initial web page data, the prompt text, and the N-level parent web page addresses corresponding to the target web page address obtained from the initial web page data into the generative language model so that the generative language model analyzes the initial web page data based on the prompt text and the content of the web pages corresponding to the N-level parent web page addresses to obtain the target web page data.

[0012] In one possible implementation, obtaining initial web page data based on the target web page address includes: obtaining a web page structure tree, the web page structure tree including multi-level element nodes, wherein the multi-level element nodes include a first-level element node representing the first web page address, the first-level element node is connected to a child element node, and the last-level element node is connected to a parent element node. For the intermediate-level element nodes between the first-level element node and the last-level element node, the intermediate-level element node is connected to a parent element node and a child element node, each element node includes text content corresponding to the target web page address, and the relationship between the multi-level element nodes is determined according to the parent-child relationship between the web page addresses corresponding to the element nodes; obtaining a text length threshold; based on the text length threshold and the web page structure tree, obtaining the text content in at least one element node, and using the text content in the at least one element node as the initial web page data, wherein the length of the text content in each element node in the at least one element node is less than the text length threshold.

[0013] In one possible implementation, the text content in at least one element node is obtained based on the text length threshold and the web page structure tree, including: starting from the first-level element node, recursively traversing each element node in the web page structure tree until all traversable element nodes in the web page structure tree are traversed; wherein, during the recursive traversal process, when the length of the text content in one of the element nodes is less than the text length threshold, further traversal of the child element nodes of one of the element nodes is terminated, and the text content in one of the element nodes is used as the initial web page data; when the length of the text content in one of the element nodes is not less than the text length threshold, the child element nodes of one of the element nodes are traversed until the traversal of the last level of child element nodes is completed or the length of the text content in one of the first level child element nodes is less than the text length threshold.

[0014] In a possible implementation, selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses includes at least one of the following:

[0015] Determining the second webpage address having a target field as a target webpage address that meets a preset condition, wherein the target field includes: a field having a similarity with a preset keyword greater than a similarity threshold;

[0016] The second webpage address that can be accessed normally is determined as the target webpage address that meets the preset conditions.

[0017] In a possible implementation, before selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses, the method further includes:

[0018] The multiple second web page addresses are subjected to preset processing, the preset processing including at least one of web page address completion or web page address normalization; the web page address completion includes: performing web page address completion on the second web page address without a subject domain name link; the web page address normalization includes: performing normalization filtering on the second web page address ending with special characters or with an attached web page view; the selecting a first target web page address that meets the preset conditions from the multiple second web page addresses includes: selecting a first target web page address that meets the preset conditions from the multiple second web page addresses after the preset processing.

[0019] In a possible implementation, obtaining initial webpage data based on the target webpage address includes:

[0020] A target user agent for accessing the target web page address is determined from multiple user agents; a web page corresponding to the target web page address is accessed through the target user agent; and initial web page data in the web page corresponding to the target web page address is acquired.

[0021] In a second aspect, an embodiment of the present application provides a data acquisition device, comprising:

[0022] The URL collection module is used to crawl and obtain multiple second web page addresses related to a pre-configured first web page address based on the first web page address; the URL screening module is used to select a target web page address that meets preset conditions from the multiple second web page addresses; the data acquisition module is used to obtain initial web page data based on the target web page address; the initial web page data and prompt text are input into a generative language model, so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data, and the prompt text is used to prompt the generative language model to analyze the initial web page data.

[0023] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, in which program code is stored, wherein the above method is executed when the program code is run.

[0025] The embodiments of the present application provide a data acquisition method, device, electronic device and storage medium, which crawl and obtain multiple second web page addresses related to a pre-configured first web page address based on the first web page address; select a target web page address that meets preset conditions from the multiple second web page addresses; obtain initial web page data based on the target web page address; input the initial web page data and prompt text into a generative language model, so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data, and the prompt text is used to prompt the generative language model to analyze the initial web page data. In this way, after crawling the target web page address related to the first web page address, the initial web page data in the web page corresponding to the target web page address can be obtained, and then the initial web page data can be analyzed using the generative language model to obtain the target web page data. Therefore, when the task requirements and website architecture change, the required data can be obtained, thereby improving the versatility of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flowchart of a data acquisition method provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of another data acquisition method provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of an embodiment of the present application for processing collected websites to obtain a target URL list;

[0030] Figure 4 A schematic diagram of a process for crawling web page text provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a key-value pair provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the result of saving web page text provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a text analysis result output by a large model provided in an embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;

[0035] Figure 9 Shows a structural block diagram of an electronic device proposed in this application;

[0036] Figure 10 It is a storage unit in an embodiment of the present application for storing or carrying program codes for implementing the data acquisition method in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] With the rapid iteration of large language models (LLM) in the development of Internet AI products, data acquisition and accumulation are becoming increasingly important. Web crawlers, as automated data capture tools, play a key role in information retrieval, data collection, and analysis. As the amount of network data explodes, the number and types of web pages continue to increase, and the content of web pages becomes more diverse and complex, the shortcomings of traditional crawler technology are gradually exposed.

[0039] First, it usually relies on customized methods such as pre-set screening rules, specific web page structure crawling strategies, algorithm logic, specific codes, etc. When faced with web pages with different organizational methods and structures, it often shows limitations and inefficiency, and it is difficult to crawl data comprehensively, accurately, and stably. Secondly, changes in task requirements and website architecture require the redesign of solutions to extract web page data, which makes maintenance difficult, increases manpower, material and financial resources, and cannot achieve the macro-universality of the solution design. Finally, based on the powerful language understanding capabilities demonstrated by the application of large-scale pre-trained models in various industries, more companies are beginning to use models to optimize business processes, improve efficiency, and reduce manual intervention. By asking guided questions to the model, companies can better understand customer needs, market trends, and internal data.

[0040] Therefore, the embodiments of the present application propose a data acquisition method, device, electronic device and storage medium, which can acquire web page data when task requirements and website architecture change, thereby improving the versatility of web page data acquisition.

[0041] In summary, this embodiment's data acquisition method, based on a web crawler and a large model, minimizes the inclusion of filtered information, utilizes the structural characteristics of web pages to structure the text, and then uses the large model for secondary analysis, thereby accurately understanding and extracting the main content of the web page. This method not only improves the applicability and accuracy of the crawler, but also automates the processing of web pages with different designs and structures, providing high-quality, low-noise data for the large model input, thereby improving the accuracy of the output data. This provides a highly efficient solution for enterprises in search engines, data analysis, and business reporting.

[0042] Next, the data acquisition method is described.

[0043] See also Figure 1 , Figure 1 The method of this embodiment can be executed by an electronic device, such as Figure 1 The method shown may include:

[0044] S110: Based on a pre-configured first webpage address, crawl and obtain a plurality of second webpage addresses related to the first webpage address.

[0045] Among them, the first web page address can be one of the Uniform Resource Locators (URL), which can be an address used to locate resources on the Internet, for example, it can point to various resources such as web pages, files, images, videos, etc. The web page address can also be referred to as the URL. In this embodiment, the first web page address is used to point to web page resources. A web page is the basic unit that constitutes a website. It is essentially a document accessed through a browser, which is used to display information, provide services or enable user interaction on the Internet. Exemplarily, the first web page address can be, for example: https: / / www.XXX.com / product, which can be a web page for a product (such as a camera), which can include an introduction to a product. In this embodiment, the first web page address can be pre-configured. When implementing the method of the embodiment of the present application, the pre-configured first web page address can be obtained, and then the second web page address can be crawled.

[0046] The second web page address can be obtained by crawling based on the first web page address. In this embodiment, the second web page address can be a web page address obtained from the web page corresponding to the first web page address. Generally speaking, web pages are usually organized in a tree structure, with the parent web page serving as an entry or directory, and the child web pages containing more specific content. The parent web page can display a link or control that jumps to the child web page, and the web page address that jumps to the child web page can be obtained through the link or control. In other words, this embodiment can obtain multiple second web page addresses from the link or jump control in the web page corresponding to the first web page address. It should be noted that in this embodiment, the second web page address can be understood as the address of a child web page of the web page corresponding to the first web page address, or the address of a child web page of a child web page of a child web page of the web page corresponding to the first web page address, and this is not limited here. It should be noted that the depth of crawling child web pages can be set as needed and is not limited here. For example, if it is set to 2, then the address of a child web page of a child web page of the web page corresponding to the first web page address can be obtained as the web page address.

[0047] S120: Select a target webpage address that meets a preset condition from the plurality of second webpage addresses.

[0048] The preset condition may be a pre-set condition. In this embodiment, the preset condition is used to filter at least some of the second webpage addresses from the plurality of second webpage addresses as the target webpage address. In this embodiment, by selecting the target webpage address that meets the preset condition from the plurality of second webpage addresses, the effectiveness, accuracy, and efficiency of data acquisition can be improved.

[0049] S130: Acquire initial web page data based on the target web page address.

[0050] The initial webpage data may be data extracted from the webpage corresponding to the target webpage address, including but not limited to text, images, or audio, and may be configured as needed without limitation. Specifically, the data may be obtained from part or all of the webpage corresponding to the target webpage address. In this embodiment, the webpage data may refer to structured or semi-structured information extracted from the webpage.

[0051] S140: Input the initial web page data and the prompt text into a generative language model, so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data, wherein the prompt text is used to prompt the generative language model to analyze the initial web page data.

[0052] Among them, generative language models (GLMs) are a type of natural language processing model based on artificial intelligence technology. Its core capability is to generate new texts that conform to grammatical rules and are semantically coherent by learning a large amount of text data. The generative language model in this embodiment can be, for example, an LLM model. LLM trains a large-scale neural network through massive amounts of text data to learn knowledge such as the grammar and semantics of a language, so that it can understand and generate natural language text. In this embodiment, the prompt text can be pre-configured, so that when analysis is required, the acquired initial web page data and the prompt text are input into the generative language model, so that the generative language model processes the initial web page data based on the prompt text, and then outputs the target web page data. In this embodiment, the target web page data can be data obtained by analyzing the initial web page data. Optionally, the target web page data can be understood as the summary analysis result of the initial web page data.

[0053] Below, in order to better understand the solution of the embodiment of the present application, an example is provided to exemplarily illustrate the comparison between the related technology and the solution of the embodiment of the present application.

[0054] For example, to obtain information about a product (such as a camera), related technologies require analyzing the data structure of a webpage to determine the specific location of the required data on the webpage. Then, a solution is designed to retrieve the required data from that specific location on the webpage. Consequently, if the website architecture changes, the solution needs to be redesigned to retrieve the required data.

[0055] In an embodiment of the present application, initial web page data can be obtained from the sub-web pages of the web page, and then the initial web page data can be processed using a generative language model to obtain the required data (such as target web page data). In this way, even if the website architecture changes, the required data can be obtained.

[0056] In this embodiment, based on a pre-configured first web page address, multiple second web page addresses related to the first web page address are crawled; a target web page address that meets preset conditions is selected from the multiple second web page addresses; initial web page data is obtained based on the target web page address; the initial web page data and prompt text are input into a generative language model, so that the initial web page data is analyzed by the generative language model based on the prompt text to obtain target web page data, and the prompt text is used to prompt the generative language model to analyze the initial web page data. In this way, after crawling the target web page address related to the first web page address, the initial web page data in the web page corresponding to the target web page address can be obtained, and then the initial web page data can be analyzed using the generative language model to obtain the target web page data. Therefore, when the task requirements and website architecture change, the required data can be obtained, thereby improving the versatility of data acquisition.

[0057] In a possible implementation, crawling and obtaining a plurality of second web page addresses related to a pre-configured first web page address includes:

[0058] A pre-configured first web page address is used as a first-level web page address, and multiple p-th level web page addresses are crawled from the web page corresponding to the first-level web page address, wherein the p-th level web page address is obtained by performing address resolution based on the content of the web page corresponding to the p-1-th level web page address, 2≤p≤M, and p and M are integers; the multiple p-th level web page addresses are used as multiple second web page addresses.

[0059] In this embodiment, the p-level web page address is obtained by performing address resolution based on the content of the web page corresponding to the p-1-level web page address. The web page corresponding to the p-1-level web page address can be understood as the parent web page of the web page corresponding to the p-level web page address. In other words, the web page corresponding to the p-level web page address can be understood as the child web page of the web page corresponding to the p-1-level web page address. The content of the web page can include, but is not limited to, the links or controls in the above embodiments. The subset web page can be the web page to which the link or control jumps; alternatively, the content of the web page includes static content or dynamically rendered content. Different classification methods have different classification results, which are not limited here. In this embodiment, M can be an integer not less than 2.

[0060] Exemplarily, taking M=2 as an example, address resolution is performed based on the content of the web page in the web page corresponding to the first-level web page address to obtain multiple second-level web page addresses.

[0061] As another example, taking M=3 as an example, address resolution is performed based on the content of the web page in the web page corresponding to the first-level web page address to obtain multiple second-level web page addresses. Then, for each second-level web page address, address resolution is performed based on the content of the web page in the web page corresponding to the second-level web page address to obtain multiple third-level web page addresses, and the second-level web page address and the multiple third-level web page addresses corresponding to each second-level web page address are used as multiple second web page addresses.

[0062] It should be understood that if M is a value greater than 4, the above exemplary description can also be referred to, and no further details are given here. The smaller the value of M, the more efficient the second webpage address is in obtaining, and thus the more efficient the data acquisition is. The larger the value of M, the deeper the sub-level webpages obtained, which can better explore the depth of the data and thus improve the accuracy of data acquisition.

[0063] It should be noted that using the first webpage address as the first-level webpage address can be understood as obtaining the webpage address of the subset webpage from the first webpage address as the parent webpage, which does not mean that the first webpage address does not have a corresponding parent webpage.

[0064] In this embodiment, by using a pre-configured first web page address as a first-level web page address, multiple p-level web page addresses are crawled based on the web page corresponding to the first-level web page address; and using the multiple p-level web page addresses as multiple second web page addresses, the depth of the web page address can be mined, thereby improving the accuracy of data acquisition.

[0065] In another possible implementation, based on the first web page address, a web page address having a certain similarity to the first web page address can be used as the second web page address. In this way, the breadth of web page addresses can be explored, thereby improving the accuracy of data acquisition.

[0066] In one possible implementation, before inputting the initial webpage data and the prompt text into a generative language model to analyze the initial webpage data based on the prompt text by the generative language model to obtain the target webpage data, the method further includes:

[0067] Obtain a web page structure tree, which is used to represent each target web page address and the N-level parent web page addresses mapped to each target web page address, wherein the target web page address is obtained by address resolution based on the content of the web page corresponding to the 1st-level parent web page address, and the q-1th-level parent web page address is obtained by address resolution based on the content of the web page corresponding to the qth-level parent web page address, 2≤q≤N<M, and q and N are integers; determine the N-level parent web page addresses corresponding to the target web page address based on the web page structure tree.

[0068] The web page structure tree may be, for example, a HyperText Markup Language (HTML) structure tree. The HTML structure tree may be a structure tree obtained by performing HTML parsing on the content of a dynamically rendered web page. For example, taking M as 3, the target web page address may be obtained by performing address resolution based on the content of the web page corresponding to the second-level web page address (i.e., the first-level parent web page address); and the second-level web page address may be obtained by performing address resolution based on the content of the web page corresponding to the first-level web page address (i.e., the second-level parent web page address).

[0069] In this embodiment, the N-level parent web page addresses mapped to each target web page address can be known through the web page structure tree.

[0070] It should be noted that the web page structure tree can be constructed when crawling the second web page address based on the first web page address, or the web page structure tree can be constructed after obtaining the target web page address from the second web page address. There is no restriction on the timing of how to construct the web page structure tree.

[0071] Accordingly, the initial webpage data and the prompt text are input into a generative language model so as to analyze the initial webpage data based on the prompt text by the generative language model to obtain target webpage data, including:

[0072] The initial web page data, prompt text, and N-level parent web page addresses corresponding to the target web page address from which the initial web page data is obtained are input into a generative language model, so that the generative language model analyzes the initial web page data based on the contents of the web pages corresponding to the prompt text and the N-level parent web page addresses to obtain the target web page data.

[0073] In this embodiment, the generative language model can obtain webpage content from the webpages corresponding to each of the N-level parent webpage addresses corresponding to the target webpage address, and analyze the initial webpage data obtained from the target webpage address based on the content of the webpages corresponding to each of the N-level parent webpage addresses. In this way, the initial webpage data can be analyzed by understanding the context of the content of the webpages corresponding to each of the N-level parent webpage addresses, thereby improving the analysis results of the initial webpage data. In this embodiment, the content of the webpage corresponding to the parent webpage address can include but is not limited to text, images, or audio; or can include but is not limited to static content or dynamically rendered content.

[0074] It should be noted that in the embodiment of the present application, after obtaining the content of the initial web page, the content of the initial web page can be stored correspondingly by a unique identifier. Optionally, a unique hash value calculated from the URL string can be used as the file name to ensure a one-to-one correspondence between the initial web page data and the URL.

[0075] In another possible implementation, only the initial web page data may be input into the generative language model. This can improve the efficiency of the generative language model in analyzing the initial web page data.

[0076] In a possible implementation, obtaining initial webpage data based on the target webpage address includes:

[0077] Obtain a web page structure tree, the web page structure tree including multi-level element nodes, wherein the multi-level element nodes include a first-level element node representing the first web page address, the first-level element node is connected to a child element node, and the last-level element node is connected to a parent element node. For intermediate-level element nodes between the first-level element node and the last-level element node, the intermediate-level element node is connected to a parent element node and a child element node, each element node includes text content corresponding to the target web page address, and the relationship between the multi-level element nodes is determined according to the parent-child relationship between the web page addresses corresponding to the element nodes; obtain a text length threshold; based on the text length threshold and the web page structure tree, obtain the text content in at least one element node, and use the text content in the at least one element node as the initial web page data, wherein the length of the text content in each element node in the at least one element node is less than the text length threshold.

[0078] In this embodiment, the text length threshold can be understood as a condition for screening initial web page data. The initial web page data can be, for example, initial text content. Accordingly, the target web page data obtained in this embodiment includes the target text content.

[0079] For example, in this embodiment, the network structure tree may include a first-level element node A, second-level element nodes A1 and A2, and third-level element nodes A11, A12, A21, and A22. A is connected to A1 and A2, respectively. A1 is connected to A12 and A11, respectively, and A2 is connected to A21 and A22, respectively. Each element node corresponds to a webpage address. For example, node A corresponds to a first webpage address, and node A may include the text content of the webpage corresponding to the first webpage address. Nodes A1 and A2 correspond to second-level webpage addresses A1 and A2 obtained from the first webpage address, respectively. Node A1 may include the text content of the webpage corresponding to the second-level webpage address A1, and node A2 may include the text content of the webpage corresponding to the second-level webpage address A2. A12 and A11 correspond to third-level webpage addresses A12 and A12 obtained from the second-level webpage address A1, respectively. Node A11 may include the text content of the webpage corresponding to the third-level webpage address A11, and node A12 may include the text content of the webpage corresponding to the third-level webpage address A12.

[0080] In this embodiment, the initial web page data is obtained through the web page structure tree, which can improve the comprehensiveness of the initial web page data.

[0081] In a possible implementation, obtaining text content in at least one element node based on the text length threshold and the webpage structure tree includes:

[0082] Starting from the first-level element node, recursively traverse each element node in the web page structure tree until all traversable element nodes in the web page structure tree are traversed; wherein, during the recursive traversal process, when the length of the text content in one of the element nodes is less than the text length threshold, terminate further traversal of the child element nodes of one of the element nodes, and use the text content in one of the element nodes as the initial web page data; when the length of the text content in one of the element nodes is not less than the text length threshold, traverse the child element nodes of one of the element nodes until the traversal of the last level of child element nodes is completed or the length of the text content in one of the first-level child element nodes is less than the text length threshold.

[0083] Exemplarily, starting from node A, if the text content in node A is not less than the text length threshold, traverse node A1. If the text content of node A1 is less than the length of the text content, stop traversing nodes A11 and A12. If the text content of node A1 is not less than the length of the text content, traverse the text content of nodes A11 and A12. If the text content of node A11 is less than the text length threshold, retain the text content of node A11 as the initial web page data. If the text content of node A11 is not less than the text length threshold, do not retain the text content of node A11 as the initial web page data. Similarly, if the text content of node A12 is less than the text length threshold, retain the text content of node A12 as the initial web page data. If the text content of node A12 is not less than the text length threshold, do not retain the text content of node A12 as the initial web page data. Similarly, if the text content in node A is not less than the text length threshold, then node A2 can also be traversed. The traversal process of node A2 can refer to the description of the traversal process of node A1, and will not be repeated here.

[0084] In general, starting from the root node (first-level element node) of the web page, recursively traverse each element node in the web page structure tree. During the traversal process, when the text content length of a certain node is less than the preset text length threshold, further traversal of the node's child nodes is terminated, and the text content of the node is used as the initial web page data. In addition, the text content of the node and its corresponding child node structural relationship can also be saved together. The saved content includes information such as the tag name, text content, hierarchical structure of the current node and child nodes, which is used to restore the contextual semantics of the node in the original web page structure. The traversal operation continues until all reachable element nodes in the current web page structure tree are traversed.

[0085] In this embodiment, it is possible to control the acquisition of a certain amount of text content, thereby reducing the problem of slow data analysis caused by excessive text content, thereby improving the efficiency of data acquisition and data analysis.

[0086] It should be noted that the text content of this embodiment can be the text content included in the web page corresponding to the target web page address, or it can be the text content converted from the image or audio of the web page corresponding to the target web page address, which is not limited here.

[0087] It should be noted that, in this embodiment, the first webpage address may be one or more. When there are multiple first webpage addresses, each of the multiple first webpage addresses may have corresponding initial webpage data.

[0088] In a possible implementation, selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses includes at least one of the following:

[0089] Determining the second webpage address having a target field as a target webpage address that meets a preset condition, wherein the target field includes: a field having a similarity with a preset keyword greater than a similarity threshold;

[0090] The second webpage address that can be accessed normally is determined as the target webpage address that meets the preset conditions.

[0091] In this embodiment, each field in the web page address can be matched with a preset keyword for similarity. If the similarity between one of the fields and the preset keyword is higher than a similarity threshold, the web page address is determined to be the target web page address. Optionally, the vectors of each field can be extracted and compared with the vectors of the preset keyword to perform vector similarity calculation, such as calculating the Euclidean distance, to determine the similarity between the field and the preset keyword. For example, assuming that the preset keyword is camera, the target field can include but is not limited to fields such as camera, camera, camera, and photo. The actual target field is related to the preset keyword, and the preset keyword is related to the actual data acquisition task, and no specific limitation is made here.

[0092] In this embodiment, web page address verification includes, but is not limited to: protocol header check: ensuring that the URL begins with http: / / or https: / / , which is the basic protocol for accessing web pages. Domain name check: checking whether the domain name is spelled correctly and complies with domain name specifications (such as using letters, numbers, hyphens, and not starting or ending with a hyphen). Path and query parameter check: ensuring that the path and query parameters (if any) are formatted correctly and do not contain unnecessary special characters or encoding errors. In addition, verification can be performed by sending a HyperText Transfer Protocol Request (HTTP request). Specifically, a browser developer tool, curl command, or programming language (such as Python's requests library) can be used to send an HTTP request to the URL. Then, the response status code is checked: 200 OK: indicates that the request was successful and the web page exists and is accessible. 404 Not Found: indicates that the web page does not exist. Other status codes, such as 403 (Forbidden) and 500 (Server Internal Error), also indicate that there may be a problem with the web page. Only web page addresses that have passed verification (valid) are accessed.

[0093] That is, in this embodiment, the preset conditions may include: the webpage address has a target field similar to the preset keyword, or the webpage address is accessible, etc.

[0094] In another possible implementation, a second web page address with a complete or valid web page address may be used as the target web page address, and screening conditions may be set as needed, which is not limited here.

[0095] In a possible implementation, before selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses, the method further includes:

[0096] The multiple second web page addresses are subjected to preset processing, and the preset processing includes at least one of web page address completion, web page address normalization, or determination of a user agent; the web page address completion includes: performing web page address completion on the second web page address without a subject domain name link; the web page address normalization includes: performing normalization filtering on the second web page address ending with special characters or with an attached web page view.

[0097] The following further describes the above preset processing methods.

[0098] For web address completion, in this embodiment, a link without a subject domain name: usually refers to an incomplete URL, which may be missing a protocol header (such as http: / / or https: / / ) or a domain name part. Web address completion: refers to completing these incomplete URLs so that they can be accessed correctly. Specifically, the missing part of the URL can be identified: check whether the URL is missing a protocol header (http: / / or https: / / ). Then, check whether the URL is missing a domain name part (such as www.example.com). Then, the protocol header can be supplemented: if the URL starts with / / , it usually means that the protocol header is missing. According to the security and common practices of the website, https: / / can be supplemented by default. If the URL does not start with / / , but obviously lacks a protocol header, https: / / also needs to be supplemented, unless there is a specific reason to use http: / / . Then, the domain name can be supplemented: if the URL completely lacks the domain name part, it is necessary to know the domain name to which the URL belongs. This can usually be inferred through context, website structure or other information. In some cases, if the URL is a relative path (such as / path / to / page), it needs to be combined with the base domain name to form a complete URL.

[0099] For the normalization of web page addresses, in this embodiment, the special character ending can be: the URL may end with some special characters (such as?, #, &, etc.). These characters have specific meanings in the URL, but if used improperly or redundantly, they may cause the URL to be invalid or inconsistent. The accompanying web page view can be: the URL may contain web page view parameters (such as?view=print,?layout=mobile, etc.). These parameters are used to control the display mode of the web page, but may need to be removed or standardized during the normalization process. Specifically, check whether there are redundant special characters (such as?, #, &, etc.) at the end of the URL. If these characters are not followed by valid query parameters or fragment identifiers, they should be removed. Example: Normalize https: / / www.example.com / page? to https: / / www.example.com / page. Then, the query parameters can be processed, and the query parameters in the URL can be analyzed to determine which parameters are required, which are redundant or used for web page views, and then the redundant query parameters can be removed or standardized, especially those related to web page views. Example: Normalize the following web page address: https: / / www.example.com / page? view=print&lang=e; The normalized result may be: https: / / www.example.com / page?lang=en (assuming the view parameter is redundant).

[0100] Accordingly, the step of selecting a first target webpage address that meets a preset condition from the plurality of second webpage addresses includes:

[0101] A first target webpage address that meets a preset condition is selected from the plurality of second webpage addresses that have undergone preset processing.

[0102] In this embodiment, by performing preset processing on the multiple second web page addresses, the preset processing includes at least one of web page address completion or web page address normalization; selecting the first target web page address that meets the preset conditions from the multiple second web page addresses after the preset processing, this can improve the success rate of accessing the web page, and further, can improve the comprehensiveness of the initial web page data obtained, so that the target web page data obtained can be more accurate when analyzed through more comprehensive initial web page data.

[0103] In another possible implementation, it is also not necessary to perform preset processing on the second webpage address, thereby improving the efficiency of data analysis.

[0104] In a possible implementation, obtaining initial webpage data based on the target webpage address includes:

[0105] A target user agent for accessing the target web page address is determined from multiple user agents; a web page corresponding to the target web page address is accessed through the target user agent; and initial web page data in the web page corresponding to the target web page address is acquired.

[0106] In this embodiment, the user agent string typically includes the following information: Browser type and version: such as Chrome, Firefox, Safari, etc. Operating system and version: such as Windows, macOS, Linux, etc. Device identifier: such as mobile device, desktop device, etc. Rendering engine: such as WebKit, Blink, etc. Factors to consider when determining the target user agent:

[0107] Web page compatibility: Browser compatibility: Ensure that the target user agent's browser is compatible with the web page technology (such as HTML5, CSS3, JavaScript, etc.) of the second web page address. Operating system compatibility: Consider the display effect and function support of the web page on different operating systems.

[0108] Access requirements: Specific functions: If the second webpage address provides specific functions (such as mobile version, desktop version, etc.), you need to select a user agent that supports these functions. User experience: Select a user agent that can provide the best user experience, such as loading speed, interactive effects, etc.

[0109] Security: Encryption support: Ensure that the target user agent supports secure protocols such as HTTPS. Privacy protection: Consider the privacy protection capabilities of the user agent, such as whether it supports the "Do Not Track" request header.

[0110] Statistics and analysis: Traffic: Analyze the traffic of different user agents and select user agents with higher traffic as target user agents. User behavior: Understand the user behavior patterns of different user agents and select user agents that match the target user group. When selecting target user agents, you can prioritize: Based on the above considerations, prioritize different user agents. Prioritize user agents with high compatibility, satisfied access requirements, strong security, high traffic, and consistent user behavior. Alternatively, verify through testing: Use the selected user agent to access the second web address to verify its compatibility and feature support. Adjust the user agent selection based on the test results until the most suitable target user agent is found.

[0111] In this embodiment, a target user agent for accessing the target web page address is determined from multiple user agents; a web page corresponding to the target web page address is accessed through the target user agent; and initial web page data in the web page corresponding to the target web page address is obtained, thereby improving the robustness and flexibility of web page access and ensuring that website data is crawled.

[0112] In another possible implementation, a fixed user agent may be used, which can improve the efficiency of web page access and data crawling, thereby improving the efficiency of data analysis.

[0113] For ease of understanding, the following embodiment takes obtaining analysis data from a camera as an example to exemplify the solution of the embodiment of the present application.

[0114] See also Figure 2 , Figure 2 This is a flow chart of another data acquisition method provided in an embodiment of the present application. Figure 2 The method shown may include:

[0115] S210, keyword screening.

[0116] In this embodiment, the function of filtering keywords includes: obtaining target web page addresses that meet the conditions from crawled web page addresses.

[0117] S220, website collection.

[0118] In this embodiment, a website can also be referred to as a web page address. This embodiment can crawl a second web page address based on a pre-configured first web page address, and then filter a target web page address from the second web page address based on a keyword. Furthermore, based on the accumulated keyword filtering experience from multiple web page address crawls, keywords can be supplemented or modified based on the crawling results of multiple web page addresses.

[0119] S230: crawling web page texts from the collected websites, and converting the crawled web page texts into a preset format.

[0120] In this embodiment, web page text (one type of initial web page data) can be crawled from the collected website (e.g., the target web page address) and stored in a preset format. The preset format can be set as needed and is not limited here.

[0121] S240, large model analysis.

[0122] In this embodiment, the large model can be used to analyze the crawled web page text to obtain the analysis result of the web page text (one type of target web page data).

[0123] In general, through a plurality of preset URLs (such as the first web page address), the URLs of products that meet the conditions (such as the target web page address) are crawled and collected and saved. In this embodiment, when collecting URLs of products that meet the conditions, general screening keywords (also called preset keywords) can be designed according to demand, and screening experience can be accumulated through multiple web page data crawling to improve the accuracy and comprehensiveness of the keywords. Specifically, general screening keywords can be designed, and screening keyword experience can be accumulated through multiple manual preliminary analysis and crawling of web page data elements to improve the accuracy and comprehensiveness of the keywords. Then, the text content of the target URL is crawled, formatted, and the data is saved in a predetermined format. Furthermore, the formatted content is used as the input of the large model to generate a structural analysis summary of the web page content, providing high-quality data for further analysis and report generation.

[0124] Among them, when collecting web page addresses, the processes that can be executed include but are not limited to: URL completion, URL normalization, regular matching, URL string header judgment, multiple user agents, URL verification, and ensuring that the URL is relevant to the subject and the access is valid.

[0125] See also Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the present application for processing collected websites to obtain a target URL list. Figure 3 The method shown may include:

[0126] S310, multi-user agent.

[0127] The multiple user agents may refer to maintaining multiple user agents, and then determining a target user agent for accessing a web page address among the multiple user agents.

[0128] S320, URL completion.

[0129] The URL completion may be to complete the URL.

[0130] S330, URL filtering.

[0131] Among them, URL filtering can also be called web page address normalization filtering.

[0132] S340: Determine whether the network request can access the web page address.

[0133] Here, it is determined whether the network request can access the webpage address, thereby determining whether the webpage address can be normally accessed. In this embodiment, if yes, S350 is executed.

[0134] S350: Output the target URL list.

[0135] The target URL list includes one or more target URL addresses (also known as target web page addresses). The target URL addresses in the target URL list are used to determine the target user agent, complete the URL, perform URL filtering, and determine the addresses that can be accessed normally.

[0136] It should be understood that, in this embodiment, at least some of the steps S310-S340 may be selected for execution. Furthermore, the execution order of the steps S310-S340 may also be adjusted, and the specific execution order is not limited here.

[0137] Specifically, in this embodiment, URL completion: web page address completion is performed for relative addresses without subject domain name links. URL normalization: URLs ending with special characters or with request content such as page views are normalized and filtered. Multiple user agents: multiple user-agent pools are maintained, and multiple simulated browsers and device accesses are set up to enhance the robustness and flexibility of the crawler program and ensure that website data is crawled. URL verification: after the crawled links are standardized, the validity of these links is verified to ensure that the system crawls relevant links.

[0138] Among them, when parsing the webpage text of the target URL, you can use the following method: Figure 4 Steps shown. Figure 4 A schematic diagram of a process for crawling web page text provided in an embodiment of the present application. Figure 4 The method shown may include:

[0139] S410: Obtain a target URL address from a target URL list.

[0140] S420: Perform HTML parsing on the content of the web page corresponding to the target URL address.

[0141] S430: Obtain the webpage text obtained by resolving the target URL address.

[0142] S440: Determine whether the length of the webpage text is less than or equal to a text length threshold.

[0143] In this example, if no, the process returns to step S430 to obtain another webpage text obtained by resolving the target URL address. If yes, the process proceeds to step S450.

[0144] S450: Record the webpage text in the form of key-value pairs.

[0145] S460: Save the web page text in the key-value pair as a Json file.

[0146] Perform HTML parsing on web page content including dynamic rendering; perform recursive judgment based on the HTML tree structure, set a text length threshold as the recursive stopping point to prevent the recursion from being too deep, ensure that the text information length is appropriate while not losing hierarchical structure information; save the text at the end of each recursion of a single node in the form of a dictionary of key-value pairs, using the unique cumulative total of the current node recursion as the dictionary key, and the value as the text in the node. The unique hash value calculated with the current URL string is used as the file name to ensure a one-to-one correspondence between the text content and the URL. Among them, S430 and S440 can be understood as the steps of recursive parsing.

[0147] In this example, when parsing HTML, you can limit crawling based on the text length threshold and the parent-child relationship of the web page. When traversing all text contents under the parent tag (also known as the parent web page address), including the content of the child tag (also known as the child web page address), if the text length threshold is exceeded, continue to recursively traverse the child tags until the text length of the child tag content is less than or equal to the text length threshold, and then save all text under the current child tag. This text contains a hierarchical structure replaced by line breaks. The specific example content is as follows Figure 5 shown. Figure 5 A schematic diagram of a key-value pair provided in an embodiment of the present application.

[0148] like Figure 5As shown, when saving the text of the current sub-tag, the line break character "\n" is used to represent the hierarchical structure. Specifically, the part before "\n" is the parent webpage address, and "\n" can be the child webpage address. Among them, text-m represents the recursive cumulative total, and m represents a specific value, such as 0, 1, 5, 7, etc., which is determined according to actual conditions and is not limited here.

[0149] In this embodiment, when saving the webpage text in the key-value pair as a Json file, that is, when saving the current webpage content, the unique hash value calculated with the URL string is used as the file name to ensure that the text content and the URL correspond one to one. Figure 6 As shown, Figure 6 This is a schematic diagram of the result of saving web page text provided by the embodiment of this application. Figure 6 As shown, it includes hash value, target web page address, crawling depth and corresponding web page content.

[0150] Among them, when performing analysis through a large model (also known as a large language model), you can call the Application Programming Interface (API) to access the large model, set the role and prompt words of the large model, batch process formatted and saved data, design appropriate guided questions, summarize and judge the content of the document, preset the content output format in the prompt words and save it.

[0151] See also Figure 7 , Figure 7 A schematic diagram of text analysis results output by a large model provided in an embodiment of the present application. Figure 7 The text analysis result shown can be a type of target web page data. Figure 7 The text analysis results shown include product names and prices, user ratings, product features or technical specifications, etc.

[0152] In this embodiment, based on the web page structure tree, the initial web page text of the target web page address and the corresponding N-level parent web page address can be input into the large model together. This enables the large model to perform contextual understanding based on the web page content of the corresponding N-level parent web page address, thereby improving the accuracy of the initial web page text analysis, and further improving the accuracy of the obtained target web page text.

[0153] This allows us to use HTML structure tree analysis to abstract webpage content and preserve relationships between data, effectively avoiding the limitations of single-webpage crawlers and increasing the diversity of data acquisition channels. Furthermore, for the collected source data, we leverage the current mainstream large-scale text question-answering method to design guided questions and rules, unifying the data format and improving the efficiency of data cleaning and analysis. Furthermore, we propose a universal data collection method for changing webpage structures, facilitating code reuse and reducing the difficulty of system-level integration.

[0154] Specifically, in this embodiment, the website collection module selects 10 camera brands based on business needs, investigates their official websites, and obtains a list of seed URLs, which are called first-level nodes (also called first-level web page addresses) here. The crawler technology is used to crawl according to the specified crawling depth, and the crawled nodes are preliminarily screened to obtain a set of second-level nodes.

[0155] Then, since this method adopts the breadth-first search principle, it only obtains the URLs in the second-level node web pages - called third-level nodes (also called third-level web page addresses or target web page addresses). After the second-level nodes (also called second-level web page addresses) are visited, they will not be visited again. In the case where the second-level nodes and third-level nodes overlap, it is necessary to reset the second-level node flag to the unvisited third-level node for subsequent content crawling and parsing, which can avoid the situation where there is unparsed content.

[0156] Then, this method sets the crawling depth to 2 based on the web page hierarchical links of brand products and user click-through experience. After crawling all URLs that meet the keywords and depth, they are integrated into a URL set, called the three-level node URL set. The set is traversed to crawl specific text content.

[0157] Specifically, in this embodiment, based on the given multiple addresses of similar products, after collecting and organizing the product link addresses of each brand, the commonly existing keyword URLs are found. In the current embodiment, 10 brands are provided, and the first and second level link keywords are selected based on experience. This method follows the majority principle. After the amount of brand data reaches a certain level, the keywords are gradually searched comprehensively. On the premise of ensuring universality and using a small number of keywords, the effect of low or no missed selection rate is achieved.

[0158] Specifically, in the present embodiment, the URL set traversal in the above steps can be performed, web page crawling and parsing are performed, and the text content of each web page is parsed by bs (Beautiful Soup) 4, wherein there are static pages and dynamically rendered loaded web pages, and the selenium library is used uniformly. The embodiment of the present application, on the basis of parsing the plain text content, saves the hierarchical structure of the text in a specified format to meet the semantic context understanding of different texts by the large model. Wherein the hierarchical structure is replaced by a line break when saving the text to ensure the front and back connection relationship. In an embodiment, a hierarchical content maximum text length threshold L is set. If the threshold is too low, the attribute field space of a single specified file will be increased, and if it is too high, the complexity of each key-value pair will be increased. Then, the threshold is determined according to the product detailed description experience value L, and the current embodiment is set to 200 words.

[0159] Among them, when parsing text, crawling is limited according to the text threshold and the parent-child relationship of the web page. When traversing all text contents under the parent tag, including the sub-tag content, if the text threshold is exceeded, the sub-tag will continue to be recursively traversed until the text length is less than or equal to the threshold, and then all text under the current sub-tag will be saved. This text contains a hierarchical structure replaced by line breaks. The specific example content is as follows Figure 5 shown.

[0160] Then, when saving the current web page content, the unique hash value calculated from the URL string is used as the file name to ensure that the text content and the URL correspond one to one. For example, Figure 6 shown.

[0161] Specifically, in this embodiment, the large model analysis module saves the output formatted content according to the analyzed URL and a unique hash value file name. By asking guided questions (prompts) to the large model, the file is screened and the data formatting analysis of the file content is performed to obtain high-quality data for subsequent report generation and other data analysis needs.

[0162] Furthermore, guided questions such as the following require setting system roles and user roles: { "role": "system", "content": "You are Kimi, an AI assistant. You are better at communicating in Chinese and English. You will provide users with safe, helpful, and accurate answers.",}, {"role": "user", "content": "Please briefly introduce and summarize the contents of the json file"}, Output format text such as Figure 7 shown.

[0163] The following is an exemplary description of the device embodiment of this embodiment.

[0164] See also Figure 8 , Figure 8 This is a structural diagram of a data acquisition device provided in an embodiment of the present application. Figure 8 The device shown can be applied to an electronic device, and the device may include:

[0165] The URL collection module 810 is used to crawl and obtain multiple second web page addresses related to a pre-configured first web page address based on the first web page address; the URL screening module 820 is used to select a target web page address that meets preset conditions from the multiple second web page addresses; the data acquisition module 830 is used to obtain initial web page data based on the target web page address; the initial web page data and prompt text are input into a generative language model so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data, and the prompt text is used to prompt the generative language model to analyze the initial web page data.

[0166] In a possible implementation, when the URL collection module 810 crawls and obtains multiple second webpage addresses related to a pre-configured first webpage address based on the first webpage address, it can be used to:

[0167] A pre-configured first web page address is used as a first-level web page address, and multiple p-th level web page addresses are crawled from the web page corresponding to the first-level web page address, wherein the p-th level web page address is obtained by performing address resolution based on the content of the web page corresponding to the p-1-th level web page address, 2≤p≤M, and p and M are integers; the multiple p-th level web page addresses are used as multiple second web page addresses.

[0168] In a possible implementation, the data acquisition module 830 is further configured to:

[0169] Obtain a web page structure tree, the web page structure tree being used to represent each target web page address and N levels of parent web page addresses mapped to each target web page address, wherein the target web page address is obtained by performing address resolution based on the content of the web page corresponding to the first-level parent web page address, and the q-1th-level parent web page address is obtained by performing address resolution based on the content of the web page corresponding to the qth-level parent web page address, where 2≤q≤N<M, and q and N are integers; determine the N levels of parent web page addresses corresponding to the target web page address based on the web page structure tree; and when the data acquisition module 830 inputs the initial web page data and prompt text into a generative language model, so that the generative language model analyzes the initial web page data based on the prompt text to obtain the target web page data, the initial web page data, the prompt text, and the N levels of parent web page addresses corresponding to the target web page address obtained from the initial web page data are input into the generative language model, so that the generative language model analyzes the initial web page data based on the prompt text and the content of the web pages corresponding to the N levels of parent web page addresses to obtain the target web page data.

[0170] In one possible implementation, when the data acquisition module 830 acquires the initial web page data based on the target web page address, it can be used to: acquire a web page structure tree, the web page structure tree includes multi-level element nodes, wherein the multi-level element nodes include a first-level element node representing the first web page address, the first-level element node is connected to a child element node, and the last-level element node is connected to a parent element node. For the intermediate-level element nodes between the first-level element node and the last-level element node, the intermediate-level element node is connected to a parent element node and a child element node, each element node includes text content corresponding to the target web page address, and the relationship between the multi-level element nodes is determined according to the parent-child relationship between the web page addresses corresponding to the element nodes; acquire a text length threshold; based on the text length threshold and the web page structure tree, acquire the text content in at least one element node, and use the text content in the at least one element node as the initial web page data, wherein the length of the text content in each element node in the at least one element node is less than the text length threshold.

[0171] In one possible implementation, when the data acquisition module 830 acquires the text content in at least one element node based on the text length threshold and the web page structure tree, it can be used to: start from the first-level element node, recursively traverse each element node in the web page structure tree until all traversable element nodes in the web page structure tree are traversed; wherein, during the recursive traversal process, when the length of the text content in one of the element nodes is less than the text length threshold, terminate further traversal of the child element nodes of one of the element nodes, and use the text content in one of the element nodes as the initial web page data; when the length of the text content in one of the element nodes is not less than the text length threshold, traverse the child element nodes of one of the element nodes until the traversal of the last level of child element nodes is completed or the length of the text content in one of the first-level child element nodes is less than the text length threshold.

[0172] In a possible implementation, when the URL screening module 820 selects a target webpage address that meets a preset condition from the plurality of second webpage addresses, it may be used for at least one of the following:

[0173] The second web page address having a target field is determined as a target web page address that meets preset conditions, and the target field includes: a field whose similarity with a preset keyword is greater than a similarity threshold; the second web page address that can be accessed normally is determined as a target web page address that meets preset conditions.

[0174] In a possible implementation, the URL screening module 820 is further configured to:

[0175] The plurality of second web page addresses are subjected to preset processing, the preset processing including at least one of web page address completion, web page address normalization, user agent determination, or web page address verification; the web page address completion includes: performing web page address completion on the second web page address without a subject domain name link; the web page address normalization includes: performing normalization filtering on the second web page address ending with special characters or with a web page view; the user agent determination includes: determining a target user agent for accessing the second web page address from a plurality of user agents; the web page address verification includes: verifying the validity of the second web page address;

[0176] When the URL screening module 820 selects a first target webpage address that meets a preset condition from the plurality of second webpage addresses, it can be used to:

[0177] A first target webpage address that meets a preset condition is selected from the plurality of second webpage addresses that have undergone preset processing.

[0178] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present application, the coupling between modules can be electrical. In addition, the various functional modules in the various embodiments of the present application can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0179] The following will be combined Figure 9 The architecture of an electronic device provided in this application is described.

[0180] In an alternative embodiment, an electronic device is provided, such as Figure 9 As shown, Figure 9 The electronic device 900 shown includes: a processor 901 and a memory 903. The processor 901 and the memory 903 are connected, for example, via a bus 902. Optionally, the electronic device 900 may further include a transceiver 904, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 904 is not limited to one, and the structure of the electronic device 900 does not constitute a limitation on the embodiments of the present application.

[0181] The processor 901 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 901 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0182] The bus 902 may include a path for transmitting information between the above components. The bus 902 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 902 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0183] The memory 903 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0184] The memory 903 is used to store the computer program for executing the embodiments of the present application, and the execution is controlled by the processor 901. The processor 901 is used to execute the computer program stored in the memory 903 to implement the steps shown in the above method embodiments.

[0185] Please refer to Figure 10 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1000 stores program code, which can be called by a processor to execute the method described in the above method embodiment.

[0186] The computer-readable storage medium 1000 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1000 has storage space for program code 1010 for executing any of the method steps described above. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed, for example, in a suitable form.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data acquisition method, characterized in that: include: Based on a pre-configured first webpage address, crawling and obtaining a plurality of second webpage addresses related to the first webpage address; Selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses; Based on the target web page address, obtaining initial web page data; The initial web page data and the prompt text are input into a generative language model so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data. The prompt text is used to prompt the generative language model to analyze the initial web page data.

2. The method according to claim 1, characterized in that The crawling and obtaining a plurality of second web page addresses related to the pre-configured first web page address includes: Using a pre-configured first webpage address as a first-level webpage address, crawling multiple p-th level webpage addresses based on the webpage corresponding to the first-level webpage address, wherein the p-th level webpage address is obtained by performing address resolution based on the content of the webpage corresponding to the p-1-th level webpage address, where 2≤p≤M, and p and M are integers; The multiple p-th level web page addresses are used as multiple second web page addresses.

3. The method according to claim 2, characterized in that The method further comprises: Obtain a web page structure tree, the web page structure tree being used to represent each target web page address and N-level parent web page addresses mapped to each target web page address, wherein the target web page address is obtained by performing address resolution based on the content of the web page corresponding to the first-level parent web page address, and the q-1-th-level parent web page address is obtained by performing address resolution based on the content of the web page corresponding to the q-th-level parent web page address, where 2≤q≤N<M, and q and N are integers; Determining the N-level parent web page addresses corresponding to the target web page address based on the web page structure tree; The step of inputting the initial webpage data and the prompt text into a generative language model, and analyzing the initial webpage data based on the prompt text by the generative language model to obtain target webpage data, includes: The initial web page data, prompt text, and N-level parent web page addresses corresponding to the target web page address from which the initial web page data is obtained are input into a generative language model, so that the generative language model analyzes the initial web page data based on the contents of the web pages corresponding to the prompt text and the N-level parent web page addresses to obtain the target web page data.

4. The method according to claim 1, wherein The acquiring of initial webpage data based on the target webpage address includes: Obtaining a web page structure tree, the web page structure tree including multiple levels of element nodes, wherein the multiple levels of element nodes include a first-level element node representing the first web page address, the first-level element node being connected to child-level element nodes, and a last-level element node being connected to a parent element node. For intermediate-level element nodes between the first-level element node and the last-level element node, the intermediate-level element nodes are connected to a parent element node and a child element node, each element node including text content corresponding to the target web page address, and relationships between the multiple levels of element nodes are determined according to parent-child relationships between the web page addresses corresponding to the element nodes. Get the text length threshold; Based on the text length threshold and the web page structure tree, the text content in at least one element node is obtained, and the text content in the at least one element node is used as the initial web page data, wherein the length of the text content in each element node in the at least one element node is less than the text length threshold.

5. The method according to claim 4, characterized in that The acquiring of text content in at least one element node based on the text length threshold and the webpage structure tree includes: Starting from the first-level element node, recursively traverse each element node in the web page structure tree until all traversable element nodes in the web page structure tree are traversed; Among them, during the recursive traversal process, when the length of the text content in one of the element nodes is less than the text length threshold, further traversal of the child element nodes of one of the element nodes is terminated, and the text content in one of the element nodes is used as the initial web page data; when the length of the text content in one of the element nodes is not less than the text length threshold, the child element nodes of one of the element nodes are traversed until the traversal of the last level of child element nodes is completed or the length of the text content in one of the first level of child element nodes is less than the text length threshold.

6. The method according to any one of claims 1 to 5, characterized in that The step of selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses includes at least one of the following: Determining the second webpage address having a target field as a target webpage address that meets a preset condition, wherein the target field includes: a field having a similarity with a preset keyword greater than a similarity threshold; The second webpage address that can be accessed normally is determined as the target webpage address that meets the preset conditions.

7. The method according to any one of claims 1 to 5, characterized in that Before selecting a target webpage address that meets a preset condition from the plurality of second webpage addresses, the method further includes: performing preset processing on the plurality of second web page addresses, the preset processing including at least one of web page address completion or web page address normalization; the web page address completion including: performing web page address completion on the second web page addresses without subject domain name links; the web page address normalization including: performing normalization filtering on the second web page addresses ending with special characters or with web page views; The selecting a first target webpage address that meets a preset condition from the plurality of second webpage addresses includes: A first target webpage address that meets a preset condition is selected from the plurality of second webpage addresses that have undergone preset processing.

8. The method according to any one of claims 1 to 5, characterized in that The acquiring of initial webpage data based on the target webpage address includes: Determining a target user agent accessing the target web page address from a plurality of user agents; Accessing the webpage corresponding to the target webpage address through the target user agent; Acquire initial web page data in the web page corresponding to the target web page address.

9. A data acquisition device, characterized in that: include: A website address collection module, configured to crawl and obtain a plurality of second web page addresses related to a pre-configured first web page address based on the first web page address; A website address screening module, configured to select a target web page address that meets a preset condition from the plurality of second web page addresses; A data acquisition module, configured to acquire initial web page data based on the target web page address; The initial web page data and the prompt text are input into a generative language model so that the generative language model analyzes the initial web page data based on the prompt text to obtain target web page data. The prompt text is used to prompt the generative language model to analyze the initial web page data.

10. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the one or more programs are configured to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images