A visual web crawler method, system, device, and medium

Abstract

A visual web crawler method, system, device, and medium are disclosed, relating to the field of information acquisition technology. The method includes the following steps: an initialization step: obtaining an initial page and at least one topic keyword; a page acquisition step: obtaining a first page from the initial page and adding it to the first-level queue of the crawling queue; a page identification step: performing a simulated operation on the first page to determine whether the DOM structure of the first page has changed before and after the simulated operation; a topic identification step: if so, determining whether a second page is a topic-related page based on the topic keyword; a queue addition step: if so, adding the second page to the second-level queue of the crawling queue; and repeatedly executing the page acquisition step, page identification step, and topic identification step to determine the crawling queue until the loop termination condition is triggered. By adopting the technical solution provided in this application, the crawling capability of the visual web crawler is effectively improved.

Description

Technical Field

[0001] This application relates to the field of information acquisition technology, and in particular to a visual web crawler method, system, device and medium. Background Technology

[0002] Visual web crawlers are web crawling tools based on a visual interface, designed to allow users to quickly build and run their own web crawlers through simple operations, thereby achieving automated collection and organization of data on the Internet.

[0003] While visual web crawlers offer many advantages in lowering the barrier to entry and improving operational efficiency, their data crawling capabilities are often limited by the templates and rules preset by their developers. Consequently, visual web crawlers may not be able to correctly parse and crawl some complex website structures and dynamic pages, making them unsuitable for some highly customized web crawling tasks. Summary of the Invention

[0004] To enhance the customized web crawling capabilities of visual web crawlers, this application provides a visual web crawler method, system, device, and medium.

[0005] In a first aspect, this application provides a visual web crawler method, the method comprising the following steps:

[0006] Initialization steps: Obtain initialization parameters, which include the initial page and at least one theme keyword;

[0007] Page acquisition steps: Obtain at least one first page from the initial page, and add all first pages to the first-level queue of the crawling queue;

[0008] Page identification steps: Perform simulated operations on all first pages and determine whether the DOM structure of the first page has changed before and after the simulated operations;

[0009] Topic identification steps: If yes, it means that the first page has been redirected to the second page after the simulation operation. Based on the topic keywords, determine whether the second page is a topic-related page.

[0010] Queue addition step: If so, add the second page to the second-level queue of the queue to be crawled;

[0011] The page acquisition step, the page discrimination step, and the topic discrimination step are executed repeatedly to determine the i-th level queue of the queue to be crawled, until the loop termination condition is triggered, and the topic data crawling is completed.

[0012] By adopting the above technical solution, the simulated operation of the page allows the visual crawler to mimic human clicks. Compared with the traditional method of obtaining URLs through crawling, the simulated operation is better suited to dynamic and complex pages, which is conducive to improving the crawling ability of the visual crawler. On the other hand, the next-level pages obtained through the simulated operation are subject-specific. Sub-pages that fail the subject-specific judgment are not added to the crawling list. This ensures that the web pages crawled by the visual crawler are highly relevant to the given subject keywords, which is conducive to improving the subject content crawling ability of the visual crawler.

[0013] Optionally, after simulating the operation on all of the first pages and determining whether the DOM structure of the first pages has changed before and after the simulation operation, the method further includes:

[0014] If not, it means that after the simulation operation, the first page jumps to the third page at the same level as the first page, and the third page is added to the first-level queue of the crawling queue.

[0015] By adopting the above technical solution, if the DOM result of the page does not change after the simulation operation, it means that the page is still at the same level as the current page after the simulation operation. Adding the page at the same level as the current page to the first level queue of the crawling queue, and crawling the page at the same level again in the future, is beneficial to expanding the data crawling range of the visual crawler.

[0016] Optionally, after determining whether the second page is a topic-related page based on the topic keywords, the method further includes: if not, then not adding the second page to the second-level queue of the crawling queue.

[0017] By adopting the above technical solution, pages that are not closely related to the given topic keywords are discarded, ensuring that the crawling results are highly relevant to the given topic keywords.

[0018] Optionally, the simulation operation on all of the first pages specifically includes:

[0019] The first page is parsed to obtain the page information of the first page, which includes webpage structure information, webpage event information, and webpage content information.

[0020] The page information of the first page is input into a preset simulation operation model, and the simulated operation action is output. The simulated operation action includes the action type and the action target.

[0021] The first page is simulated based on the simulated operation actions.

[0022] By adopting the above technical solution, when the visual crawler enters a new page, it analyzes the page information, locates interactive events on the page based on the page information, simulates human browsing behavior to perform simulated operations on the interactive events, thereby obtaining more page information, or finding the sub-pages of the page through interactive events.

[0023] Optionally, after determining whether the second page is a topic-related page based on the topic keywords, the specific steps include:

[0024] Obtain the page information of the second page;

[0025] The simulation operation model is trained based on the page information of the second page and the judgment result of whether the second page is a topic-related page.

[0026] By adopting the above technical solution, the simulation operation model is fed back based on the topic relevance of the new page entered through the simulation operation, ensuring the accuracy of the simulation operation and making the new page obtained through the simulation operation as highly relevant to the topic keywords as possible.

[0027] Optionally, determining whether the second page is a topic-related page based on the aforementioned topic keywords specifically includes:

[0028] Using Word2Vec, extract the page features of the second page based on the page information of the second page;

[0029] Input the page features of the second page into a preset topic relevance analysis model to calculate the topic relevance of the second page;

[0030] Determine whether the topic relevance of the second page is greater than a set threshold;

[0031] If so, then the second page is a page related to the topic.

[0032] A second aspect of this application provides a visual web crawler system, the system comprising the following modules:

[0033] An initialization module is used to obtain initialization parameters, which include an initial page and at least one topic keyword; a page acquisition module is used to obtain at least one first page from the initial page and add all the first pages to the first-level queue of the crawling queue;

[0034] The page discrimination module is used to perform simulated operations on all the first pages and determine whether the DOM structure of the first page has changed before and after the simulated operation.

[0035] The topic identification module is used to determine whether the second page is a topic-related page based on the topic keywords;

[0036] The queue addition module is used to add the second page to the second-level queue of the queue to be crawled;

[0037] The loop execution module is used to repeatedly execute the page acquisition step, the page discrimination step, and the topic discrimination step to determine the i-th level queue of the queue to be crawled, until the loop termination condition is triggered to complete the topic data crawling.

[0038] In a third aspect of this application, an electronic device is provided;

[0039] A fourth aspect of this application provides a computer-readable storage medium;

[0040] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0041] 1. By simulating human operation, the visual crawler can better adapt to dynamic and complex pages, while also improving its crawling capabilities.

[0042] 2. By conducting topic relevance analysis, we can ensure that the pages to be crawled in the crawling queue are highly relevant to the topic keywords, which helps to improve the topic data crawling capabilities of the visual crawler.

[0043] 3. Optimize the simulation operation model based on the correlation analysis results, so that the generated simulation operation actions can more accurately find interactive events to enter the next level of topic-related pages, ensuring the high topic relevance of the pages in the crawling queue. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a visual web crawling method provided in an embodiment of this application.

[0045] Figure 2 This is a schematic diagram of the structure of a visual crawler system disclosed in an embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0047] Explanation of reference numerals in the attached diagram: 201, Initialization module; 202, Page acquisition module; 203, Page discrimination module; 204, Topic discrimination module; 205, Queue addition module; 206, Loop execution module; 300, Electronic device; 301, Processor; 302, Communication bus; 303, User interface; 304, Network interface; 305, Memory. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0049] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0050] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0051] Reference Figure 1 This application provides a visual web crawler method, which specifically includes the following steps:

[0052] Initialization steps: Obtain initialization parameters;

[0053] Specifically, the initialization parameters include topic keywords and an initial page. The visual crawler is configured based on these parameters. The topic keywords are set by the user and are used to describe the data the user wishes to obtain, providing a thematic basis for the desired data. Multiple topic keywords are allowed. The initial page is also provided by the user. For better acquisition of topic data, a portal page or navigation homepage with numerous subpages is preferred.

[0054] Page retrieval steps: Retrieve at least one first page from the initial page and add all first pages to the first-level queue of the crawling queue;

[0055] Specifically, the first page contained in the initial page is obtained through the URL of the initial page. There can be multiple first pages. The obtained first pages are added to the first-level queue of the crawling queue.

[0056] The crawling queue is the path used by the visual web crawler to crawl web data. This queue contains multiple levels of queues; as the queue number increases, the visual web crawler progresses from the initial page to subsequent sub-pages. It's important to note that the visual web crawler will not crawl the data from the initial page it first obtains; instead, it will use multiple first pages as the starting point for data crawling.

[0057] Page identification steps: Perform simulated operations on all first pages and determine whether the DOM structure of the first page has changed before and after the simulated operation;

[0058] Specifically, after obtaining multiple first pages, the visual crawler will simulate operations on the first pages. The simulated operation refers to inputting code representing human manipulation into the page, thereby simulating human manipulation. The simulated operation includes operations such as clicking, turning pages, scrolling, and inputting. Through the simulated operation, the page can be made to make specific changes according to the simulated operation, thereby obtaining the sub-pages or sibling pages of the current page to achieve further data crawling.

[0059] The simulated operation on the first page first involves preloading all first pages to obtain the page information contained within them. Specifically, the page information includes the webpage structure information, webpage event information, and webpage content information. Among them, webpage event information refers to elements on the page that have events, such as specific HTML tag elements like "href", "acito-n", "click", "src", and "hover". Users browse and obtain more text information in dynamic webpages or jump to other webpages through the URLs contained in the page by responding to these events.

[0060] Page features are extracted from the page information. These features describe both the specific structure of the page and its topic relevance. The page features are then input into a pre-set simulation operation model to generate simulated operation actions that are applied to the page. These simulated operation actions include the action type and the action target. The page is then simulated based on these simulated operation actions.

[0061] By simulating operations on the page, we can obtain information about structural or content changes. We can then identify the DOM structure of the first page after the simulation, thus determining whether the page underwent structural or content changes. If the DOM structure changes, it means the page has been redirected from the current page to a sub-page through the simulation. If the DOM structure remains unchanged, it means the page is still on the current page or has redirected to a third page at the same level as the current page. Since the third page has the same hierarchy as the first page, it is added to the first-level queue of the crawling queue for subsequent data crawling.

[0062] For the identification of the DOM structure of a page, traditional program identification methods such as DOM selectors or regular expressions can be used. In one feasible embodiment of this application, the page DOM structure can also be identified based on neural networks. The above methods are all existing technologies and will not be described in detail here.

[0063] Theme identification steps: If the DOM structure of the first page changes before and after the simulation operation, determine whether the second page is a theme-related page based on the theme keywords;

[0064] Specifically, if the DOM structure of the first page changes before and after the simulation operation, it means that after the simulation operation, the user is redirected from the first page to the second page, which is a subpage of the first page. At this point, the second page is used for topic identification. Based on the topic keywords provided by the user, the topic relevance of the second page to the topic keywords is calculated to determine whether the second page is a topic-related page.

[0065] Calculating topic relevance first requires extracting page features based on page information. In one feasible embodiment of this application, word vectors of the page information are calculated using Word2Vec, and the calculated word vectors are used as the page information of the second page. The extracted page information is input into a preset topic relevance analysis model, which outputs the topic relevance of the second page. The topic relevance analysis model can be constructed based on a recurrent neural network.

[0066] When the topic relevance of the second page is greater than the set threshold, it means that the second page is a topic-related page. Conversely, when the topic relevance of the second page is less than the set threshold, it means that the second page is not a topic-related page. In this case, the second page will not be added to the crawling queue to avoid affecting the data finally crawled by the visualization crawler.

[0067] After determining the theme of the page, the process also includes optimizing the simulated operation based on the theme determination results. Since the second page is a secondary page accessed through the simulated operation from the first page, and the simulated operation steps are automatically output by the simulation operation model, it is necessary to ensure that the simulated operation on the first page leads to its secondary page as much as possible, and that this secondary page is thematically relevant to the topic keywords. A feedback training mechanism is introduced for the simulation operation model. Specifically, when the simulated operation action generated by the model causes the first page to respond and enter the second page, and the second page is thematically relevant, the simulation operation model is given strong positive feedback to reward this behavior; when the simulated operation action causes the first page to respond and enter the second page, but the second page is not thematically relevant, the simulation operation model is given positive feedback; when the simulated operation action causes the first page to respond and enter the third page, the simulation operation model is given negative feedback to punish this behavior. By establishing a feedback training mechanism, as the visual crawler crawls deeper, the generated simulated operation actions for manipulating the page will more accurately redirect the current page to a secondary page, thereby further improving the data crawling capability of the visual crawler and enabling more in-depth mining of topic-related data.

[0068] Queue addition steps: If the second page is a topic-related page, then add the second page to the second-level queue of the crawling queue;

[0069] Specifically, when the second page is a topic-related page, the visualization crawler will crawl the topic-related data contained on that page and add the second page to the second-level queue of the crawling queue.

[0070] The loop execution steps are: the page acquisition step, the page discrimination step, and the topic discrimination step are executed in a loop to determine the i-th level queue of the queue to be crawled, until the loop termination condition is triggered, and the topic data crawling is completed.

[0071] Specifically, after obtaining the second page, the page acquisition step, page discrimination step, and theme discrimination step are repeated. That is, the sub-pages contained in the second page are further acquired. For the acquired sub-pages, simulated operations, page discrimination, and theme discrimination are performed to determine whether the sub-pages obtained from the second page can be stored in the list to be crawled.

[0072] Repeat the above process until the loop termination condition is triggered. The loop termination condition can be a user's command to stop crawling, or a condition set internally by the visual crawler, such as when no new secondary pages can be obtained through simulated operations, or when the obtained secondary pages are all irrelevant to the topic.

[0073] Reference Figure 2 This application also provides a visual web crawler system, which specifically includes the following modules:

[0074] Initialization module 201 is used to obtain initialization parameters, including an initial page and at least one topic keyword; page acquisition module 202 is used to obtain at least one first page from the initial page and add all first pages to the first-level queue of the crawling queue.

[0075] Page discrimination module 203 is used to perform simulated operations on all first pages and determine whether the DOM structure of the first page has changed before and after the simulated operation.

[0076] The topic identification module 204 is used to determine whether the second page is a topic-related page based on topic keywords.

[0077] The queue addition module 205 is used to add the second page to the second-level queue of the queue to be crawled;

[0078] The loop execution module 206 is used to repeatedly execute the page acquisition step, the page discrimination step, and the topic discrimination step to determine the i-th level queue of the queue to be crawled, until the loop termination condition is triggered to complete the topic data crawling.

[0079] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0080] This application also discloses an electronic device 300. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device 300 disclosed in an embodiment of this application. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0081] The communication bus 302 is used to enable communication between these components.

[0082] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0083] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0084] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0085] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. (Refer to...) Figure 3 The memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application for a visual web crawling method.

[0086] exist Figure 3 In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 301 can be used to call an application program storing a visual web crawler method in the memory 305. When executed by one or more processors 301, the electronic device 300 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device 305. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage device 305 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage device 305 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0092] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0093] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A visual web crawler method, characterized in that, The method includes the following steps: Initialization steps: Obtain initialization parameters, which include the initial page and at least one theme keyword; Page acquisition steps: Obtain at least one first page from the initial page, and add all first pages to the first-level queue of the crawling queue; Page identification steps: Perform simulated operations on all first pages, specifically including: parsing the first page to obtain page information, which includes webpage structure information, webpage event information, and webpage content information; inputting the page information of the first page into a preset simulated operation model and outputting simulated operation actions, which include action type and action target; performing simulated operations on the first page based on the simulated operation actions; and determining whether the DOM structure of the first page has changed before and after the simulated operation. Topic identification steps: If yes, it means that the first page has jumped to the second page after the simulation operation. Based on the topic keywords, it is determined whether the second page is a topic-related page. Specifically, this includes: extracting the page features of the second page based on the page information of the second page using Word2Vec; inputting the page features of the second page into a preset topic relevance analysis model to calculate the topic relevance of the second page; determining whether the topic relevance of the second page is greater than a set threshold; if yes, the second page is a topic-related page; if no, it means that after the simulation operation, the first page has jumped to a third page at the same level as the first page, and the third page is added to the first-level queue of the crawling queue. Obtain the page information of the second page; perform feedback training on the simulation operation model based on the page information of the second page and the judgment result of whether the second page is a topic-related page, including: when the simulated operation action generated by the simulation operation model causes the first page to enter the second page in response to the simulated operation action and the second page is topic-related, give the simulation operation model strong positive feedback; when the simulated operation action generated by the simulation operation model causes the first page to enter the second page in response to the simulated operation action but the second page is not topic-related, give the simulation operation model positive feedback; when the simulated operation action generated by the simulation operation model causes the first page to enter the third page in response to the simulated operation action, give the simulation operation model negative feedback. Queue addition step: If so, add the second page to the second-level queue of the queue to be crawled; The process of repeatedly executing the page acquisition, page identification, and topic identification steps determines the i-th level queue of the queue to be crawled until the loop termination condition is triggered, thus completing the topic data crawling.

2. The visual web crawler method according to claim 1, characterized in that, After determining whether the second page is a topic-related page based on the aforementioned topic keywords, the process also includes: If not, the second page will not be added to the second-level queue of the crawling queue.

3. A visual web crawler system, characterized in that, The system includes: An initialization module (201) is used to obtain initialization parameters, which include an initial page and at least one topic keyword; The page acquisition module (202) is used to acquire at least one first page from the initial page and add all the first pages to the first-level queue of the crawling queue; The page discrimination module (203) is used to perform simulated operations on all the first pages, specifically including: parsing the first page to obtain the page information of the first page, the page information of the first page including web page structure information, web page event information and web page content information; inputting the page information of the first page into a preset simulation operation model and outputting simulated operation actions, the simulated operation actions including action type and action target; performing simulated operations on the first page based on the simulated operation actions; and determining whether the DOM structure of the first page has changed before and after the simulated operation. The topic identification module (204) is used to determine whether the second page is a topic-related page based on the topic keywords. Specifically, it includes: extracting the page features of the second page based on the page information of the second page using Word2Vec; inputting the page features of the second page into a preset topic relevance analysis model to calculate the topic relevance of the second page; determining whether the topic relevance of the second page is greater than a set threshold; if yes, then the second page is a topic-related page; if no, it means that after the simulation operation, the first page jumps to a third page at the same level as the first page, and the third page is added to the first-level queue of the crawling queue. Obtain the page information of the second page; perform feedback training on the simulation operation model based on the page information of the second page and the judgment result of whether the second page is a topic-related page, including: when the simulated operation action generated by the simulation operation model causes the first page to enter the second page in response to the simulated operation action and the second page is topic-related, give the simulation operation model strong positive feedback; when the simulated operation action generated by the simulation operation model causes the first page to enter the second page in response to the simulated operation action but the second page is not topic-related, give the simulation operation model positive feedback; when the simulated operation action generated by the simulation operation model causes the first page to enter the third page in response to the simulated operation action, give the simulation operation model negative feedback. The queue addition module (205) is used to add the second page to the second-level queue of the queue to be crawled; The loop execution module (206) is used to execute the page acquisition module, the page discrimination module and the topic discrimination module to determine the i-th level queue of the queue to be crawled until the loop termination condition is triggered to complete the topic data crawling.

4. An electronic device, characterized in that, The device includes a processor (301), a memory (305), a user interface (303), and a network interface (304). The memory (305) is used to store instructions. The user interface (303) and the network interface (304) are used to communicate with other devices. The processor (301) is used to execute the instructions stored in the memory (305) to cause the electronic device (300) to perform the method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the steps of the method as described in any one of claims 1-2.

Images