A website cloning method, device, equipment and medium

By using a web crawler in a headless browser to obtain and save website information, and dynamically generating response pages to achieve a complete clone of the website, the problem of being unable to clone dynamically generated pages in existing technologies is solved.

CN115269952BActive Publication Date: 2026-03-27HANGZHOU DBAPPSECURITY CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-01
Publication Date
2026-03-27

AI Technical Summary

Technical Problem

Existing website cloning tools cannot effectively clone pages dynamically generated by JavaScript, resulting in incomplete or inadequate website cloning.

Method used

A blank window is created in a headless browser using a pre-set crawler program to obtain the current page information and redirect page information of the target website, and save them to a pre-set database. The corresponding response page is then generated based on the database information to build a clone website.

Benefits of technology

It achieves a complete clone of the target website, can crawl dynamically generated redirect pages from the front end and obtain response information, and solves the problem that existing technologies cannot clone dynamically generated pages.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115269952B_ABST
    Figure CN115269952B_ABST
Patent Text Reader

Abstract

The application discloses a website cloning method and device, equipment and medium, and relates to the technical field of information. The method comprises the following steps: creating a blank window in an interfaceless browser and loading a target website through a preset crawler program; obtaining current page information of the target website to obtain static element information in the current page, and saving the static element information into a preset database; crawling jump page information corresponding to the current page of the target website, and saving the jump page information into the preset database; sending a request to the target website according to the jump page information in the preset database, receiving response information returned by the target website, and then saving the response information corresponding to the jump page information into the preset database, so as to construct a cloned website for the target website according to data in the preset database. The above scheme can achieve the purpose of completely cloning a website when cloning the website.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, in particular to a website cloning method, device, equipment and medium. BACKGROUND

[0002] At present, the traditional website cloning tool cloning method is to download the directory structure of the website to the local to build all directories in a recursive manner. However, this method can only obtain static HTML pages, pictures and other files of the website. In another existing website cloning technology, static files downloaded are parsed to obtain more links, but currently front-end technologies mostly use dynamic scripts to generate pages to display the pages. The traditional page cloning tool cannot obtain the new pages generated by the jump of the request links generated by the JavaScript scripts. In addition, if the same link has different request parameters, the website displays the response pages according to different request parameters, and the traditional website cloning tool also cannot clone the pages generated by different parameters, resulting in that the cloned website is not comprehensive and complete. In summary, how to generate corresponding response pages according to different requests when cloning a website to achieve the purpose of completely cloning the website needs to be further solved. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a website cloning method, device, equipment and medium, which can generate corresponding response pages according to different requests when cloning a website to achieve the purpose of completely cloning the website. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a website cloning method, comprising:

[0005] creating a blank window in an interfaceless browser and loading a target website through a preset crawler program;

[0006] obtaining current page information of the target website to obtain static element information in the current page, and saving the static element information into a preset database;

[0007] obtaining jump page information corresponding to the current page of the target website, and saving the jump page information into the preset database;

[0008] sending a request to the target website according to the jump page information in the preset database, receiving response information returned by the target website, and then saving the response information corresponding to the jump page information into the preset database to construct a cloned website for the target website according to the data in the preset database.

[0009] Optionally, the obtaining the current page information of the target website to obtain static element information in the current page, and saving the static element information into a preset database comprises:

[0010] obtaining the current page information of the target website through a web page attribute interface in the preset crawler program;

[0011] parsing elements in the current page according to the current page information, extracting a label field of a static element link in the current page to obtain static element information, and saving the static element information into a preset database.

[0012] Optionally, the crawling jump page information corresponding to the current page of the target website and saving the jump page information into the preset database comprises:

[0013] the first jump page information and the second jump page information corresponding to the current page of the target website are crawled, and the first jump page information and the second jump page information are saved into the preset database.

[0014] Optionally, the first jump page information corresponding to the current page of the target website is crawled and saved into the preset database, comprising:

[0015] the active jump request sent by the interfaceless browser is intercepted through a preset hijacking function;

[0016] the first jump page information corresponding to the active jump request in the target website is crawled, and the first jump page information is saved into the preset database.

[0017] Optionally, the second jump page information corresponding to the current page of the target website is crawled and saved into the preset database, comprising:

[0018] listening to a jump trigger event in the current page, and simulating the jump trigger event through a preset interface in the preset crawler program;

[0019] the second jump page information corresponding to the jump trigger event in the target website is crawled, and the second jump page information is saved into the preset database.

[0020] Optionally, the sending a request to the target website according to the jump page information in the preset database comprises:

[0021] reading the preset database to obtain a web link, a request method and a request parameter of the jump page corresponding to the jump page information;

[0022] According to the web link, the request method and the request parameter of the jump page, a request is sent to the target website.

[0023] Optionally, the method further comprises:

[0024] When the request message for the cloned website is acquired, corresponding response information is searched in the preset database according to the web link, the request method and the request parameter corresponding to the request message, and the response information is replied to the sender of the request message.

[0025] In a second aspect, the present application discloses a network cloning device, comprising:

[0026] A website loading module is configured to create a blank window in the interfaceless browser and load a target website through a preset crawler program.

[0027] A static element information acquisition module is configured to acquire current page information of the target website to obtain static element information in the current page, and save the static element information into a preset database.

[0028] A jump page information acquisition module is configured to crawl jump page information corresponding to the current page of the target website, and save the jump page information into the preset database.

[0029] A response information acquisition module is configured to send a request to the target website according to the jump page information in the preset database, receive response information returned by the target website, and then save the response information corresponding to the jump page information into the preset database, so as to construct a cloned website for the target website according to data in the preset database.

[0030] In a third aspect, the present application discloses an electronic device, comprising:

[0031] A memory is configured to save a computer program.

[0032] A processor is configured to execute the computer program to realize the steps of the website cloning method disclosed above.

[0033] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to realize the steps of the website cloning method disclosed above.

[0034] In the website cloning, the application first creates a blank window in the interfaceless browser through a preset crawler program, loads a target website, obtains the current page information of the target website to obtain the static element information in the current page, and saves the static element information into a preset database. The application crawls the jump page information corresponding to the current page of the target website, and saves the jump page information into the preset database. Finally, the application sends a request to the target website according to the jump page information in the preset database, receives the response information returned by the target website, and saves the response information corresponding to the jump page information into the preset database, so as to construct a cloned website of the target website according to the data in the preset database. As can be seen, in the website cloning, the application first crawls the target website through a preset crawler program to obtain the current page information of the target website and obtain the static element information in the current page through the current page information. Then, the application crawls the jump page information in the current page, sends a request to the target website according to the obtained jump page information, receives the response information returned by the target website, and saves the response information corresponding to the jump page information into the preset database, so as to construct a cloned website of the target website through the information in the preset database. Thus, in the website cloning, the application crawls all element information of the target website through a dynamic crawler, can also crawl the jump page dynamically generated by the front end and obtain the response information returned by the target website, so as to solve the problem that the static file is parsed and the static element of the target website is obtained in the prior art, and the new page generated by the jump of the request link dynamically generated by the JavaScript script cannot be cloned, thereby achieving the purpose of completely cloning the target website. In summary, the application can generate a corresponding response page according to different requests to achieve the purpose of completely cloning the website. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative effort.

[0036] Figure 1 A website cloning method flowchart provided by the present application;

[0037] Figure 2 A specific website cloning method flowchart provided by the present application;

[0038] Figure 3 A website cloning device structure schematic diagram provided by the present application;

[0039] Figure 4 An electronic device structure diagram is provided. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0041] The conventional website cloning tool cloning method is to download the directory structure of the website to the local to build all directories in a recursive manner. However, this method can only obtain static HTML pages, pictures and other files of the website. In another existing website cloning technology, static files are parsed to obtain more links, but currently, front-end technologies mostly use dynamic scripts to generate pages to display the pages. The traditional page cloning tool cannot obtain the new page generated by the jump of the request link generated by the JavaScript script. In addition, if the same link has different request parameters, the website displays the response page according to different request parameters, and the traditional website cloning tool cannot clone the pages generated by different parameters, resulting in that the cloned website is not comprehensive and complete. Therefore, the present application provides a website cloning method which can generate corresponding response pages according to different requests when cloning a website to achieve the purpose of completely cloning the website.

[0042] The embodiment of the present application discloses a website cloning method, as shown in Figure 1 The method comprises the following steps:

[0043] Step S11: creating a blank window in an interfaceless browser through a preset crawler program and loading a target website.

[0044] In the embodiment, the interfaceless browser is a browser without a graphical user interface, that is, a headless browser. A blank window is created in the interfaceless browser through a preset crawler program, and the website address of the target website is loaded through the blank window to obtain the webpage of the target website. Through the above technical solution, the target website is crawled through the crawler program, so that the static element information and the jump page information in the target website can be crawled subsequently to realize complete cloning of the target website.

[0045] Step S12: obtaining the current page information of the target website to obtain the static element information in the current page, and saving the static element information into a preset database.

[0046] In the embodiment, the current page of the target website is acquired, elements in the current page are parsed, static element information in the current page is acquired, and the static element information is saved into a preset database. It should be noted that the static element information includes but is not limited to static element connection, static element request method and static element request parameter; and the preset database is a database for storing information crawled by the target website. By the above technical solution, the static element information is saved into the preset database, so as to clone the target website through the static element information.

[0047] Step S13: Crawling jump page information corresponding to the current page of the target website, and saving the jump page information into the preset database.

[0048] In the embodiment, the jump page information corresponding to the current page of the target website is crawled, and the jump page information is saved into the preset database. It should be noted that the jump page information includes but is not limited to jump page link, jump page request method and jump page request parameter. By the above technical solution, the jump page information corresponding to the current page of the target website is dynamically crawled, and the jump page information is saved, so as to send a request to the target website through the jump page information in the future.

[0049] Step S14: Sending a request to the target website according to the jump page information in the preset database, receiving response information returned by the target website, and then saving the response information corresponding to the jump page information into the preset database, so as to construct a cloned website of the target website according to data in the preset database.

[0050] In the embodiment, the request is sent to the target website according to the jump page information in the preset database, specifically including: reading the preset database to obtain the web link, request method and request parameter of the jump page corresponding to the jump page information; and sending a request to the target website according to the web link, request method and request parameter of the jump page. Further, the response information returned by the target website is received, and the response information corresponding to the jump page information is saved into the preset database.

[0051] In a specific embodiment, when a request message for a clone website is acquired, the corresponding response information is found in the preset database according to the webpage link, the request method and the request parameter corresponding to the request message, and the response information is replied to the sender of the request message. Through the above technical solution, the website simulation process is started, when the request initiated by the client is received, the response information corresponding to the request can be obtained by querying the preset database according to the link, the request method and the request parameter and the response information is replied to the request sender, so as to achieve the purpose of completely cloning the target website.

[0052] It can be seen that, in the website cloning process of the embodiment, the target website is first crawled by the preset crawler program to obtain the current page information of the target website and obtain the static element information in the current page through the current page information, then the jump page information in the current page is crawled, the jump page information is sent to the target website according to the obtained jump page information, and the response information returned by the target website is received, and the response information corresponding to the jump page information is saved in the preset database, so as to construct the clone website of the target website through the information in the preset database. Therefore, in the website cloning process of the present application, all element information of the target website is crawled through the dynamic crawler, and the jump page dynamically generated by the front end can also be crawled and the response information returned by the target website can be obtained, so that the problem that the new page generated by the jump of the request link dynamically generated by the JavaScript script cannot be cloned by parsing and obtaining the static elements of the target website in the prior art is solved in the website cloning process, thereby achieving the purpose of completely cloning the target website. In summary, the present application can generate corresponding response pages according to different requests to achieve the purpose of completely cloning the website.

[0053] Referring to Figure 2 The embodiment of the present application discloses a specific website cloning method, which is further described and optimized compared with the previous embodiment.

[0054] Step S21: creating a blank window in the interfaceless browser through the preset crawler program and loading the target website.

[0055] In the embodiment, when the preset crawler program is started, the puppeteer in the preset crawler program is called to start an interfaceless browser, and a blank window is created to load the target website address to be crawled. Through the above technical solution, the target website is crawled by the crawler program, so that the static element information and the jump page information in the target website can be crawled in the subsequent process to realize the complete cloning of the target website.

[0056] Step S22: Obtain the current page information of the target website to obtain the static element information in the current page, and save the static element information into a preset database.

[0057] In the embodiment, the current page information of the target website is obtained to obtain the static element information in the current page, and the static element information is saved into a preset database, specifically including: obtaining the current page information of the target website through a web page attribute interface in the preset crawler program; parsing the elements in the current page according to the current page information, and extracting the label field of the static element link in the current page to obtain the static element information, and then saving the static element information into a preset database. In a specific embodiment, after the browser loads the web page, the puppeteer can obtain the page information through the page attribute, parse all elements in the page, extract the label field of the static element link, and store the web page link, request method and request parameter information of the static element into the database.

[0058] Step S23: Crawl the first jump page information and the second jump page information corresponding to the current page of the target website, and save the first jump page information and the second jump page information into the preset database.

[0059] In the embodiment, the first jump page information corresponding to the current page of the target website is crawled and saved into the preset database, specifically including: intercepting the active jump request sent by the interfaceless browser through a preset hijacking function; crawling the first jump page information in the target website corresponding to the active jump request, and saving the first jump page information into the preset database. The second jump page information corresponding to the current page of the target website is crawled and saved into the preset database, specifically including: listening to the jump trigger event in the current page, and simulating the jump trigger event through a preset interface in the preset crawler program; crawling the second jump page information in the target website corresponding to the jump trigger event, and saving the second jump page information into the preset database. Specifically, a hook hijacking function is set to intercept the active jump request sent by the browser, to prevent the page from being accidentally jumped or closed. The intercepted request also needs to be crawled again for the first jump link. After the function is hijacked, the first jump web page link, request method and request parameter information are inserted into the database through the self-defined interface; all trigger events contained in the web page are listened to, the trigger events are simulated through the puppeteer interface, the changes of the web page after the events are triggered are collected, and if the event triggers the jump to a new web page, the content triggering the event is inserted into the database as the second jump web page link, request method and request parameter information.

[0060] Step S24: sending a request to the target website according to the first jump page information and the second jump page information in the preset database, receiving response information returned by the target website, and then saving the response information corresponding to the jump page information in the preset database to construct a cloned website for the target website according to data in the preset database.

[0061] It can be seen that in the embodiment, the current page information of the target website is obtained to obtain static element information in the current page, and the first jump page information and the second jump page information corresponding to the current page of the target website are crawled to obtain information corresponding to dynamic jump pages in the target website, all requests and response information of the website are obtained by using a dynamic crawler, various mouse and keyboard events in the website are automatically triggered to obtain dynamic web page information, and all requests and responses are only saved in a database file, so that a large number of folders generated by a traditional website cloning tool are avoided, file management is facilitated, and the purpose of completely cloning the target website is achieved.

[0062] Referring to Figure 3 The embodiment of the application discloses a website cloning device, which comprises:

[0063] The website loading module 11 is configured to create a blank window in the interfaceless browser and load the target website by using a preset crawler program.

[0064] The static element information acquisition module 12 is configured to obtain current page information of the target website to obtain static element information in the current page, and save the static element information in a preset database.

[0065] The jump page information acquisition module 13 is configured to crawl jump page information corresponding to the current page of the target website, and save the jump page information in the preset database.

[0066] The response information acquisition module 14 is configured to send a request to the target website according to the jump page information in the preset database, receive response information returned by the target website, and then save the response information corresponding to the jump page information in the preset database to construct a cloned website for the target website according to data in the preset database.

[0067] It can be seen that, in the website cloning, the target website is first crawled by the preset crawler program to obtain the current page information of the target website, and the static element information in the current page is obtained through the current page information, then the jump page information in the current page is crawled, the jump page information obtained is used to send a request to the target website and receive response information returned by the target website, and the response information corresponding to the jump page information is saved in the preset database, so as to construct the cloned website of the target website through the information in the preset database. Thus, in the website cloning, all element information of the target website is crawled through the dynamic crawler, and the jump page dynamically generated by the front end and the response information returned by the target website can also be crawled, so that in the website cloning, the problem that the static file is parsed and the static element of the target website is obtained in the prior art, and the new page generated by the jump of the request link dynamically generated by the JavaScript script cannot be cloned, so that the purpose of cloning the target website completely is achieved. In summary, the application can generate corresponding response pages according to different requests to achieve the purpose of cloning the website completely.

[0068] In some specific embodiments, the static element information obtaining module 12 specifically comprises:

[0069] The page information obtaining unit is configured to obtain the current page information of the target website through a web page attribute interface in the preset crawler program.

[0070] The static element information saving unit is configured to parse the elements in the current page according to the current page information, extract the label field of the static element link in the current page to obtain the static element information, and save the static element information in the preset database.

[0071] In some specific embodiments, the jump page information obtaining module 13 is specifically configured to: crawl the first jump page information and the second jump page information corresponding to the current page of the target website, and save the first jump page information and the second jump page information in the preset database.

[0072] In some specific embodiments, the jump page information obtaining module 13 specifically comprises:

[0073] The request interception unit is configured to intercept the active jump request sent by the no-interface browser through a preset hijacking function.

[0074] The first jump page information obtaining unit is configured to crawl the first jump page information in the target website corresponding to the active jump request, and save the first jump page information in the preset database.

[0075] In some embodiments, the jump page information obtaining module 13 specifically comprises:

[0076] The event listening unit is configured to listen to a jump trigger event in a current page and simulate the jump trigger event through a preset interface in the preset crawler program.

[0077] The second jump page information obtaining unit is configured to crawl second jump page information in the target website corresponding to the jump trigger event and save the second jump page information in the preset database.

[0078] In some embodiments, the response information obtaining module 14 specifically comprises:

[0079] The database reading unit is configured to read the preset database to obtain a web link, a request method and a request parameter of the jump page corresponding to the jump page information.

[0080] The request sending unit is configured to send a request to the target website according to the web link, the request method and the request parameter of the jump page.

[0081] In some embodiments, the website cloning apparatus further comprises:

[0082] The information searching module is configured to, when a request message for a cloned website is obtained, search for corresponding response information in the preset database according to the web link, the request method and the request parameter corresponding to the request message, and reply the response information to a sender of the request message.

[0083] Figure 4 The electronic device 20 provided by the embodiment of the present application is shown. The electronic device 20 specifically can further comprise at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the website cloning method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiment can be an electronic computer.

[0084] In the embodiment, the power supply 23 is configured to provide voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0085] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc. The storage mode can be temporary storage or permanent storage.

[0086] The operating system 221 is used to manage and control various hardware devices on the electronic device 20 and the computer program 222 can be Windows Server, Netware, Unix, Linux, etc. In addition to including the computer program capable of completing the website cloning method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0087] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the website cloning method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0088] Finally, it should be noted that in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0089] The above provides a detailed introduction to the website cloning method, device, equipment and medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples in this document. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A website cloning method, characterized in that, include: The web crawler creates a blank window in a headless browser and loads the target website. Obtain the current page information of the target website to obtain static element information in the current page, and save the static element information to a preset database; The system crawls the redirect page information corresponding to the current page of the target website and saves the redirect page information to the preset database. The step of crawling the redirect page information corresponding to the current page of the target website and saving the redirect page information to the preset database includes: crawling the first redirect page information and the second redirect page information corresponding to the current page of the target website and saving the first redirect page information and the second redirect page information to the preset database; The step of crawling the first redirect page information corresponding to the current page of the target website and saving the first redirect page information to the preset database includes: intercepting the active redirect request sent by the headless browser through a preset hijacking function; crawling the first redirect page information in the target website corresponding to the active redirect request and saving the first redirect page information to the preset database; the preset hijacking function is a HOOK hijacking function; The step of crawling the second redirect page information corresponding to the current page of the target website and saving the second redirect page information to the preset database includes: listening to the redirect trigger event in the current page and simulating the redirect trigger event through a preset interface in the preset crawler program; crawling the second redirect page information in the target website corresponding to the redirect trigger event and saving the second redirect page information to the preset database. A request is sent to the target website based on the redirect page information in the preset database, and the response information returned by the target website is received. Then, the response information and the redirect page information are saved to the preset database to construct a clone website for the target website based on the data in the preset database.

2. The website cloning method according to claim 1, characterized in that, The step of obtaining the current page information of the target website to obtain static element information in the current page, and saving the static element information to a preset database, includes: The current page information of the target website is obtained through the webpage attribute interface in the preset crawler program; Based on the current page information, the elements in the current page are parsed, and the tag fields of the static element links in the current page are extracted to obtain static element information. Then, the static element information is saved to a preset database.

3. The website cloning method according to any one of claims 1 to 2, characterized in that, Sending a request to the target website based on the redirect page information in the preset database includes: Read the preset database to obtain the webpage link, request method, and request parameters of the redirected page corresponding to the redirected page information; A request is sent to the target website based on the webpage link, request method, and request parameters of the redirected page.

4. The website cloning method according to claim 3, characterized in that, Also includes: When a request message for a cloned website is received, the corresponding response information is searched in the preset database according to the webpage link, request method, and request parameters corresponding to the request message, and the response information is sent back to the sender of the request message.

5. A network cloning device, characterized in that, include: The website loading module is used to create a blank window in a headless browser and load the target website using a preset crawler program; The static element information acquisition module is used to acquire the current page information of the target website to obtain the static element information in the current page, and save the static element information to a preset database; The step of crawling the redirect page information corresponding to the current page of the target website and saving the redirect page information to the preset database includes: crawling the first redirect page information and the second redirect page information corresponding to the current page of the target website and saving the first redirect page information and the second redirect page information to the preset database; The step of crawling the first redirect page information corresponding to the current page of the target website and saving the first redirect page information to the preset database includes: intercepting the active redirect request sent by the headless browser through a preset hijacking function; crawling the first redirect page information in the target website corresponding to the active redirect request and saving the first redirect page information to the preset database; the preset hijacking function is a HOOK hijacking function; The step of crawling the second redirect page information corresponding to the current page of the target website and saving the second redirect page information to the preset database includes: listening to the redirect trigger event in the current page and simulating the redirect trigger event through a preset interface in the preset crawler program; crawling the second redirect page information in the target website corresponding to the redirect trigger event and saving the second redirect page information to the preset database. The redirect page information acquisition module is used to crawl the redirect page information corresponding to the current page of the target website and save the redirect page information to the preset database; The response information acquisition module is used to send a request to the target website according to the redirect page information in the preset database, receive the response information returned by the target website, and then save the response information and the redirect page information corresponding to each other to the preset database, so as to construct a clone website for the target website according to the data in the preset database.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the website cloning method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when executed by a processor, the computer program implements the steps of the website cloning method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Method and device for crawling dynamic webpage

    CN106844486A

  • Website access method, device and system

    CN109861951A

  • Crawling method and device of crawler

    CN110020044A