General website mirror image and local storage management method and system

Abstract

The invention relates to a general website mirror image and local storage management method and system, and the method comprises the steps: configuring the hierarchical structure of a mirror image website based on the hierarchical number of a target website, and traversing the original network resources of the target website according to the hierarchical sequence of the hierarchical structure; simulating user operation by utilizing a Web automation tool, and capturing target network resources from the original network resources by utilizing an initial capturing strategy to create a corresponding mirror image website; obtaining resource files of all target network resources in the mirror image website, and generating a unique corresponding file identifier; classifying resource files of target network resources in the mirror image website; generating a storage path of the resource file based on a classification result of the resource file and the file identifier; and establishing the path mapping table based on the file identifier and the storage path of the resource file, so that the website resources can be more efficiently and completely copied to the local server through the method, and efficient classified storage and rapid local retrieval of the resource file are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of website data processing, and in particular to a general website mirror and local storage management method and system. Background Art

[0002] With the rapid development of the Internet, websites have become an important platform for obtaining information, services and entertainment. However, website access usually relies on a stable network connection. In some scenarios, such as no network or limited internal network environment, the need to achieve the same display as the original website becomes increasingly important. Under these restrictions, traditional website access methods may be hindered. In addition, for some enterprises and institutions that need to use in the intranet environment, how to access and use website resources without relying on external networks has become an urgent problem to be solved.

[0003] At present, although there are some solutions for website mirroring and offline browsing, they usually face problems such as untimely content updates, incomplete functions, and large differences from the original website experience. These limitations have hindered users' website access experience in environments with no network or limited network access. Summary of the invention

[0004] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] As a first aspect of the present application, in order to solve the technical problems of untimely updating of offline website browsing content, incomplete data, insufficient functions, etc. under network restrictions, the present application provides a general website mirroring and local storage management method, the method comprising:

[0006] Configuring a hierarchical structure of the mirror website based on the number of hierarchies of the target website, and traversing the original network resources of the target website according to the hierarchical order of the hierarchical structure;

[0007] Using a Web automation tool to simulate user operations, using an initial crawling strategy to capture target network resources from the original network resources to create a corresponding mirror website;

[0008] Obtaining the resource file of the target network resource in the mirror website and generating a unique corresponding file identifier;

[0009] Classifying resource files of target network resources in the mirror website;

[0010] Based on the classification result of the resource file and the file identifier, generating a storage path of the mirror website;

[0011] A path mapping table is established based on the file identifier and the storage path of the mirror website.

[0012] As a second aspect of the present application, in order to solve the technical problem of repeated storage or missing content of website resource content in the process of creating a mirror website, the following technical solution is provided:

[0013] Obtaining file identifiers of resource files of all target network resources in the mirror website;

[0014] Querying the path mapping table based on the file identifier whether the file identifier exists;

[0015] If the file identifier exists, ignoring the resource file;

[0016] In the case that the file identifier does not exist, a storage path of the mirror website corresponding to the target network resource is generated.

[0017] As the third aspect of this application, in order to solve the technical problem that the network connection is unstable, the website information cannot be accurately obtained when the network is abnormal, or the website content cannot be updated in time, the following technical solutions are provided:

[0018] When the target network resources are captured from the original network resources using the initial crawling strategy to create the corresponding mirror website, the initial crawling strategy needs to be adjusted in real time based on the normal network traffic.

[0019] The specific steps of adjusting the initial crawling strategy in real time when the network traffic is abnormal include:

[0020] In the process of capturing the target network resources of the target website to create a corresponding mirror website, monitoring the network response of the target website through a first network traffic interception tool;

[0021] When the network condition of the target website is abnormal, the initial crawling strategy of the target network resources is adjusted.

[0022] As a fourth aspect of the present application, a general website mirroring and local storage management system is also provided, and the system specifically includes the following modules:

[0023] A website configuration module, configured to configure a hierarchical structure of a mirror website based on the number of hierarchies of a target website, and traverse the original network resources of the target website according to the hierarchical order of the hierarchical structure;

[0024] A mirror website creation module is used to use a Web automation tool to simulate user operations and use an initial crawling strategy to capture target network resources from the original network resources to create a corresponding mirror website;

[0025] A file identification generation module, used to obtain resource files of all target network resources in the mirror website and generate unique corresponding file identification;

[0026] A classification storage module, used for classifying resource files of target network resources in the mirror website;

[0027] A path storage module, used for generating a storage path of the mirror website based on the classification result of the resource file and the file identifier;

[0028] A mapping table creation module is used to establish a path mapping table based on the file identifier and the storage path of the mirror website.

[0029] As a fifth aspect of the present application, a method for local access to network resources is also provided, and the method specifically includes:

[0030] Get the user's website access request;

[0031] When the website access request is intercepted by the second network traffic interception tool, parsing the website access request to obtain a resource path of the website;

[0032] Acquire a corresponding verification file identifier based on the resource path;

[0033] querying a path mapping table according to the second file identifier, wherein the path mapping table is obtained according to the general website mirroring and local storage management method described above;

[0034] A response to the website access request is returned according to the query result of the path mapping table.

[0035] The specific step of returning a response to the website access request according to the query result of the path mapping table includes:

[0036] querying the path mapping table according to the verification file identifier whether there is a file identifier with the same identifier;

[0037] If the same file identifier does not exist, an error response that the network resource does not exist is returned;

[0038] In the case where the same file identifier exists, the storage path of the resource file corresponding to the file identifier is obtained; the resource files of all target network resources under the storage path are read, and the response of the mirror website is returned.

[0039] Compared with the prior art, the method and system of the present invention have the following beneficial effects:

[0040] 1. It can quickly, accurately and completely copy the content of any website to the local server, which improves the efficiency of creating mirror websites, ensures that the local mirror website is highly consistent with the original website in terms of functions and operations, and solves the problem of limited traditional offline browsing functions;

[0041] 2. The present invention provides a hash mapping and typed file storage algorithm for local management and classification of data captured from mirror websites, achieving efficient classified storage and fast local retrieval, and realizing a technology that can maintain a display and interactive experience consistent with the original website even in an environment without a network or with a limited network, thereby enhancing the user's access experience and access restrictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 is an exemplary flow chart of a general website mirroring and local storage management method described in an embodiment;

[0044] Figure 2 is an exemplary structural diagram of a general website mirror and local storage management system described in the embodiments;

[0045] Figure 3 is an exemplary flow chart of a method for locally accessing network resources described in an embodiment;

[0046] Figure 4 It is a block diagram of the composition of the electronic device described in the embodiment. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0048] Embodiment 1

[0049] Figure 1 is an exemplary flow chart of a general website mirroring and local storage management method described in an embodiment, such as Figure 1 As shown, the general website mirroring and local storage management method specifically includes the following steps:

[0050] S110, configuring a hierarchical structure of the mirror website based on the number of hierarchies of the target website, and traversing the original network resources of the target website according to the hierarchical order of the hierarchical structure.

[0051] In some embodiments, the target website refers to a website that needs to be mirrored, for example: the website of the People's Government of City A (https: / / www.A.gov.cn / ), which contains all web pages under the domain name of the website.

[0052] In some embodiments, the hierarchical structure refers to the hierarchy of the website. For example, when the website of the People's Government of City A (https: / / www.A.gov.cn / ) is opened, it is the first layer. In the first layer, clicking 'Important News (https: / / www. A.gov.cn / yaowen / )' jumps to the important news column of the website of the People's Government of City A, which is the second layer. In the first layer, clicking 'Policy' (https: / / www. A.gov.cn / zhengce / index.htm) jumps to the policy column of the website of the People's Government of City A, which is also the second layer. When you click on a piece of news in 'Important News' or 'Policy', you jump to the third layer of the website, and so on.

[0053] In some embodiments, the user can configure the number of hierarchical structures of the mirrored website according to needs, and the configured hierarchical structure should be less than the original number of hierarchical structures of the target website.

[0054] In some embodiments, the original network resources may refer to a collection of original or existing resource content in the target website. For example, when using a browser to access the website of the People's Government of City A (https: / / www.A.gov.cn / ), the original network resources refer to all resource files displayed on the browser page and all front-end and back-end network requests including the background.

[0055] In some embodiments, the process of traversing the target website may adopt a breadth-first search (BFS) strategy or other strategies such as a depth-first traversal search strategy to gradually traverse the target website content in hierarchical order until a preset hierarchical depth is reached.

[0056] Preferably, a breadth-first search strategy can be used to traverse the target website. Through this strategy, the links of the target website are visited layer by layer to ensure that all pages at the same level are visited, so that the structure of the target website can be fully understood; and its feature of finding the shortest path is used to ensure that the shortest path from the start page to the target page is found, so that specific information or pages can be effectively found; because the breadth-first search strategy visits all nodes at the same level, it can easily achieve parallel processing and speed up the overall access speed. The selection of the breadth-first search strategy has the following advantages: 1) Wide coverage: It can cover a wide area of ​​the website faster, especially when the website structure is complex, which can help quickly understand the overall layout of the website; 2) Avoid repeated visits: It can effectively manage the pages that have been visited and avoid repeated visits to the same page at the same level, thereby improving efficiency; 3) Strong adaptability: It has good adaptability to dynamic websites or websites that are updated in real time, and can discover newly added pages and links in a shorter time.

[0057] In some embodiments, the original network resources of the target website are traversed according to the hierarchical order of the hierarchical structure. Specifically, taking the target website as the website of the People's Government of City A (https: / / www.A.gov.cn / ) as an example, the traversal process can be as follows:

[0058] 1. The network of the People's Government of City A has a total of 5 layers. The mirror website that the user needs to configure is 3 layers. According to the network hierarchy structure of the People's Government of City A, traverse the original network resources of the first 3 layers.

[0059] 2. Use the automated tool to open the homepage of the A City People's Government website (https: / / www.A.gov.cn / ), intercept and save all the original network resources that respond, including source code, image attachments, etc. Furthermore, the operation of the automated tool here can actually simulate the process of a user opening a browser to browse the target website.

[0060] 3. Take out all the external links on the homepage as the second level, then traverse the external links on the second level and click to visit them, intercept and store the original network resources that respond, and parse all the external links from the responses on the second level as the third level.

[0061] 4. The third-level external links are processed in turn using automated tools to intercept and save responses. Since the configuration level is 3, after the third level of processing is completed, no external links will be obtained for deeper processing.

[0062] S120, using a Web automation tool to simulate user operations, using an initial crawling strategy to capture target network resources from original network resources, and creating a corresponding mirror website.

[0063] In some embodiments, a web automation tool refers to a tool that is responsible for automatically opening a network address and capturing web page content while the page is fully loaded. During the loading process, the web automation tool can simulate user operations (such as sliding a window) to trigger and capture dynamically loaded content. Specifically, the web automation tool may include Selenium, Playwright, Puppeteer, etc.

[0064] In some embodiments, original network resources: refers to the original network resources, which refers to the collection of all original resource files in the target website, for example, when a browser is used to access the homepage of the People's Government of City A (https: / / www.A.gov.cn / ), all resources displayed on the browser page and all front-end and back-end network requests including the background. Target network resources refer to the collection of resource files to be captured from the original network resources for creating a mirror website, and the target network resources include at least one resource file. Furthermore, all the traversed web page content will be captured as the resource content of the mirror website, that is, the original network resources of the traversed web pages will all be captured as target network resources and included in the mirror website. In essence, in general, the specific content of the two is the same.

[0065] In some embodiments, the initial capture strategy refers to the steps of capturing target network resources when the network traffic is normal. Specifically:

[0066] 1. Crawl layer by layer: Starting from the homepage of the target website, use the breadth-first search strategy and other methods to crawl the original network resources layer by layer until the set number of layers is reached to ensure that the content of the original network resources in the target website is captured comprehensively and in order.

[0067] 2. Real-time monitoring of network traffic: Real-time monitoring of the network traffic of the target website during the crawling process, real-time capture of original network resources, and complete storage of all content, including dynamically loaded content in the page.

[0068] 3. Automatic processing of tasks: Automatically process capture actions and execute them in order of priority. It will automatically open each web page, wait for the page to be fully loaded, and simulate user operations such as sliding to ensure that dynamic content is also captured.

[0069] S130, obtaining resource files of all target network resources in the mirror website, and generating unique corresponding file identifiers.

[0070] In some embodiments, the target network resource includes at least one resource file. Specifically, the resource file may include different types of files such as pictures, HTML, JavaScript, CSS, and resource paths of a website.

[0071] In some embodiments, the file identifier is a hash value generated corresponding to the resource file of the target network resource. Specifically, the system generates unique file identifiers (for example, using hash values) for each of these resource files and maps them to local storage paths. Therefore, file identifiers of multiple resource files will be generated under the same mirror website.

[0072] Specifically, various resource files (including HTML, CSS, JS, images, etc.) in the mirror website can usually generate corresponding hash values. For example, when a user clicks or the system automatically captures a link or option in the target website, the browser will send a request to the server to obtain the resource files required for the page. The content of the server response may include various types of resource files such as HTML, CSS, JS, images, attachments, etc. These resource files are hashed to generate a unique hash value, that is, a file identifier, which is used to verify the consistency of the resource file or as a unique identifier for the resource file. The generated hash value can be used in scenarios such as resource file deduplication and integrity verification to ensure that the resource file has not been tampered with or lost during transmission and storage.

[0073] In some embodiments, for each resource file captured, a unique hash value is generated as a file identifier. This hash value can be used to check whether the resource file already exists to avoid duplicate storage. The generation of the hash value can be shown as follows:

[0074]

[0075] Among them, F i Represents a resource file. Here, the MD5 message digest algorithm is used to perform hash calculation on the resource file to obtain a unique hash value. The MD5 message digest algorithm (MD5 Message-Digest Algorithm) is a cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure that information transmission is complete and consistent.

[0076] S140, classifying resource files of the target network resources in the mirror website.

[0077] In some embodiments, resource files may include different types of files such as pictures, HTML, JavaScript, CSS, and resource paths of websites. Different resource files are classified according to the type of the resource file. For the specific content of the resource file of the captured target network resource, it is classified and stored according to the data type of the resource file (such as HTML, JavaScript, CSS, pictures, etc.). The purpose is to clearly and completely store all resource files in the target network resource when storing the mirror website content locally, and to manage the data by partition and classification. At the same time, a content hashing mechanism is introduced to reduce the storage of duplicate content and improve storage efficiency.

[0078] S150: Generate a storage path for the resource file based on the classification result of the resource file and the file identifier.

[0079] In some embodiments, resource files are stored in layers according to their types and domain names, and a unique storage path is generated using a hash value. Specifically, the storage path generation formula is as follows:

[0080]

[0081] Among them, Root_Dir is the root directory of local storage; Domain(F i ) indicates the domain name currently being captured, and is used to classify resource files according to network sources; Type (F i ) indicates the resource file type, such as HTML, CSS, JS, pictures, etc.; H(F i ) represents the hash value (i.e., file identifier) ​​generated by the MD5 information digest algorithm; Ext represents the file extension.

[0082] S160: Establish a path mapping table based on the file identifier and the storage path of the mirror website.

[0083] In some embodiments, a path mapping table is introduced to manage all resource files in the mirror website. The correspondence between the hash value (i.e., file identifier) ​​of the resource file and the storage path facilitates fast search and retrieval of files. Specifically, the path mapping table structure can be shown as follows:

[0084]

[0085] Among them, H(F i ) represents the hash value of the resource file (i.e., the file identifier); P(F i ) indicates the storage path of the resource file.

[0086] In some embodiments, the beneficial effects of obtaining the hash value of the resource file and establishing the path mapping table by the above method are:

[0087] 1. Resource files have unique identifiers: The system generates an independent file identifier for each resource file, and each identifier corresponds to a specific resource file (image, HTML file, CSS file, etc.). As long as the file identifier of any resource file can be queried, it means that the resource file has been mirrored to the local computer.

[0088] 2. Inference of the overall existence of the website: In the process of mirroring a website, the system usually traverses the entire website structure and downloads the necessary resource files. Therefore, querying any resource file (such as the HTML file of a web page or the main CSS file) means that the mirrored content of the website exists locally, or at least some pages or resource files can be displayed normally.

[0089] 3. Priority of key resources: Generally, the system will give priority to mirroring the main HTML pages and important static resources (such as the homepage and public styles). Therefore, as long as these main resource files exist, it can be proved that the website mirror is valid locally.

[0090] Furthermore, when the system processes a user request, querying the identifier of any resource file can indeed prove that the target network resource of the mirror website exists in the local storage.

[0091] In some embodiments, the mirror website can also be updated regularly according to the needs of the user. The user can customize the update frequency, time node and target website to be updated. According to the time interval predetermined by the user, the update task of the mirror website is automatically started. Each time the task is executed, the latest content of the target website can be automatically detected and synchronized, including static resources and dynamically generated content, to ensure that the data of the mirror website is consistent with the content of the target website.

[0092] Furthermore, the update task of the mirror website can adopt the incremental update method, synchronizing only the parts that have changed since the last mirroring. This method can improve the update efficiency and reduce unnecessary data transmission and storage consumption.

[0093] Furthermore, every update operation of the mirror site will be recorded in detail in the log, which is convenient for subsequent review and troubleshooting. At the same time, system monitoring ensures that any anomalies in the update process can be discovered and handled in a timely manner. During the update process, the system takes measures to ensure the consistency and integrity of the data. Any update operation is strictly verified to prevent data corruption or loss.

[0094] Embodiment 2

[0095] This embodiment is based on the general website mirror and local storage management method described in the first embodiment, and further improves and optimizes the method after the path mapping table is constructed.

[0096] In some embodiments, according to the general website mirroring and local storage management method provided by the present application, after the path mapping table is constructed, the following steps may also be included:

[0097] S210, obtaining file identifiers of resource files of all target network resources in the mirror website;

[0098] S220, querying whether the file identifier exists in the path mapping table based on the file identifier;

[0099] S230, if the file identifier exists, ignoring the resource file;

[0100] S240: If the file identifier does not exist, generate a storage path of the resource file corresponding to the target network resource.

[0101] The above method can provide the following beneficial effects: comprehensive and systematic collection of website content is ensured, and repeated collection of resource files can be avoided.

[0102] Embodiment 3

[0103] This embodiment optimizes the initial crawling strategy used in S120 in the general website mirroring and local storage management method described in Example 1. The initial crawling strategy described in step S120 is a method for capturing target network resources from the original network resources based on normal network traffic. In actual applications, when network traffic is abnormal, the initial crawling strategy can be adjusted in real time. The specific steps are as follows:

[0104] In the process of capturing the target network resources of the target website to create a corresponding mirror website, monitoring the network response of the target website through a first network traffic interception tool;

[0105] When the network condition of the target website is abnormal, the initial crawling strategy of the target network resources is adjusted.

[0106] In some embodiments, the entire process will continuously monitor the specified port to ensure that the service does not miss any resources responded by the other server during operation. The specific steps of adjusting the initial crawling strategy of the target network resource include:

[0107] 1. Automatically retry requests: When the system encounters a failure during the crawling process (such as network interruption or server abnormality), it will automatically try to resend the request. The system will set the number of retries, such as a maximum of three retries.

[0108] 2. Task requeue: If the request still fails after multiple retries, the system will reinsert the crawling task to the end of the task queue so that it can be tried again in the subsequent crawling process. This method can avoid the overall crawling progress being blocked due to individual request failures.

[0109] 3. Use IP proxy: If you encounter frequent request failures, the system can also circumvent some server restrictions on requests by switching IP proxy. This can help the system crawl more effectively on the same website and avoid inaccessibility due to IP blocking.

[0110] 4. Slow down the crawling speed: When crawling fails, the system may appropriately reduce the crawling speed to reduce the pressure on the target server and avoid blocking or denial of service caused by too frequent requests.

[0111] 5. Error logging: The system will record the error information of each crawling failure and generate logs for subsequent analysis. These logs help identify the root cause of the problem and adjust the crawling strategy if necessary.

[0112] The above method can provide the following beneficial effects: by monitoring the network response of the designated port through the network traffic interception tool, not only can the dynamic content update of the original website be captured and recorded in time, but also the network status can be monitored during the data transmission process to ensure efficient and stable data capture. In addition, by monitoring the network request and response, the system can realize real-time feedback on abnormal situations, so as to adjust the capture strategy in time to improve the integrity and accuracy of the mirror website. This active monitoring method ensures a high degree of consistency between the mirror website and the original website. In addition, all intercepted response data are then classified and stored according to the file storage and management protocol to ensure the orderliness and retrievability of the data.

[0113] Embodiment 4

[0114] Figure 2 is an exemplary structural diagram of a general website mirror and local storage management system described in the embodiment, such as Figure 2 As shown, a general website mirroring and local storage management system specifically includes the following modules:

[0115] A website configuration module 210, configured to configure a hierarchical structure of the target website based on the number of hierarchies of the target website, and traverse the original network resources of the target website according to the hierarchical order of the hierarchical structure;

[0116] A mirror website creation module 220 is used to use a Web automation tool to simulate user operations and use an initial crawling strategy to capture target network resources from the original network resources to create a corresponding mirror website;

[0117] The file identification generation module 230 is used to obtain resource files of all target network resources in the mirror website and generate unique corresponding file identifications;

[0118] The classification storage module 240 is used to classify the resource files of the target network resources in the mirror website;

[0119] A path storage module 250, configured to generate a storage path for the resource file based on the classification result of the resource file and the file identifier;

[0120] The mapping table creation module 260 is used to establish a path mapping table based on the file identifier and the storage path of the resource file.

[0121] In some embodiments, the file identifier generation module 230 can also be used to obtain the file identifier of the resource files of all target network resources in the mirror website; query whether the file identifier exists in the path mapping table based on the file identifier; if the file identifier exists, ignore the resource file; if the file identifier does not exist, generate the storage path of the resource file corresponding to the target network resource.

[0122] In some embodiments, the mirror website creation module 220 can also be used to monitor the network response of the target website through a first network traffic interception tool during the process of capturing the target network resources of the target website to create a corresponding mirror website; and adjust the initial crawling strategy of the target network resources when the network response of the target website is abnormal.

[0123] Embodiment 5

[0124] Figure 3 FIG. 1 is an exemplary flow chart of a method for locally accessing network resources described in an embodiment. Figure 4 As shown, the method for local access to network resources specifically includes the following steps:

[0125] S310, obtaining a website access request from a user.

[0126] In some embodiments, the user's website access request may refer to a URL, including protocol, domain name, path, etc., such as https: / / www.A.gov.cn / . Further, the system may identify a specific resource file by parsing the URL in the request, and then use the mapping table to query and obtain the locally stored image file.

[0127] S320, when the website access request is intercepted by the second network traffic interception tool, parse the website access request to obtain the resource path of the website.

[0128] In some embodiments, after the interception tool captures the request, it will extract the URL (including protocol, domain name and resource path) in the request, such as https: / / www.A.gov.cn / info / 1021 / 1076962.htm. The system will parse the structure of the URL and decompose it into the following parts: - Protocol (for example: http) - Domain name (for example: tianshui.gov.cn) - Path (for example: / info / 1021 / 1076962.htm).

[0129] S330: Obtain a corresponding verification file identifier based on the resource path of the website.

[0130] In some embodiments, the verification file identifier may refer to a hash value generated corresponding to the website resource path. The specific method for obtaining the verification file identifier may refer to the detailed process of steps S130 to S160, which will not be repeated here.

[0131] S340, querying a path mapping table according to the verification file identifier, wherein the path mapping table is obtained according to the method described in the general website mirroring and local storage management method.

[0132] In some embodiments, based on the path mapping table obtained by the method described in the general website mirror and local storage management method, the path mapping table can represent the correspondence between resource files and local storage paths, wherein the resource file includes the resource path of the mirror website, and the path mapping table identifies each resource file by generating a hash value of the resource file. The specific query steps are as follows:

[0133] 1. Generate a hash value of the resource path: The system will extract the resource path portion from the requested URL and use the path to generate a unique hash value (for example, using the MD5 algorithm). Assuming the requested resource path is / info / 1021 / 1076962.htm, the system will calculate the hash value of MD5 (" / info / 1021 / 1076962.htm") and generate a unique verification file identifier.

[0134] 2. Search for hash value: The system will verify the hash value corresponding to the file identifier and search it in the path mapping table. If the same hash value is found, it means that the resource file already exists in the local storage, and the system can directly obtain the corresponding file path.

[0135] S350: Return a response to the website access request according to the query result of the path mapping table.

[0136] The S350 specifically includes the following steps:

[0137] S351, querying the address mapping table based on the verification file identifier to see whether there is a file identifier with the same identifier;

[0138] S352: if the same identifier does not exist, an error response that the network resource does not exist is returned;

[0139] S353, when there is an identical identifier, obtain the storage path of the resource file corresponding to the file identifier; read the resource files of all target network resources under the storage path, and return a response of the mirror website page.

[0140] Specifically, if the corresponding hash value is found in the path mapping table, the system will return the storage path corresponding to the resource file. The system then reads the contents of the resource files of all target network resources under the storage path, and prepares to return the mirror website to the user; the system constructs an HTTP response based on the read resource file content, that is, the response of the mirror website, and returns the mirror website to the user's browser. This process simulates the interaction with the original website server, but in fact all operations are completed locally; after the user's browser receives the HTTP response, it will display the locally mirrored page or resource file, and the user experience is almost the same as visiting an online website.

[0141] Through the above method, the following beneficial effects can be achieved: Since all operations are performed locally and no real requests are sent to the outside world, the risk of external attackers invading or intercepting data is greatly reduced. Secondly, since there is no external network connection, sensitive data will not be transmitted outside the intranet, which further enhances the confidentiality of the data. In addition, by processing network requests locally and returning responses, data is avoided from being sent to a remote server, which not only reduces the possibility of data leakage, but also reduces dependence on external services.

[0142] Embodiment 6

[0143] like Figure 4 As shown, this embodiment also provides an electronic device, which may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that this figure is exemplary, and other types of structures may be used to supplement or replace this structure to achieve data extraction, report generation, communication or other functions.

[0144] like Figure 4 As shown, the electronic device may further include: an input unit 43, a display unit 44 and a power supply. It is worth noting that the electronic device does not necessarily have to include Figure 4 In addition, electronic devices may also include Figure 4 For components not shown, reference may be made to the prior art.

[0145] The processor 41 is sometimes also called a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 41 receives inputs and controls the operations of various components of the electronic device.

[0146] The memory 42 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices, and may store the configuration information of the processor 41, instructions executed by the processor 41, recorded table data and other information. The processor 41 may execute the program stored in the memory 42 to implement information storage or processing, etc. In one embodiment, the memory 42 also includes a buffer memory, i.e., a buffer, to store intermediate information.

[0147] The input unit 43 is used, for example, to provide the processor 41 with text data to be annotated. The display unit 44 is used to display various results in the processing process, such as input text data, output results of the adjustment module, output results of the CRF module, etc. The display unit can be, for example, an LCD display, but the present invention is not limited thereto. The power supply 45 is used to provide power to the electronic device.

[0148] An embodiment of the present invention further provides a computer-readable instruction, wherein when the instruction is executed in an electronic device, the program causes the electronic device to execute the operation steps included in the method of the present invention.

[0149] An embodiment of the present invention further provides a storage medium storing computer-readable instructions, wherein the computer-readable instructions enable an electronic device to execute the operation steps included in the method of the present invention.

[0150] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0152] Those of ordinary skill in the art will appreciate that the modules of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the constituent modules and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0154] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A general website mirroring and local storage management method, characterized in that: include: Configuring a hierarchical structure of the mirror website based on the number of hierarchies of the target website, and traversing the original network resources of the target website according to the hierarchical order of the hierarchical structure; Using a Web automation tool to simulate user operations, using an initial crawling strategy to capture target network resources from the original network resources to create a corresponding mirror website; Obtaining the resource file of the target network resource in the mirror website and generating a unique corresponding file identifier; Classifying resource files of target network resources in the mirror website; Based on the classification result of the resource file and the file identifier, generating a storage path of the resource file; A path mapping table is established based on the file identifier and the storage path of the resource file.

2. The general website mirroring and local storage management method according to claim 1, characterized in that: The original network resources refer to the collection of all original resource files in the target website; the target network resources refer to the collection of resource files to be captured from the original network resources; the target network resources include at least one resource file.

3. The general website mirroring and local storage management method according to claim 1, characterized in that: The method further comprises: Obtaining file identifiers of resource files of all target network resources in the mirror website; Querying the path mapping table based on the file identifier whether the file identifier exists; If the file identifier exists, ignoring the resource file; In the case that the file identifier does not exist, a storage path of the resource file corresponding to the target network resource is generated.

4. The general website mirroring and local storage management method according to claim 1, characterized in that: When the target network resources are captured from the original network resources using the initial crawling strategy to create the corresponding mirror website, the initial crawling strategy needs to be adjusted in real time based on the normal network traffic. The specific steps of adjusting the initial crawling strategy in real time when the network traffic is abnormal include: In the process of capturing the target network resources of the target website to create a corresponding mirror website, monitoring the network response of the target website through a first network traffic interception tool; When the network response of the target website is abnormal, the initial crawling strategy of the target network resource is adjusted.

5. The general website mirroring and local storage management method according to claim 1, characterized in that: The file identifier is a hash value generated corresponding to the resource file of the target network resource.

6. A general website mirror and local storage management system, used to implement the method described in any one of claims 1 to 5, characterized in that: include: A website configuration module, configured to configure a hierarchical structure of a mirror website based on the number of hierarchies of a target website, and traverse the original network resources of the target website according to the hierarchical order of the hierarchical structure; A mirror website creation module is used to use a Web automation tool to simulate user operations and use an initial crawling strategy to capture target network resources from the original network resources to create a corresponding mirror website; A file identification generation module is used to obtain the resource file of the target network resource in the mirror website and generate a unique corresponding file identification; A classification storage module, used for classifying resource files of target network resources in the mirror website; A path storage module, used for generating a storage path of the resource file based on the classification result of the resource file and the file identifier; The mapping table creation module is used to establish a path mapping table based on the file identifier and the storage path of the resource file.

7. A method for local access to network resources, characterized in that: include; Get the user's website access request; When the website access request is intercepted by the second network traffic interception tool, parsing the website access request to obtain a resource path of the website; Acquire a corresponding verification file identifier based on the resource path; querying a path mapping table according to the verification file identifier, wherein the path mapping table is obtained by the method according to any one of claims 1 to 5; A response to the website access request is returned according to the query result of the path mapping table.

8. The method for local access to network resources according to claim 7, characterized in that: The specific step of returning a response to the website access request according to the query result of the path mapping table includes: querying the path mapping table according to the verification file identifier whether there is a file identifier with the same identifier; If the same file identifier does not exist, an error response that the network resource does not exist is returned; In the case where the same file identifier exists, the storage path of the resource file corresponding to the file identifier is obtained; the resource files of all target network resources under the storage path are read, and the response of the mirror website is returned.

9. A computer-readable storage medium comprising computer-readable instructions, characterized in that: When the computer readable instructions are executed, the processor performs the operations in any one of the methods of claims 1-5.

10. An electronic device, characterized in that: The device comprises: Memory, which stores program instructions; A processor is connected to the memory and executes program instructions in the memory to implement the steps in the method according to any one of claims 1 to 5.

Images