Website snapshot playback method, program product, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a website snapshot playback method, a program product, electronic equipment and a storage medium, and relates to the technical field of cloud security. The method comprises the steps that URL access is called, a URL address is requested, and a response result is obtained; judging an access type according to the response result and a preset strategy library; and if the access is local access, executing local playback based on the HAR file. According to the method, the local storage and playback of the webpage request are realized based on the standard HTTP archiving format, the network request frequency is reduced, the website monitoring service efficiency and the resource utilization rate are improved, and the problems of low website monitoring service efficiency and low resource utilization rate caused by frequent website access in the existing method are solved.

Description

Technical Field

[0001] The present application relates to the field of cloud security technology, and in particular to a website snapshot playback method, a program product, an electronic device, and a storage medium. Background Art

[0002] Among the security services provided in the cloud, website monitoring services mainly conduct remote monitoring of websites in different dimensions. The monitoring business involves operations such as crawling and comparing web page content. In this process, due to factors such as network bandwidth resources, website authorization, and website load capacity, frequent network requests to obtain web page content will inevitably encounter various problems, such as limited access, slow access, and excessive pressure on the target website affecting normal business. In addition, traditional website security monitoring technology generally builds a website baseline through real-time crawling comparison, screenshots, etc. For most websites that do not change frequently, frequent crawling and comparison is a waste of resources for both the service provider and the customer. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a website snapshot playback method, program product, electronic device and storage medium, which realize local storage and playback of web page requests based on the standard HTTP archive format, reduce the frequency of network requests, improve website monitoring business efficiency and resource utilization, and solve the problems of low website monitoring business efficiency and resource utilization caused by frequent website visits in existing methods.

[0004] The present application provides a website snapshot playback method, which is applied to website monitoring services. The method includes:

[0005] Call URL access, request URL address, and obtain response result;

[0006] Determine the access type according to the response result and a preset policy library;

[0007] If it is local access, local playback is performed based on the HAR file.

[0008] In the above implementation process, local storage and playback of web page requests are realized based on the standard HTTP archive format, which reduces the frequency of network requests, effectively reduces unnecessary Internet access requests, reduces the occupation of bandwidth resources, improves website monitoring business efficiency and resource utilization, and solves the problem of low website monitoring business efficiency and resource utilization caused by frequent website visits in existing methods.

[0009] Furthermore, before the step of determining the access type according to the response result and a preset policy library, the method further includes:

[0010] Set the URL-based access type parameters in the policy library.

[0011] In the above implementation process, the policy library is used to construct access policies to determine whether the access type is local access or Internet access, and to achieve localized playback of website requests and automatic switching of Internet access.

[0012] Further, the determining the access type according to the response result and a preset policy library includes:

[0013] Obtaining a URL access type parameter from the policy library according to the URL address;

[0014] If it is determined that the cache is enabled based on the URL access type parameter, the Hash value of the URL address is calculated, and the local HAR file and the update time of the HAR file are obtained according to the Hash value;

[0015] The access type is determined according to the local HAR file and the update time.

[0016] In the above implementation process, the access type determination includes two elements, namely, whether the HAR file already exists and the update time.

[0017] Further, the determining the access type according to the local HAR file and the update time includes:

[0018] If the local HAR file exists, and the difference between the current time and the update time is less than the cache validity period, it is a local access.

[0019] In the above implementation process, if both factors are met, that is, the HAR file already exists and the update time meets the requirements, it is local access.

[0020] Further, the determining the access type according to the local HAR file and the update time includes:

[0021] If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache validity period, it is an Internet access.

[0022] In the above implementation process, if one of the two elements is not met, that is, the HAR file does not exist or the update time does not meet the requirements, it is Internet access.

[0023] Furthermore, if the access is local, local playback is performed based on the HAR file, including:

[0024] Starts a simulated browser, loads the HAR file, and performs local playback based on the URL request.

[0025] In the above implementation process, local playback is based on the local HAR file and combines with the simulated browser to execute the URL access request.

[0026] Furthermore, the method further comprises:

[0027] If it is Internet access, Internet access is performed and the access process is saved as a local HAR file. The HAR file name is calculated by the Hash value corresponding to the URL. If the HAR file already exists, the HAR file update operation is performed.

[0028] In the above implementation process, if it is Internet access, it is saved as a local HAR file during the execution process for subsequent local access, thereby reducing frequent Internet requests and the occupation of broadband resources.

[0029] The embodiment of the present application also provides a website snapshot playback device, which includes:

[0030] The access request module is used to call URL access, request URL address, and obtain response results;

[0031] An access type determination module, used to determine the access type according to the response result and a preset policy library;

[0032] The local playback execution module is used to execute local playback based on the HAR file if it is a local access.

[0033] The embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned website snapshot playback method is implemented.

[0034] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned website snapshot playback method.

[0035] The embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the above-mentioned website snapshot playback method is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flowchart of a website snapshot playback method provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a method for implementing website snapshot playback based on a HAR package provided in an embodiment of the present application;

[0039] Figure 3 A flowchart of access type determination provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of website monitoring provided in an embodiment of the present application;

[0041] Figure 5 A structural block diagram of a website snapshot playback device provided in an embodiment of the present application.

[0042] icon:

[0043] 100 - access request module; 200 - access type determination module; 300 - local playback execution module. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0045] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0046] Please see Figure 1 , Figure 1 A flowchart of a website snapshot playback method provided in an embodiment of the present application, which can be applied to website monitoring services, combined with cloud-based website security monitoring technology, to achieve localized response to web page requests, and to use the standard HTTP archive format (HTTP Archive format, referred to as HAR, a JSON-formatted archive file format, mostly used to record the interaction process between web browsers and websites. The file extension is usually .har, IE, Chrome, Firefox, all of which support the HAR format) and a simulated browser (Playwright, an open source automation library for browser testing and network crawling) to achieve localized storage and playback of web page snapshots, support content monitoring services, reduce network request frequency, and improve website monitoring business efficiency.

[0047] A web page snapshot is a snapshot of a website at a certain moment in the process of web page access, which is stored through certain technical means. It is a snapshot of the web page at a certain moment, referred to as a web page snapshot. In this application, the web page snapshot is saved in the form of a HAR file.

[0048] The method specifically comprises the following steps:

[0049] Step S100: calling URL access, requesting the URL address, and obtaining the response result;

[0050] Specifically, URL access is generally called by the main business process of website monitoring, requesting the URL address and returning the request response result; further, according to business needs, page operations are performed including but not limited to obtaining internal content, page screenshots, obtaining titles, obtaining URL links, etc.

[0051] Step S200: determining the access type according to the response result and a preset policy library;

[0052] Step S300: If it is a local access, local playback is performed based on the HAR file.

[0053] Local playback, that is, URL access requests are based on request services provided by local HAR files, which can be accessed offline. The local playback process needs to be combined with a simulated browser.

[0054] like Figure 2 As shown in FIG. 1 , it is a flow chart of a method for implementing website snapshot playback based on a HAR package, which, based on the above embodiment, further includes:

[0055] Before the step of determining the access type according to the response result and the preset policy library, a URL-based access type parameter is set in the policy library.

[0056] The policy library is used to configure and manage the settings of access type parameters for URLs, including whether to enable caching and the effective duration of the cache. It is maintained in the form of a UI interface. The definition of the policy can be customized according to different dimensions such as global, domain name, second-level domain name, single page, customer assets, custom scenarios, etc. The specific policy management capabilities are not described in detail in this application. As a user, the policy library provides capabilities including: the ability to obtain the corresponding URL access type parameters based on the URL, and further expand the business-related access types according to the business type.

[0057] like Figure 3 As shown in FIG. 1 , it is a flow chart of access type determination, and step S200 specifically includes the following steps:

[0058] Step S201: Obtaining a URL access type parameter from the policy library according to the URL address;

[0059] Step S202: If it is determined based on the URL access type parameter that the cache is enabled, the Hash value of the URL address is calculated, and a local HAR file and an update time of the HAR file are obtained according to the Hash value;

[0060] Step S203: Determine the access type according to the local HAR file and the update time.

[0061] Specifically, if the local HAR file exists, and the difference between the current time and the update time is less than the cache validity period, it is a local access.

[0062] If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache validity period, it is an Internet access.

[0063] If it is Internet access, Internet access is performed and the access process is saved as a local HAR file. The HAR file name is calculated by the Hash value corresponding to the URL. If the HAR file already exists, the HAR file update operation is performed.

[0064] Get the URL access type parameters {whether to enable cache, cache validity period} from the policy library according to the URL address. First, determine whether to enable cache. If it is enabled, further calculate the hash value of the URL address, and obtain the local HAR file and the update time of the HAR file based on the hash value. If the HAR file does not exist, perform Internet access; if the HAR file exists, take the difference between the current time and the update time (in minutes). If the difference is less than the cache validity period, it means that the local HAR file is valid, and perform local playback; if the difference is greater than the cache validity period, it means that the local HAR file is invalid, and perform Internet access.

[0065] Local playback process: First, start the simulated browser (Playwright), load the HAR file, and finally execute the local access request according to the URL, that is, local playback; during local playback, there is no need to access the Internet, and the request response process can be completed in offline mode.

[0066] The Internet access process first starts the simulated browser (Playwright), calls the browser to perform Internet access according to the URL, and directly accesses the target site. At the same time, the access process can be saved as a local HAR file. The HAR file name is calculated by the hash value corresponding to the URL. If the HAR file already exists, the file update operation is performed.

[0067] Figure 2After completing the request access based on local playback or Internet access, further business-related page operations can be performed, including but not limited to: obtaining internal content, page screenshots, obtaining titles, obtaining URL links, etc.

[0068] For both local playback and Internet access, the page operation after completing the request is consistent, and the response time is greatly reduced; unnecessary Internet requests are reduced through strategies, effectively reducing time consumption and resource usage.

[0069] like Figure 4 As shown, it is a schematic diagram of website monitoring. This method can be used as an adaptation layer for URL access requests to provide two modes of URL access: local playback and Internet access, thereby reducing bandwidth resource usage.

[0070] This method can be used for tampering monitoring, Trojan horse monitoring, sensitive word monitoring, etc. Among them, tampering monitoring specifically includes the following steps:

[0071] Step 11: In the website monitoring business, send a website<aaa.cn> The remote tampering monitoring task is dispatched by the unified service. For example, the specific page is http: / / 1.aaa.cn / , which requires web page access and information capture.

[0072] Step 12: Execute the website crawling request, and request the access type policy parameters from the policy management service according to http: / / 1.aaa.cn / . The result is returned: {true, 60}, which corresponds to: enable caching, and the cache validity period is 60 minutes.

[0073] Step 13: Based on the URL http: / / 1.aaa.cn / , calculate the Hash value: 037CC658E15A474FC8F9DF45E7EA76DF, and check whether the HAR file exists based on the Hash value. If the HAR file does not exist (not cached), perform Internet access.

[0074] Step 14: Internet access includes: starting a browser, requesting URL access, and saving / updating files. Specifically, the steps are as follows:

[0075] playwright = sync_playwright() / / Initialize the object;

[0076] browser = playwright.start().chromium.launch(headless = True) / / Start the browser;

[0077] context = browser.new_context() / / Create a new context;

[0078] page = context.new_page() / / Create a new page;

[0079] page.goto(url) / / Access the URL through the Internet request;

[0080] page.screenshot(path=screenshot path) / / screenshot operation;

[0081] page.content() / / Operation to obtain web page content;

[0082] Step 16: After completing this Internet access, save the HAR file as: 037CC658E15A474FC8F9DF45E7EA76DF.har. At the same time, complete the business-related operations of obtaining page content and taking screenshots to complete this access request.

[0083] Trojan monitoring includes the following steps:

[0084] Step 21: In the website monitoring business, send a website<aaa.cn> The remote Trojan horse monitoring task is dispatched through a unified service. For example, the specific page is http: / / 1.aaa.cn / , which requires web page access and information capture.

[0085] Step 22: Execute the website crawling request, and request the access type policy parameters from the policy management service according to http: / / 1.aaa.cn / . The result is returned: {true, 60}, which corresponds to: enable caching, and the cache validity period is 60 minutes.

[0086] Step 23: Based on the URL http: / / 1.aaa.cn / , calculate the hash value: 037CC658E15A474FC8F9DF45E7EA76DF. Find out whether the HAR file exists based on the hash value. The HAR file (037CC658E15A474FC8F9DF45E7EA76DF.har) exists. Further obtain the HAR file update time (2024-10-28 15:30:00) and the current time (2024-10-28 15:35:00). The difference between the update time and the current time is 5 minutes, which is less than the cache validity period of 60 minutes. Perform local playback.

[0087] Step 24: Local playback includes: starting the browser, loading the HAR file, and requesting local playback. Specifically, the steps are as follows:

[0088] playwright = sync_playwright() / / Initialize the object;

[0089] browser = playwright.start().chromium.launch(headless = True) / / Start the browser;

[0090] context = browser.new_context() / / Create a new context;

[0091] page = context.new_page() / / Create a new page;

[0092] page.route_from_har(HAR file path, update = True, not_found = "abort") / / local playback, read from HAR file;

[0093] page.goto(url) / / Access the URL through the Internet request;

[0094] page.screenshot(path=screenshot path) / / screenshot operation;

[0095] page.content() / / Operation to obtain web page content;

[0096] Step 25: This access meets the policy requirements and accesses the local cache file through local playback. At the same time, the business-related operations of obtaining page content and taking screenshots are completed to complete this access request.

[0097] The specific steps for sensitive word monitoring are as follows:

[0098] Step 31: In the website monitoring business, send a website<aaa.cn> The remote sensitive word monitoring task is dispatched by the unified service. For example, the specific page is http: / / 1.aaa.cn / , which requires web page access and information crawling.

[0099] Step 32: Execute the website crawling request, and request the access type policy parameters from the policy management service according to http: / / 1.aaa.cn / . The result is returned: {true, 60}, which corresponds to: enable caching, and the cache validity period is 60 minutes.

[0100] Step 33: According to the URL http: / / 1.aaa.cn / , calculate the Hash value: 037CC658E15A474FC8F9DF45E7EA76DF, and find out whether the HAR file exists according to the Hash value. The HAR file (037CC658E15A474FC8F9DF45E7EA76DF.har) exists. Further obtain the HAR file update time (2024-10-28 15:30:00) and the current time (2024-10-28 16:35:00). The difference between the update time and the current time is 65 minutes, which is greater than the cache validity period of 60 minutes, indicating that the current file has expired, and perform Internet access.

[0101] Step 34: Internet access includes: starting a browser, requesting URL access, and saving / updating files. The specific steps are the same as step 14 and will not be repeated here.

[0102] Step 31: Assuming that it takes 1 minute to complete this request, after completing this Internet access, update the HAR file: 037CC658E15A474FC8F9DF45E7EA76DF.har, and modify the HAR file update time to: (2024-10-28 16:36:00); further, complete the business-related operations of obtaining page content and taking screenshots to complete this access request.

[0103] Based on the above embodiment, the policy attributes can be further extended. Specifically, the service type field can distinguish and identify the cache validity period corresponding to different monitoring services, such as the same URL: the tampering monitoring validity period is 10 minutes, the Trojan horse monitoring validity period is 30 minutes, and the sensitive word monitoring validity period is 60 minutes.

[0104] In the above embodiment, website monitoring can share local cache files for multiple services. For example, when performing tampering monitoring, Trojan horse monitoring, and sensitive word monitoring, the locally cached HAR files can be shared, which can further reduce the frequency of Internet access.

[0105] This application effectively reduces unnecessary Internet access requests, reduces bandwidth resource usage, and improves resource utilization through policy definition and local playback. By combining local playback with Internet access, the local playback response time is much shorter than the Internet access method, which improves the execution efficiency of website monitoring tasks.

[0106] The present application also provides a website snapshot playback device, such as Figure 5 FIG. 1 is a structural block diagram of a website snapshot playback device, which is applied to a website snapshot playback method. The device includes but is not limited to:

[0107] The access request module 100 is used to call URL access, request the URL address, and obtain the response result;

[0108] An access type determination module 200, configured to determine the access type according to the response result and a preset policy library;

[0109] The local playback execution module 300 is used to execute local playback based on the HAR file if it is a local access.

[0110] The specific implementation process of each of the above modules has been specifically described in the method embodiment and will not be repeated here.

[0111] The embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned website snapshot playback method is implemented.

[0112] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned website snapshot playback method.

[0113] The embodiment of the present application further provides a readable storage medium, wherein the readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the above-mentioned website snapshot playback method is executed.

[0114] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0116] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0119] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A website snapshot playback method, characterized in that: Applied to website monitoring services, the method includes: Call URL access, request URL address, and obtain response result; Determine the access type according to the response result and a preset policy library; If it is local access, local playback is performed based on the HAR file.

2. The website snapshot playback method according to claim 1, characterized in that: Before the step of determining the access type according to the response result and the preset policy library, the method further includes: Set the URL-based access type parameters in the policy library.

3. The website snapshot playback method according to claim 1, characterized in that: The determining the access type according to the response result and a preset policy library includes: Obtaining a URL access type parameter from the policy library according to the URL address; If it is determined that the cache is enabled based on the URL access type parameter, the Hash value of the URL address is calculated, and the local HAR file and the update time of the HAR file are obtained according to the Hash value; The access type is determined according to the local HAR file and the update time.

4. The website snapshot playback method according to claim 3, characterized in that: The determining the access type according to the local HAR file and the update time includes: If the local HAR file exists, and the difference between the current time and the update time is less than the cache validity period, it is a local access.

5. The website snapshot playback method according to claim 3, characterized in that: The determining the access type according to the local HAR file and the update time includes: If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache validity period, it is an Internet access.

6. The website snapshot playback method according to claim 1, characterized in that: If it is a local access, local playback is performed based on the HAR file, including: Starts a simulated browser, loads the HAR file, and performs local playback based on the URL request.

7. The website snapshot playback method according to claim 1, characterized in that: The method further comprises: If it is Internet access, Internet access is performed and the access process is saved as a local HAR file. The HAR file name is calculated by the Hash value corresponding to the URL. If the HAR file already exists, the HAR file update operation is performed.

8. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the website snapshot playback method according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the website snapshot playback method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the website snapshot playback method according to any one of claims 1 to 7 is executed.

Images