A method, program product, electronic device, and storage medium for website snapshot playback

Abstract

An embodiment of the present application provides a website snapshot playback method, a program product, an electronic device, and a storage medium, which relate to the field of cloud security technologies. The method includes invoking a URL access, requesting a URL address, and obtaining a response result; judging an access type according to the response result and a preset policy library; if it is a local access, performing local playback based on a HAR file. This method realizes the local storage and playback of web page requests based on the standard HTTP archive format, reduces the network request frequency, improves the efficiency of website monitoring services and resource utilization rate, and solves the problem of low efficiency of website monitoring services and low resource utilization rate caused by frequent access to websites in the existing methods.

Description

Technical Field

[0001] This application relates to the field of cloud security technology. Specifically, it relates to a method for website snapshot playback, a program product, an electronic device, and a storage medium. Background Art

[0002] In the security services provided in the cloud, website monitoring services mainly conduct remote monitoring of websites in different dimensions. In the monitoring business, operations such as web content crawling and comparison are involved. In this process, due to factors such as network bandwidth resources, website authorization, and website load capacity, frequently obtaining web content through network requests will inevitably encounter various problems, such as access restrictions, slow access, and excessive pressure on the target website affecting normal business. In addition, traditional website security monitoring technologies generally construct website baselines through methods such as instant crawling and comparison, screenshots, etc. For the situation where most websites do not change frequently, frequent crawling and comparison are a waste of resources for both service providers and customers. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a method for website snapshot playback, a program product, an electronic device, and a storage medium, which realize local storage and playback of web requests based on the standard HTTP archive format, reduce the frequency of network requests, improve the efficiency of website monitoring services and resource utilization, and solve the problem of low efficiency and low resource utilization of website monitoring services caused by frequent access to websites in existing methods.

[0004] The embodiments of this application provide a method for website snapshot playback, which is applied to website monitoring services. The method includes:

[0005] Invoke a URL access, request a URL address, and obtain a response result;

[0006] Judge the access type according to the response result and a preset policy library;

[0007] If it is a local access, perform local playback based on the HAR file.

[0008] In the above implementation process, realizing local storage and playback of web requests based on the standard HTTP archive format reduces the frequency of network requests, effectively reduces unnecessary Internet access requests, reduces the occupation of bandwidth resources, improves the efficiency of website monitoring services and resource utilization, and solves the problem of low efficiency and low resource utilization of website monitoring services caused by frequent access to websites in existing methods.

[0009] Further, before the step of judging the access type according to the response result and a preset policy library, the method further includes:

[0010] Set the access type parameter based on the URL in the policy library.

[0011] In the above implementation process, the policy library is used to construct an access policy to determine whether the access type is local access or Internet access, and to implement the local playback of website requests and the automatic switching of Internet access.

[0012] Further, the judging of the access type according to the response result and the preset policy library includes:

[0013] Obtain the URL access type parameter from the policy library according to the URL address;

[0014] If it is determined that the cache is enabled based on the URL access type parameter, calculate the Hash value of the URL address, and obtain the local HAR file according to the Hash value, and the update time of the HAR file;

[0015] Judge the access type according to the local HAR file and the update time.

[0016] In the above implementation process, the judgment of the access type includes two elements, namely whether the HAR file already exists and the update time.

[0017] Further, the judging of the access type according to the local HAR file and the update time includes:

[0018] If the local HAR file exists and the difference between the current time and the update time is less than the cache valid duration, it is local access.

[0019] In the above implementation process, if both elements are satisfied, that is, the HAR file already exists and the update time meets the requirements, it is local access.

[0020] Further, the judging of the access type according to the local HAR file and the update time includes:

[0021] If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache valid duration, it is Internet access.

[0022] In the above implementation process, if one of the two elements is not satisfied, that is, the HAR file does not exist or the update time does not meet the requirements, it is Internet access.

[0023] Further, if it is local access, the local playback is performed based on the HAR file, including:

[0024] Start the simulated browser, load the HAR file, and perform local playback according to the URL request.

[0025] In the above implementation process, local playback is based on the local HAR file and combines with a simulated browser to execute URL access requests.

[0026] Furthermore, the method further includes:

[0027] If it is an Internet access, then execute the Internet access and save the access process as a local HAR file. The HAR file name is calculated through the Hash value corresponding to the URL. If the HAR file already exists, then execute the operation of updating the HAR file.

[0028] In the above implementation process, if it is an Internet access, then save it as a local HAR file during the execution process for subsequent local access, thereby reducing frequent Internet requests and occupying broadband resources.

[0029] The embodiment of the present application also provides a website snapshot playback device, and the device includes:

[0030] An access request module, configured to call a URL access, request a URL address, and obtain a response result;

[0031] An access type determination module, configured to determine the access type according to the response result and a preset policy library;

[0032] A local playback execution module, configured to, if it is a local access, perform local playback based on the HAR file.

[0033] The embodiment of the present application also provides a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the above-mentioned website snapshot playback method is implemented.

[0034] The embodiment of the present application also provides an electronic device, and the electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned website snapshot playback method.

[0035] The embodiment of the present application also provides a readable storage medium, and computer program instructions are stored in the readable storage medium. When the computer program instructions are read and run by a processor, the above-mentioned website snapshot playback method is executed. Description of the Drawings

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation of the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0037] Figure 1 It is a flowchart of a website snapshot playback method provided by an embodiment of the present application;

[0038] Figure 2 It is a flowchart of a method for realizing website snapshot playback based on HAR package provided by an embodiment of the present application;

[0039] Figure 3 It is a flowchart for judging access types provided by an embodiment of the present application;

[0040] Figure 4 It is a schematic diagram of website monitoring provided by an embodiment of the present application;

[0041] Figure 5 It is a structural block diagram of a website snapshot playback device provided by an embodiment of the present application.

[0042] Icons:

[0043] 100 - Access request module; 200 - Access type judgment module; 300 - Local playback execution module. Specific embodiments

[0044] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0045] It should be noted that: Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first", "second", etc. are only used for differential description and cannot be understood as indicating or implying relative importance.

[0046] Please refer to Figure 1 , Figure 1 It is a flowchart of a website snapshot playback method provided by an embodiment of the present application. This method can be applied to website monitoring services. Combining cloud website security monitoring technology, it realizes local response to web requests. Using the standard HTTP Archive format (HTTP Archive format, abbreviated as HAR, a JSON - formatted archive file format, mostly used to record the interaction process between a web browser and a website, with file extensions usually being.har, supported by IE, Chrome, and Firefox), and a simulated browser (Playwright, an open - source automation library for browser testing and web scraping), it realizes local storage and playback of web page snapshots, supports content monitoring services, reduces the frequency of network requests, and improves the efficiency of website monitoring services.

[0047] A web page snapshot means that during the process of accessing a web page, the state of the website at a certain moment is stored by certain technical means, which is the snapshot of the web page at a certain moment, abbreviated as web page snapshot. In this application, the web page snapshot is saved in the form of a HAR file.

[0048] The method specifically includes the following steps:

[0049] Step S100: Invoke URL access, request the URL address, and obtain the response result;

[0050] Specifically, URL access is generally invoked by the main business process of website monitoring to request the URL address and return the request response result. Further, according to business requirements, page operations are performed, including but not limited to obtaining inner content, taking page screenshots, obtaining titles, obtaining URL links, etc.

[0051] Step S200: Judge the access type according to the response result and the preset policy library;

[0052] Step S300: If it is a local access, perform local playback based on the HAR file.

[0053] Local playback means that the URL access request provides the request service based on the local HAR file and can be accessed offline. The local playback process needs to be combined with a simulated browser.

[0054] Such as Figure 2 shown, is a flowchart of the method for realizing website snapshot playback based on the HAR package. On the basis of the above embodiments, it further includes:

[0055] Before the step of judging the access type according to the response result and the preset policy library, set the access type parameters based on the URL in the policy library.

[0056] The policy library is used to configure and manage the setting of access type parameters for the URL, including: whether to enable caching and the effective duration of the cache. It is maintained in the form of a UI interface. The definition of the policy can be customized according to different dimensions such as global, domain name, sub-domain name, single page, customer assets, and custom scenarios. The specific policy management capabilities are not elaborated in this application. As a user, the capabilities provided by the policy library include: being able to obtain the corresponding URL access type parameters according to the URL, and further being able to expand the access types related to the business according to the business type.

[0057] Such as Figure 3 shown, is a flowchart of access type judgment. Step S200 specifically includes the following steps:

[0058] Step S201: Obtain the URL access type parameters from the policy library according to the URL address;

[0059] Step S202: If it is determined that caching is enabled based on the URL access type parameter, calculate the Hash value of the URL address, and obtain the local HAR file according to the Hash value, as well as the update time of the HAR file;

[0060] Step S203: Determine the access type according to the local HAR file and the update time.

[0061] Specifically, if the local HAR file exists and the difference between the current time and the update time is less than the cache valid duration, it is a local access.

[0062] If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache valid duration, it is an Internet access.

[0063] If it is an Internet access, perform an Internet access and save the access process as a local HAR file. The HAR file name is calculated through the Hash value corresponding to the URL. If the HAR file already exists, perform the operation of updating the HAR file.

[0064] Obtain the URL access type parameters {whether caching is enabled, cache valid duration} from the policy library according to the URL address. First, judge whether caching is enabled. If it is enabled, further calculate the Hash value of the URL address, and obtain the local HAR file according to the Hash value, as well as the update time of the HAR file. If the HAR file does not exist, perform an Internet access; if the HAR file exists, take the difference between the current time and the update time (in minutes). If the difference is less than the cache valid duration, it means the local HAR file is valid and perform a local playback; if the difference is greater than the cache valid duration, it means the local HAR file is invalid and perform an Internet access.

[0065] Local playback process: First, start the simulated browser (Playwright), load the HAR file, and finally execute a local access request according to the URL, that is, local playback; during the local playback process, there is no need to access the Internet, and the request response process can be completed in offline mode.

[0066] Internet access process: First, start the simulated browser (Playwright), call the browser to perform an Internet access according to the URL, directly access the target site, and at the same time, the access process can be saved as a local HAR file. The HAR file name is calculated through the Hash value corresponding to the URL. If the HAR file already exists, perform the operation of updating the file.

[0067] Figure 2After the requested access is completed based on local playback or Internet access, further page operations related to the business can be performed, including but not limited to: obtaining internal content, taking page screenshots, obtaining titles, obtaining URL links, etc.

[0068] For both local playback and Internet access, the page operations after the request is completed are the same, and the response time is greatly reduced; by means of policies, unnecessary Internet requests are reduced, effectively reducing the time consumption and resource occupancy.

[0069] As Figure 4 shown, it is a schematic diagram of website monitoring. This method can be used as an adaptation layer for URL access requests, providing two ways of local playback and Internet access for URL access, thereby reducing the occupancy of bandwidth resources.

[0070] This method can be used for tampering monitoring, malware monitoring, sensitive word monitoring, etc. Among them, the tampering monitoring specifically includes the following steps:

[0071] Step 11: In the website monitoring service, issue a remote tampering monitoring task for a certain website <aaa.cn>. After unified scheduling service, for the specific page such as: http: / / 1.aaa.cn / , web page access and information scraping need to be performed.

[0072] Step 12: Execute the website crawling request. According to http: / / 1.aaa.cn / , request the access type policy parameters from the policy management service, and the returned result is: {true, 60}, corresponding to: cache enabled, cache valid duration is 60 minutes.

[0073] Step 13: Calculate the Hash value according to the URL http: / / 1.aaa.cn / : 037CC658E15A474FC8F9DF45E7EA76DF. Check whether the HAR file exists according to the Hash value. If the HAR file does not exist (not cached), perform Internet access.

[0074] Internet access includes: starting the browser, making a URL request, and saving / updating files; specifically, the steps are as follows:

[0075] playwright = sync_playwright() / / Initialize the object;

[0076] browser = playwright.start().chromium.launch(headless = True) / / Start the browser;

[0077] context = browser.new_context() / / Create a new context;

[0078] page = context.new_page() / / Create a new page;

[0079] page.goto(url) / / Access the URL via an Internet request;

[0080] page.screenshot(path = screenshot_path) / / Take a screenshot;

[0081] page.content() / / Get the web page content operation;

[0082] Step 16: After completing this Internet access, save the HAR file as: 037CC658E15A474FC8F9DF45E7EA76DF.har. At the same time, complete the operations of obtaining the page content and taking screenshots related to the business, and complete this access request.

[0083] Malware injection monitoring includes the following steps:

[0084] Step 21: In the website monitoring service, issue a remote malware injection monitoring task for a certain website <aaa.cn>. After unified scheduling service, for the specific page such as: http: / / 1.aaa.cn / , web page access and information scraping need to be performed.

[0085] Step 22: Execute the website crawling request. According to http: / / 1.aaa.cn / , request the access type policy parameters from the policy management service, and the returned result is: {true, 60}, corresponding to: cache enabled, cache valid duration is 60 minutes.

[0086] Step 23: Calculate the Hash value according to the URL http: / / 1.aaa.cn / : 037CC658E15A474FC8F9DF45E7EA76DF. Check if the HAR file exists according to the Hash value. The HAR file (037CC658E15A474FC8F9DF45E7EA76DF.har) exists. Further obtain the HAR file update time (2024-10-28 15:30:00), the current time (2024-10-28 15:35:00). The difference between the update time and the current time is 5 minutes, which is less than the cache valid duration of 60 minutes, so perform local playback.

[0087] Step 24: Local playback includes: starting the browser, loading the HAR file, and local request playback; specifically, the steps are as follows:

[0088] playwright = sync_playwright() / / Initialize the object;

[0089] browser = playwright.start().chromium.launch(headless = True) # Launch the browser;

[0090] context = browser.new_context() # Create a new context;

[0091] page = context.new_page() # Create a new page;

[0092] page.route_from_har(HAR file path, update = True, not_found = "abort") # Local playback, read from HAR file;

[0093] page.goto(url) # Access the URL via an Internet request;

[0094] page.screenshot(path = screenshot path) # Screenshot operation;

[0095] page.content() # Operation to get the web page content;

[0096] Step 25: This access meets the policy requirements. Access the local cache file through local playback. At the same time, complete the operations related to the business of getting the page content and taking screenshots to complete this access request.

[0097] The specific steps for sensitive word monitoring are as follows:

[0098] Step 31: In the website monitoring service, issue a remote sensitive word monitoring task for a certain website <aaa.cn>. After unified scheduling service, the specific page is, for example: http: / / 1.aaa.cn / , and web page access and information scraping need to be performed.

[0099] Step 32: Execute the website crawling request. According to http: / / 1.aaa.cn / , request the access type policy parameters from the policy management service, and the returned result is: {true, 60}, corresponding to: cache enabled, cache validity period is 60 minutes.

[0100] Step 33: Calculate the Hash value based on the URL http: / / 1.aaa.cn / : 037CC658E15A474FC8F9DF45E7EA76DF. Check if the HAR file exists based on the Hash value. The HAR file (037CC658E15A474FC8F9DF45E7EA76DF.har) exists. Further, obtain the HAR file update time (2024-10-28 15:30:00), the current time (2024-10-28 16:35:00). The difference between the update time and the current time is 65 minutes, which is greater than the cache valid duration of 60 minutes, indicating that the current file has expired. Perform an Internet access.

[0101] Step 34: Internet access includes: starting the browser, making a URL request, and saving / updating the file. The specific steps are the same as those in Step 14 and will not be elaborated here.

[0102] Step 31: Assume that it takes 1 minute to complete this request. After completing this Internet access, update the HAR file: 037CC658E15A474FC8F9DF45E7EA76DF.har, and at the same time modify the HAR file update time to: (2024-10-28 16:36:00). Further, complete the operations of obtaining the page content and taking screenshots related to the business, and complete this access request.

[0103] Based on the above embodiments, the policy attributes can be further extended. Specifically, the business type field can be used to distinguish and identify the cache valid durations corresponding to different monitoring services. For example, for the same URL, the valid duration for tampering monitoring is 10 minutes, the valid duration for malware monitoring is 30 minutes, and the valid duration for sensitive word monitoring is 60 minutes.

[0104] In the above embodiments, website monitoring can share the local cache file for multiple services. For example, when performing tampering monitoring, malware monitoring, and sensitive word monitoring, the local cached HAR file can be shared, which can further reduce the frequency of Internet access.

[0105] This application effectively reduces unnecessary Internet access requests, reduces bandwidth resource occupancy, and improves resource utilization through the methods of policy definition and local playback. By combining local playback and Internet access, and based on the fact that the local playback response time is much less than the Internet access method, the execution efficiency of the website monitoring task is improved.

[0106] The embodiment of this application also provides a website snapshot playback device, as Figure 5 shown, which is a structural block diagram of a website snapshot playback device applied to the website snapshot playback method. The device includes but is not limited to:

[0107] An access request module 100 is used to call a URL access, request a URL address, and obtain a response result;

[0108] An access type judgment module 200 is used to judge the access type according to the response result and a preset policy library;

[0109] A local playback execution module 300 is used to, if it is a local access, perform local playback based on a HAR file.

[0110] The specific implementation processes of the above-mentioned various modules have been specifically described in the method embodiments and will not be elaborated here.

[0111] The embodiment of the present application also provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned website snapshot playback method is implemented.

[0112] The embodiment of the present application also provides an electronic device, where the electronic device includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned website snapshot playback method.

[0113] The embodiment of the present application also provides a readable storage medium, where computer program instructions are stored in the readable storage medium, and when the computer program instructions are read and run by a processor, the above-mentioned website snapshot playback method is executed.

[0114] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order from that marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0115] In addition, each functional module in various embodiments of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0116] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0117] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0118] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application and should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0119] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

Claims

1. A method for website snapshot playback, characterized in that, Applied to website monitoring services, the method includes: Invoking a URL access to request a URL address and obtaining a response result; Judging the access type according to the response result and a preset policy library. Specifically: obtaining a URL access type parameter from the policy library according to the URL address; if it is determined that caching is enabled based on the URL access type parameter, calculating the Hash value of the URL address, and obtaining a local HAR file according to the Hash value, as well as the update time of the HAR file; judging the access type according to the local HAR file and the update time; If the local HAR file exists and the difference between the current time and the update time is less than the cache validity period, it is a local access. If it is a local access, perform a local playback based on the HAR file.

2. The website snapshot playback method according to claim 1, characterized in that Before the step of judging the access type according to the response result and the preset policy library, the method further includes: Setting a URL-based access type parameter in the policy library.

3. The website snapshot playback method according to claim 1, characterized in that, The judging the access type according to the local HAR file and the update time includes: If the local HAR file does not exist, or the difference between the current time and the update time is not less than the cache validity period, it is an Internet access.

4. The website snapshot playback method according to claim 1, characterized in that, The performing a local playback based on the HAR file if it is a local access includes: Starting a simulated browser, loading the HAR file, and performing a local playback according to the URL request.

5. The website snapshot playback method according to claim 1, characterized in that The method further includes: If it is an Internet access, perform an Internet access and save the access process as a local HAR file. The HAR file name is calculated through the Hash value corresponding to the URL. If the HAR file already exists, perform an operation to update the HAR file.

6. A computer program product, characterized in that, The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the website snapshot playback method described in any one of claims 1-5.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the website snapshot playback method described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, Computer program instructions are stored in the readable storage medium, and when the computer program instructions are read and run by a processor, they execute the website snapshot playback method described in any one of claims 1 to 5.

Images