DOM (Document Object Model) reconstruction browser resource acquisition method and device, equipment and medium

Abstract

The invention provides a browser resource acquisition method and device for DOM reconstruction, equipment and a medium, and the method comprises the steps: modifying a kernel of a remote isolation browser, and enabling a cache file to be accessible; if the website resources in the stored file do not exist, the resources of the set website are obtained through the server in an HTTP request mode and processed to form browser data, and then the browser data are forwarded to the client side; if website resources exist in the cache file, analyzing the cache file, obtaining corresponding content-type and setting file content as browser data, and sending the browser data to the client; and the client synchronizes and displays the acquired browser data, so that the resource acquisition speed is higher, and the pressure of the server is relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of browser resource processing, in particular to a DOM reconstruction browser resource acquisition method, device, equipment and medium. BACKGROUND

[0002] The core features of remote isolation browsers are embodied in the two dimensions of "remote" and "isolation", which together form the basis of the technical architecture: Remote: means relying on a cloud-deployed browser instance, rather than a locally running browser program. Users do not need to install a complete browser environment locally, and all basic browser operations (such as kernel parsing and script execution) are completed on a cloud server. Isolation: refers to the fact that all browser simulation operations (including DOM construction, page rendering, resource loading, etc.) are carried out in a cloud-independent environment, completely isolated from the user's local browser environment. Finally, only through data synchronization technology, the DOM structure and page view results rendered by the cloud are transmitted to the user's local browser page in real time, realizing the separation mode of "cloud computing + local display". In combination with its "cloud running + local synchronization" working logic, the current remote isolation browser mainly faces the following five types of problems in technical landing, and there is a certain correlation between each problem: 1. Efficiency loss caused by resource processing mode Currently, most remote isolation browsers use the "HTTP download cache + forwarding" resource processing flow: the cloud browser first downloads various resources (such as images, JS scripts, CSS files, etc.) of the target page through HTTP request, caches the resources and then forwards them to the local browser. This mode has obvious shortcomings - resources need to go through "cloud download - cloud cache - cloud forwarding - local reception" multiple links, and each step will produce data transmission delay, especially when dealing with large volume resources (such as high-definition pictures, video clips), which is prone to cause local page loading lag. 2. Additional storage costs caused by cache strategy To improve resource loading efficiency, remote isolation browsers need to develop cache strategies for commonly used resources (such as fixed domain JS libraries and repeatedly accessed page static resources). However, the implementation of the cache function relies on additional storage space provided by the cloud server: on the one hand, the types of resources to be stored are diverse (text, pictures, binary files, etc.), requiring compatibility with storage media; on the other hand, as the total amount of cached resources increases, regular resource cleanup (such as eliminating long-term unused caches) is required, otherwise it will continuously occupy storage resources, leading to increased hardware costs. 3. Technical difficulties in cache expiration management Cache resources have a "time limit" problem: some resources (such as API return data of dynamic pages, pictures with timestamps) will be updated over time, and if the cloud still uses the old cache to forward to the local, it will cause the local page to display inconsistent content with the actual content (i.e. "dirty cache data"). But there are difficulties in managing cache expiration: first, it is difficult to accurately determine the update frequency of all resources (such as some resources without explicit expiration identifier); second, frequent checking of resource validity (such as checking whether the resource is updated before each request) will increase the computing pressure of the cloud server, and thus reduce the overall operating efficiency. 4. Resource acquisition speed bottleneck caused by network dependence All resource acquisition of the remote isolated browser depends on network requests between the cloud and the target resource server, rather than local direct requests, which makes the resource acquisition speed affected by multiple network factors: if the network link between the cloud server and the target resource server is unstable (such as packet loss, high delay), it will directly cause slow resource download; at the same time, if the network transmission rate between the user's local and the cloud server is low, even if the cloud has acquired the resource, it will also cause local page loading lag due to data synchronization delay, and the core advantage of better overall experience than local browsers is difficult to realize. 5. IP throttling or blocking risk caused by platform restrictions The operation of the cloud browser depends on a fixed IP address segment (or a dynamically allocated IP pool), and if multiple users access the same platform (such as e-commerce websites, social media platforms) through the same batch of cloud IPs, it is easy to trigger the platform's anti-crawler or anti-abnormal access mechanism: the platform will determine the IP with a large number of requests in a short time as "abnormal IP", and then take throttling measures (such as limiting request frequency), or even directly block the IP. Once the cloud IP is blocked, all remote isolated browser users corresponding to the IP will be unable to normally access the target platform, causing service interruption, and the replacement and maintenance of IP resources require additional technical and time costs. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a DOM reconstruction browser resource acquisition method, device, equipment and medium, which fully utilizes the local cache of the server, optimizes the network request strategy and ensures the consistency of the cache behavior, and achieves a good balance between performance improvement, resource saving and reliability guarantee.

[0004] In a first aspect, the present application provides a DOM reconstruction browser resource acquisition method, a remote isolated browser is provided on a server, and the remote isolated browser is a Chromium kernel browser, comprising the following steps: Step 1, modify the kernel of the remote isolated browser, so that the cache file can be accessed; Step 2: If the website resources in the stored file do not exist, the website resources are obtained from the server via an HTTP request and processed to form browser data, which is then forwarded to the client; if the website resources in the stored file exist, the cached file is parsed, the corresponding content-type and the content of the specified file are obtained as browser data, and sent to the client. Step 3: The client synchronizes the obtained browser data and displays it.

[0005] Secondly, the present invention provides a browser resource acquisition device for DOM reconstruction, wherein a remotely isolated browser is located on a server, and the remotely isolated browser is a Chromium-based browser, comprising: Modify the permissions module and the kernel of the remotely isolated browser to allow cached files to be accessed; The server-side data module, if the website resource in the stored file does not exist, retrieves the website resource from the server via an HTTP request, processes it to form browser data, and then forwards it to the client; if the website resource in the stored file exists, it parses the cached file, obtains the corresponding content-type and the content of the specified file as browser data, and sends it to the client. The data synchronization module synchronizes the browser data obtained by the client and then displays it.

[0006] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0007] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0008] One or more technical solutions provided by this invention have at least the following technical effects or advantages: 1. Improve resource loading efficiency Reading directly from the local cache instead of requesting network resources significantly reduces loading time and network bandwidth consumption, especially for frequently accessed resources, which can significantly improve the user experience (such as static resources like images and scripts).

[0009] 2. Enhance system stability and reliability It implements a two-layer retrieval mechanism of caching + network request, which can rely on local caching when the network is unstable.

[0010] 3. Optimize resource utilization By modifying the locking mechanism of the Chrome kernel, shared access to the cache database is achieved, avoiding multi-process conflicts, and the cache strategy synchronization reduces invalid network requests, reducing server load and client energy consumption.

[0011] 4. Flexible degradation mechanism When the cache does not exist or is invalid, it is automatically degraded to an HTTP request to ensure the continuity of resource acquisition.

[0012] 5. Accuracy of data processing Accurately parse the binary format of the Chrome cache to ensure that the extracted HTTP header information and resource data are accurate; standardize the header fields (such as converting to lowercase) to facilitate unified management and query.

[0013] Overall, the present application achieves a good balance between performance improvement, resource saving, and reliability assurance by fully utilizing server local cache, optimizing network request strategy, and ensuring cache behavior consistency.

[0014] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0015] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0016] Figure 1 The flowchart in the method of the first embodiment of the present application; Figure 2 The structure diagram of the device in the second embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application provides a DOM reconstruction browser resource acquisition method, device, equipment and medium.

[0018] The technical solutions in the embodiments of the present application have the following overall ideas: Step 1, modify the kernel to make the cache file accessible, otherwise the cache file is locked; Step 2, parse the cache file to obtain the corresponding content-type and file content; Step 3, degradation processing, if the cache is not in place, pull the source again through HTTP; Step 4, cache the cache strategy of the cache platform resource to the client to further reduce the pulling of resources.

[0019] The main implementation code is as follows: 1. Modify the Chrome kernel in the file sql_persistence_store.cc class Backend { public: Backend(const base::FilePath& path, int64_t max_bytes, net::CacheTypetype) path_(path), max_bytes_( / / If the specified max_bytes is valid, use it. Otherwise, calculate / / a preferred size based on available disk space. max_bytes > 0 ? max_bytes : PreferredCacheSize(base::SysInfo::AmountOfFreeDiskSpace(path), type)), high_watermark_(max_bytes_ - max_bytes_ / kSqlBackendEvictionMarginDivisor), low_watermark_(max_bytes_ - 2 * (max_bytes_ / kSqlBackendEvictionMarginDivisor)), db_(sql::DatabaseOptions() .set_exclusive_locking(false) / / Modification: Here it is set to false, replacing it with a shared lock. #if BUILDFLAG(IS_WIN) .set_exclusive_database_file_lock(false) / / Modification: For Windows systems, removes file locks. #endif / / IS_WIN .set_preload(true) .set_wal_mode(true), / / Tag for metrics collection. sql::Database::Tag("HttpCacheDiskCache")) { } 2. Implement parsing of cache files Obtain the corresponding resource header information, corresponding to the http response header def get_headers(head: bytes) -> dict: head = head[4:] flags = struct.unpack("<I", head[0:4])[0] head = head[4:] extra_flags = 0 if flags & (1 << 31): extra_flags = struct.unpack("<I", head[0:4])[0] head = head[4:] head = head[16:] if extra_flags & (1 << 2): head = head[8:] header_length = struct.unpack("<I", head[0:4])[0] header = head[0:header_length] headers = header.split(b"\00") header_dict = {} for header in headers: if header: pair = header.decode("latin-1").split(":", 1) if len(pair) == 2: key, value = pair header_dict[key.lower()] = value.lower() return header_dict / / Get the corresponding resource data def get_cache(url:str) -> CacheItem: user_data_dir = pathlib.Path(config.USER_DATA_DIR) cache_data_dir = user_data_dir / 'Default' / 'Cache' / 'Cache_Data' db_path = cache_data_dir / 'sqldb' with sqlite3.connect(db_path, timeout=1) as conn: cur = conn.cursor() cur.execute(f"SELECT token_high, token_low, body_end, head FROM resources WHERE cache_key like '%{url}'") / / Query the metadata of the resource based on the url result = cur.fetchone() if result: token_high = result[0] token_low = result[1] body_end = result[2] header = result[3] headers = get_headers(header) / / Parse the header mime_type = headers.get("content-type", "application / octet-stream") / / Get the content-type content_encoding = headers.get("content-encoding", "") cur.execute(f"SELECT blob FROM blobs WHERE token_high = ? AND token_low = ? AND start <= ? order by start asc", (token_high, token_low, body_end)) / / Retrieves resource data based on offset. result = cur.fetchall() data = b"".join([row[0] for row in result]) return CacheItem(url, data, mime_type, mime_type, content_encoding) return None 3. If the cache does not exist, then fall back to HTTP retrieval. if content_data is None: try: timeout = aiohttp.ClientTimeout(total=8, connect=3) headers = { 'User-Agent': 'Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 139.0.0.0 Safari / 537.36' } async with aiohttp.ClientSession(timeout=timeout, headers=headers) assessment: async with session.get(full_url) as response: if response.status == 200: content_data = await response.read() mime_type = response.headers.get('content-type', 'application / octet-stream') logger.info(f) [HTTP] Resource downloaded successfully: {full_url}") else: logger.error(f) [HTTP] Download failed, status code {response.status}: {full_url}") except Exception as e: logger.error(f) [HTTP] Download error: {full_url}, Reason: {e}") 4. Synchronize the cloud-based caching strategy to the client browser, which involves three HTTP cache headers: cache-control: controls the maximum lifespan of the cache and cache behavior (such as max-age, no-cache, etc.); expires: Specifies the absolute time for the resource to expire; age: Indicates the time elapsed since the resource was created on the server; By synchronizing these strategies, client browsers can follow caching rules consistent with those in the cloud, reducing unnecessary resource requests, improving loading speed, and reducing server load. Example

[0020] like Figure 1 As shown, this embodiment provides a method for obtaining browser resources for DOM reconstruction. A remotely isolated browser is located on the server. This remotely isolated browser is a Chromium-based browser, and the method includes the following steps: Step 1: Modify the kernel of the remotely isolated browser to make cached files accessible; Step 2: If the website resources in the cache file do not exist, the browser retrieves the website resources from the server via an HTTP request, processes them to form browser data, and then forwards it to the client. If the website resources in the cache file exist, the browser parses the cache file, obtains the corresponding content-type and the content of the cache file as browser data, and sends it to the client. The browser caches resources in a file (SQLite), called a cache file, and the content of the cache file is the website resources. Website resources include CSS, fonts, images, etc. Step 3, the client synchronizes the obtained browser data and displays.

[0021] In this embodiment, preferably, the step 1 is specifically: modifying the Backend class constructor in the sql_persistence_store.cc file: setting set_exclusive_locking(false) as a shared lock instead of an exclusive lock; and removing the file lock set_exclusive_database_file_lock(false) to enable the cache file of the remote isolated browser to be accessed.

[0022] In this embodiment, preferably, the step 2 of obtaining the corresponding content-type and setting the file content as browser data is specifically: parsing the HTTP response header information in the cache file, processing the binary data according to the set format; extracting the flag bit, the additional flag, and then parsing the actual HTTP header field; converting the parsing result into a dictionary form, with the key being the lowercase header field name and the value being the corresponding header field value; constructing the remote isolated browser cache database path, connecting the database, querying the corresponding resource metadata according to the URL, parsing the response header using get_headers, extracting the content type and encoding method, querying the actual resource data according to the token information in the metadata, obtaining the corresponding data block, and splicing all the data blocks into a complete resource as the browser data.

[0023] In this embodiment, preferably, the step 3 is specifically: the client obtains the browser data and the cache strategy of the remote isolated browser, the client synchronizes and takes effect on the cache strategy, synchronizes and displays the obtained browser data.

[0024] Based on the same inventive concept, the application also provides a device corresponding to the method in embodiment one, which is described in detail in embodiment two. Embodiment

[0025] As shown in Figure 2 In this embodiment, a DOM reconstruction browser resource acquisition device is provided, and the remote isolated browser is arranged on a server. The remote isolated browser is a Chromium kernel browser, which comprises: a permission modification module, which modifies the kernel of the remote isolated browser to enable the cache file to be accessed; The service end data module acquires the resource of the set website through the server by using the HTTP request mode and processes the acquired resource to form browser data, and then forwards the browser data to the client if the website resource in the stored file does not exist; if the website resource in the stored file exists, the service end data module parses the cache file, acquires the corresponding content-type and set file content as the browser data, and sends the browser data to the client. The data synchronization module synchronizes the acquired browser data and displays the browser data.

[0026] In this embodiment, preferably, the modification permission module is specifically: modifying the Backend class constructor in the sql_persistence_store.cc file: setting set_exclusive_locking(false) as a shared lock instead of an exclusive lock; and removing the file lock set_exclusive_database_file_lock(false) to enable the cache file of the remote isolated browser to be accessed.

[0027] In this embodiment, preferably, the service end data module acquires the corresponding content-type and set file content as the browser data specifically as follows: Parsing the HTTP response header information in the cache file, processing the binary data according to a set format; extracting a flag bit and an additional flag, and then parsing the actual HTTP header field; converting the parsing result into a dictionary form, with the key being a lower-case header field name and the value being the corresponding header field value; Constructing the database path of the remote isolated browser cache, connecting the database, querying the corresponding resource metadata according to the URL, parsing the response header by using get_headers, extracting the content type and encoding mode, querying the actual resource data according to the token information in the metadata, acquiring the corresponding data block, and splicing all the data blocks into a complete resource as the browser data.

[0028] In this embodiment, preferably, the data synchronization module is specifically: the client acquires the browser data and the cache strategy of the remote isolated browser, the client synchronizes and takes effect of the cache strategy, synchronizes and displays the acquired browser data.

[0029] The device introduced in the second embodiment of the present application is the device used to implement the method of the first embodiment of the present application, and therefore, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and modification of the device, and therefore, the specific structure and modification of the device are not described here. Any device used in the method of the first embodiment of the present application belongs to the scope of the present application.

[0030] Based on the same inventive concept, the present application provides an electronic device corresponding to the embodiment one, which is described in detail in the embodiment three. Embodiment

[0031] The embodiment provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor can realize any of the implementation manners of the embodiment one when executing the computer program.

[0032] Since the electronic device introduced in the embodiment is the device used for implementing the method in the embodiment one of the present application, the specific implementation manners of the electronic device in the embodiment and various changes thereof can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, so the method implemented by the electronic device in the embodiment one of the present application is not introduced in detail here. As long as the device used for implementing the method in the embodiment one of the present application is implemented by the person skilled in the art, it belongs to the scope of the present application.

[0033] Based on the same inventive concept, the present application provides a storage medium corresponding to the embodiment one, which is described in detail in the embodiment four. Embodiment

[0034] The embodiment provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to realize any of the implementation manners of the embodiment one.

[0035] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0036] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The device for implementing the function specified in one block or multiple blocks.

[0037] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The flow or flows and / or blocks Figure 1 The functions specified in the flow or flows and / or blocks

[0038] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks Figure 1 The functions specified in the flow or flows and / or blocks

[0039] While the present application has been described with reference to the specific embodiments thereof, it should be understood by those skilled in the art that the personal described are intended to be illustrative only and not used to limit the scope of the application. Other embodiments can be made by those skilled in the art which fall within the scope of the application. Accordingly, the scope of the present application should be determined only by the claims which follow.

Claims

1. A method for browser resource acquisition of DOM reconstruction, a remote isolated browser is arranged on a server, the remote isolated browser is a browser with Chromium kernel, characterized in that: It comprises the following steps: Step 1, modifying the kernel of the remote isolated browser to make the cache file accessible; Step 2, if the website resource in the cache file does not exist, the server is requested to obtain the resource of the set website and process the browser data, and then forward to the client; if the website resource in the cache file exists, the cache file is parsed to obtain the corresponding content-type and set file content as browser data, and sent to the client; Step 3, the client synchronizes and displays the obtained browser data.

2. The DOM-reconstructed browser resource acquisition method of claim 1, wherein: The step 1 is specifically: modifying the Backend class constructor in the sql_persistence_store.cc file: setting set_exclusive_locking(false) to shared lock instead of exclusive lock; and removing the file lock set_exclusive_database_file_lock(false) to make the cache file of the remote isolated browser accessible.

3. The DOM-reconstructed browser resource acquisition method of claim 1, wherein: The step 2 is specifically: Parsing the HTTP response header information in the cache file, processing the binary data according to the set format; Extracting the flag bit, additional flag, and then parsing the actual HTTP header field; converting the parsing result to a dictionary form, with the key as the lowercase header field name and the value as the corresponding header field value; Building the remote isolated browser cache database path, connecting the database, querying the corresponding resource metadata according to the URL, parsing the response header using get_headers, extracting the content type and encoding method, querying the actual resource data according to the token information in the metadata, obtaining the corresponding data block, and concatenating all data blocks into a complete resource as browser data.

4. The DOM-reconstructed browser resource acquisition method of claim 1, wherein: The step 3 is specifically: the client obtains the browser data and the cache strategy of the remote isolated browser, the client synchronizes and enables the cache strategy, and synchronizes and displays the obtained browser data.

5. A DOM reconstruction browser resource acquisition apparatus, a remote isolation browser is arranged on a server, the remote isolation browser is a browser of a Chromium kernel, characterized in that: It comprises: Modifying the permission module, modifying the kernel of the remote isolated browser to make the cache file accessible; The server data module, if the website resource in the cache file does not exist, the server is requested to obtain the resource of the set website and process the browser data, and then forward to the client; if the website resource in the cache file exists, the cache file is parsed to obtain the corresponding content-type and set file content as browser data, and sent to the client; The data synchronization module, the client synchronizes and displays the obtained browser data.

6. The DOM-reconstructed browser resource fetching apparatus according to claim 5, wherein: The modification permission module is specifically: modifying the Backend class constructor in the sql_persistence_store.cc file: setting set_exclusive_locking(false) to a shared lock instead of an exclusive lock; and removing the file lock set_exclusive_database_file_lock(false) to enable the cache file of the remote isolated browser to be accessed.

7. The DOM-reconstructed browser resource fetching apparatus according to claim 5, wherein: The service end data module specifically includes: Parsing the HTTP response header information in the cache file, processing the binary data according to the set format; Extracting the flag bit and the additional flag, and then parsing the actual HTTP header field; converting the parsing result into a dictionary form, with the key being the lowercase header field name and the value being the corresponding header field value; Constructing the remote isolated browser cache database path, connecting the database, querying the corresponding resource metadata according to the URL, parsing the response header using get_headers, extracting the content type and the encoding method, querying the actual resource data according to the token information in the metadata, obtaining the corresponding data block, and splicing all the data blocks into a complete resource as the browser data.

8. The DOM-reconstructed browser resource fetching apparatus according to claim 5, wherein: The data synchronization module is specifically: the client obtains the browser data and the cache strategy of the remote isolated browser, the client synchronizes and takes effect on the cache strategy, and synchronizes and displays the obtained browser data.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method of any one of claims 1 to 4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 4.

Images