Client big data loading method based on IndexedDB and Web Worker

Abstract

The invention discloses a client big data loading method based on IndexedDB and Web Worker, and the method comprises the following steps: 1, when a system is initialized, obtaining a target data hash value stored locally, generating a verification request carrying the hash value, and sending the verification request to a server; 2, receiving a verification result returned by the server; 3, a Web Worker thread is started, the read local data or the pulled latest data is transmitted to a Web Worker, and the Web Worker executes data analysis; 4, the main thread monitors the processing result of the Web Worker, receives the processed data and executes batch rendering; and 5, when the user triggers data reloading or data source switching operation, repeating the steps 1-4. The method has the advantages that the network dependence is low, the network request frequency can be reduced, bandwidth resources are saved, the expandability of the system and the user experience are enhanced, the method is suitable for various big data interaction scenes, page jamming is eliminated, and loading is rapid.

Description

Technical Field

[0001] The present invention belongs to the technical field of Web front-end data processing, and in particular relates to a client-side big data loading method based on IndexedDB and WebWorker. Background Art

[0002] As Web application functionality continues to increase, the amount of data that clients need to process and display is growing exponentially. For example, online data analysis platforms need to render more than 100,000 user behavior data points in real time, and product listing pages on e-commerce platforms need to load product information containing multi-dimensional attributes. These scenarios place extremely high demands on the client's data processing capabilities. In traditional Web architectures, large amounts of data need to be loaded in real time through server-side interfaces and processed in the main thread, which can easily lead to slow page responses and page freezes, seriously affecting the user experience.

[0003] The traditional big data loading solution under the Web architecture has the following significant flaws: (1) Slow page loading and lag In traditional solutions, data needs to be pulled in real time through the server interface, and data parsing, formatting and other processing are all performed on the main thread. When the amount of data exceeds a certain value, the main thread will be blocked due to intensive calculations, resulting in UI rendering delays, page freezes or even pseudo-freezes.

[0004] (2) Frequent network requests and waste of resources Existing technologies lack an effective local caching mechanism. Every time a user refreshes a page or switches views, they need to request complete data from the server again. For data containing unchanging fields such as image URLs and text descriptions, repeated requests will lead to bandwidth waste, especially in mobile network environments, which will significantly increase user data consumption.

[0005] (3) Conflict between data processing and UI rendering The browser's main thread is responsible for both JavaScript execution and DOM rendering. When data processing tasks take longer than a certain amount of time, page response delays occur. Traditional solutions perform data processing and UI rendering in series, further exacerbating this conflict.

[0006] To address the above issues, some optimization attempts have been made in the existing technology, but they cannot meet the needs of large data storage. Therefore, there is an urgent need for a client-side data loading method that supports local processing of large data without affecting page responsiveness. Summary of the Invention

[0007] In order to address the deficiencies in the prior art, the present invention provides a client-side big data loading method based on IndexedDB and Web Worker, so as to solve the problems of page freeze and slow loading when the existing client loads and processes big data.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions: The client-side big data loading method based on IndexedDB and Web Worker includes the following steps: Step 1: When the system is initialized, the hash value of the target data stored locally is obtained, a verification request carrying the hash value is generated and sent to the server; Step 2: Receive the verification result returned by the server. If the local data is invalid, pull the latest data from the server and asynchronously write the latest data to the client's IndexedDB database; If the local data is valid, read the data directly from IndexedDB; Step 3: Start the Web Worker thread, transfer the read local data or the latest data to the Web Worker, and let the Web Worker perform data parsing; Step 4: The main thread monitors the processing results of the Web Worker, receives the processed data and performs batch rendering; Step 5: When the user triggers data reloading or switching data sources, repeat steps 1 to 4 above.

[0009] Preferably, the validation of the local data in step 2 includes: If the client has no local data, the verification request does not carry a hash value, and the server returns the complete data; If the client has local data, the verification request carries the data's unique identifier and the corresponding hash value. The server compares the hash values ​​and returns an empty response if they are consistent. Otherwise, it returns the difference data.

[0010] Preferably, the asynchronous writing to the IndexedDB database in step 2 includes: Create a database named after the business identifier, wherein the database includes a data storage table, and the table structure includes at least a data unique identifier, original data content, a hash value, a storage timestamp, and a version number field; The write operation is performed using a transaction mechanism. If the write fails, the retry mechanism is triggered, and the number of retries does not exceed 3 times.

[0011] Preferably, the Web Worker data parsing in step three includes formatting, indexing, and filtering structures.

[0012] Preferably, a data loading planning optimization step is also included: when the server returns difference data, only the corresponding data records in IndexedDB are updated.

[0013] By adopting the above technical solution, the present invention has the following beneficial effects: This application uses IndexedDB local cache to reduce the number of network requests during repeated loading, and the data access latency after the first load is reduced. Web Workers take on most data processing tasks, the main thread blocking time is reduced, the page frame rate is improved, and the hash value-based verification mechanism ensures the consistency of local data and server data. Incremental updates reduce the amount of data transmission. It can also optimize data loading planning, significantly improve the response speed and operation smoothness of pages in large data volume scenarios, reduce the frequency of network requests, save bandwidth resources, enhance the scalability of the system and user experience, and is suitable for a variety of big data interaction scenarios. It reduces network dependence as a whole, eliminates page jams, and loads quickly. In summary, the present invention has the advantages of low network dependence, can reduce network request frequency, save bandwidth resources, enhance system scalability and user experience, is suitable for a variety of big data interaction scenarios, eliminates page freezes, and loads quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0016] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0017] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0018] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0019] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0020] Example 1 In this embodiment, the present invention provides a client-side big data loading solution based on IndexedDB and Web Worker, which realizes efficient loading and processing of big data through the collaborative design of local caching, asynchronous processing, dynamic verification and strategy optimization. The existing technology has problems such as slow page loading and jamming, frequent network requests and resource waste, conflict between data processing and UI rendering, and lack of terminal adaptation capabilities. To solve this problem, some optimization attempts have appeared in the existing technology, for example: using localStorage for data caching, but localStorage only supports string storage, with a capacity limited to 5-10MB, which cannot meet the big data storage requirements, or using Web Worker to process part of the data, but not combined with local storage, and still requires frequent requests to the server. Therefore, there is an urgent need for a client-side big data loading solution that integrates local persistent storage, asynchronous processing and dynamic adaptation.

[0021] The core technical points of this application include local data caching, asynchronous data processing, data validity verification mechanism and data loading strategy optimization. It uses the browser's built-in IndexedDB database to perform persistent local storage of large amounts of data. After the data is loaded for the first time, it is stored locally and can be read directly from the local area later, reducing network requests. The processing logic of big data is put into Web Worker to run, realizing asynchronous data parsing and processing. The main thread is only responsible for UI rendering to avoid blocking page response due to data calculation. Before each loading, it interacts with the server through a specific version number, hash value or check mark to determine whether the local data is valid. If the data is invalid, it will be pulled from the server and overwritten with the IndexedDB data. It supports incremental data loading and batch rendering to avoid loading all data at once, and provides loading parameter configurations adapted to different terminal performance. Among them, supporting batch rendering means dynamically adjusting the amount of data rendered in each batch according to the performance of the client device (number of CPU cores, memory capacity). The amount of data rendered at a single time can not exceed 1,000. Configurable parameters include IndexedDB storage validity period, Web The number of worker threads and rendering batch size are automatically adapted to the terminal type (PC / mobile device).

[0022] The present invention can be widely used in front-end data loading modules of large-scale Web systems, online data analysis platforms, BI systems (such as processing millions of business data, including sales reports, user behavior, etc., asynchronously calculating chart data through Web Workers to avoid rendering blocking), visualization systems and client caching scenarios (such as offline data access, slow network environment optimization, etc., in an offline environment, caching necessary user data such as browsed products and form drafts through IndexedDB to ensure the availability of core functions), e-commerce platforms (loading hundreds of thousands of product lists and user evaluation data, supporting offline browsing of product details, and improving loading speed in weak network environments), social applications (loading frequently accessed data such as user dynamics and message records, reducing repeated network requests and reducing server pressure), etc.

[0023] like Figure 1 As shown, in one embodiment of the present invention, the client-side big data loading method based on IndexedDB and Web Worker of the present invention includes the following steps: Step 1: When the system is initialized, the hash value of the target data stored locally is obtained, a verification request carrying the hash value is generated and sent to the server; Step 2: Receive the verification result returned by the server. If the local data is invalid, pull the latest data from the server and asynchronously write the latest data to the client's IndexedDB database; If the local data is valid, read the data directly from IndexedDB; Specifically, the validation of the local data in step 2 includes: If the client has no local data, the verification request does not carry a hash value, and the server returns the complete data; If the client has local data, the verification request carries the data's unique identifier and corresponding hash value. The server compares the hash values ​​and returns an empty response if they match, otherwise it returns the difference data. The asynchronous writing to the IndexedDB database in step 2 includes: Create a database named after the business identifier, wherein the database includes a data storage table, and the table structure includes at least a data unique identifier, original data content, a hash value, a storage timestamp, and a version number field; A transaction mechanism is used to perform write operations. If the write fails, a retry mechanism is triggered. The number of retries may not exceed 3 times, for example. Steps 1 and 2 of the present invention can be implemented in conjunction with the data verification module, the local storage module, and the asynchronous processing module. That is, the present invention is implemented based on the data verification module, the local storage module, and the asynchronous processing module. Specifically, the data verification module is responsible for verifying the validity of local data during system initialization. The specific process is as follows: Hash value generation: For each piece of locally stored data, the SHA-256 algorithm is used to calculate its content digest, generating a 64-bit hash string as a unique identifier. The hash value calculation scope includes all fields of the data (such as product ID, name, price, etc.), ensuring that even the slightest change in the data content will result in a change in the hash value; Verification request construction: If the data exists locally, the request carries the data's unique identifier (such as the data ID) and the corresponding hash value. If the data does not exist locally, the request does not carry the hash value. Server verification: After receiving the request, the server compares the local hash value with the hash value of the current data on the server: If the hash values ​​match (or there is no local data but the server detects that the data has not been updated), an empty response is returned, indicating that the local data is valid; If the hash values ​​do not match, return the difference data (containing only the modified fields) or the complete data (when loading for the first time); The local storage module implements persistent storage of big data based on IndexedDB, supporting terabyte-level data capacity. Furthermore, the IndexedDB database in the local storage module uses a multi-level index structure, including a primary index keyed by the data's unique identifier and secondary indexes keyed by business fields (such as timestamps and data types). The specific design is as follows: Database structure: Create databases named after business scenarios (such as e-commerce product data). Each database contains multiple object storage spaces corresponding to different data types. Storage fields: Each data record contains the following fields: dataId: unique identifier of data (such as product ID); content: original data content (JSON format); hash: SHA-256 hash value of the data content; timestamp: stores timestamp (millisecond level); version: data version number (used for incremental updates); Index design: Establish multi-level indexes to improve query efficiency: Primary index: uses dataId as the key to support fast query of single data; Secondary index: uses timestamp and version as keys, and supports filtering data by time range or version; Transaction management: Data write, update, and delete operations are all performed through IndexedDB's transaction mechanism to ensure the atomicity of the operations. If a transaction fails (for example, due to insufficient storage space), a retry mechanism is triggered (up to three times), and an error log is recorded after a retry fails. The asynchronous processing module of the present invention decouples data processing from the main thread based on Web Worker. Its specific functions are as follows: Worker initialization: The main thread creates a Worker thread through newWorker('data_processor.js'). Each business scenario corresponds to an independent Worker to avoid interference between different types of data processing; Data communication: The main thread transmits data to the Worker (using a structured cloning algorithm, supporting types such as JSON and ArrayBuffer). After the Worker completes the processing, it returns the result to the main thread. Data processing tasks: Workers are responsible for performing computationally intensive operations, including formatting, indexing, and filtering structures: Data formatting: converting raw data into a structure that can be directly used by the front-end framework (such as converting the snake-case named fields returned by the back-end into camel-case names); Indexing: Create B-tree indexes for high-frequency query fields (such as product categories and price ranges) to reduce query time complexity; Filtering structure: Eliminate invalid data (such as records with negative prices or expired timestamps) and filter data based on business rules (such as only retaining user behavior data from the past 30 days); Step 3: Start the Web Worker thread, transfer the read local data or the latest data to the Web Worker, and let the Web Worker perform data parsing; Web Worker data parsing in step 3 includes formatting, indexing, and filtering structures; Step 4: The main thread monitors the processing results of the Web Worker, receives the processed data and performs batch rendering. The batch rendering in step 4 is implemented according to the rendering control module, that is, the client big data loading method based on IndexedDB and WebWorker of the present invention is implemented based on the rendering control module. The main thread of the rendering control module receives the data processed by the Worker and adopts a batch rendering strategy to avoid UI blocking. It dynamically adjusts the amount of data rendered in each batch according to the device performance. It first obtains the number of CPU cores. When the number of cores is ≤4, 500 data can be rendered in each batch. When the number of cores is >4, 1000 data can be rendered in each batch. It can also implement the scheduling of rendering frames to ensure that the rendering of each batch of data is completed within the refresh cycle of the browser. If the rendering time of a single batch is too long, such as exceeding 15ms, the rendering amount of the next batch is automatically reduced. The rendering control module can also only render the data within the visible area; Step 5: When the user triggers data reloading or switching data sources, repeat steps 1 to 4 above.

[0024] The present invention also includes a data loading planning optimization step: when the server returns difference data, only the corresponding data record in IndexedDB is updated. Specifically, the data loading planning optimization step is implemented by a policy optimization module, that is, the present invention dynamically adjusts the loading and processing strategies based on the policy optimization module to adapt to different terminals and network environments. When the server returns difference data, only the record of the corresponding dataId in IndexedDB is updated, rather than full coverage. For example, after the product price is modified, only the content.price field and the corresponding hash and version are updated.

[0025] In summary, the data verification module based on the present invention is used to obtain the local data hash value during initialization, send a verification request to the server and receive the verification result. The local storage module implements persistent storage of data based on IndexedDB, including data writing, reading and updating operations, supporting transaction management and failure retry. The asynchronous processing module is based on Web Worker implements data parsing, formatting, indexing and filtering, and communicates with the main thread through a message mechanism. The rendering control module receives the results of the asynchronous processing module, performs batch rendering according to device performance, and controls the parallel execution of the UI thread and data processing. The strategy optimization module configures incremental loading rules, batch rendering parameters and terminal adaptation strategies, and dynamically adjusts the data processing flow. The asynchronous processing module specifically includes a data parsing unit, an index building unit and a filtering unit. The data parsing unit converts the original data into a structured JSON format. The index building unit establishes a B-tree index for high-frequency query fields to improve data retrieval efficiency. The filtering unit eliminates invalid data according to preset rules (such as data expiration time and field non-empty check). The slightly optimized module also includes a performance monitoring submodule, which collects client CPU usage, memory usage and network status in real time. When the performance threshold is detected (such as CPU usage > 80% or memory usage > 2GB), the amount of data processed in each batch is automatically reduced.

[0026] This embodiment does not impose any formal restrictions on the shape, material, structure, etc. of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of protection of the technical solution of the present invention.

Claims

1. A client-side big data loading method based on IndexedDB and Web Worker, characterized in that: The following steps are involved: Step 1: When the system is initialized, the hash value of the target data stored locally is obtained, a verification request carrying the hash value is generated and sent to the server; Step 2: Receive the verification result returned by the server. If the local data is invalid, pull the latest data from the server and asynchronously write the latest data to the client's IndexedDB database; If the local data is valid, read the data directly from IndexedDB; Step 3: Start the Web Worker thread, transfer the read local data or the latest pulled data to the WebWorker, and let the Web Worker perform data parsing; Step 4: The main thread monitors the processing results of the Web Worker, receives the processed data and performs batch rendering; Step 5: When the user triggers data reloading or switching data sources, repeat steps 1 to 4 above.

2. The client-side big data loading method based on IndexedDB and Web Worker according to claim 1 is characterized in that: The validation of the local data in step 2 includes: If the client has no local data, the verification request does not carry a hash value, and the server returns the complete data; If the client has local data, the verification request carries the data's unique identifier and the corresponding hash value. The server compares the hash values ​​and returns an empty response if they are consistent. Otherwise, it returns the difference data.

3. The client-side big data loading method based on IndexedDB and Web Worker according to claim 1 is characterized in that: The asynchronous writing to the IndexedDB database in step 2 includes: Create a database named after the business identifier, wherein the database includes a data storage table, and the table structure includes at least a data unique identifier, original data content, a hash value, a storage timestamp, and a version number field; The write operation is performed using a transaction mechanism. If the write fails, the retry mechanism is triggered, and the number of retries does not exceed 3 times.

4. The client-side big data loading method based on IndexedDB and Web Worker according to claim 1, characterized in that: The Web Worker data parsing in step 3 includes formatting, indexing, and filtering structures.

5. The client-side big data loading method based on IndexedDB and Web Worker according to claim 2, characterized in that: It also includes a data loading optimization step: when the server returns differential data, only the corresponding data record in IndexedDB is updated.

Images