A front-end big data computing processing method and system

Abstract

The application discloses a front-end big data computing processing method and system, and the method comprises the following steps: starting a browser main thread, obtaining to-be-processed data according to the browser main thread, presetting a compression algorithm, compressing the to-be-processed data according to the compression algorithm, and generating a single slice; presetting a slice offset for the single slice, setting a key value for each slice, generating a corresponding worker for each slice, and converting slice data into a Transferable object array; each Transferable object array is distributed to a corresponding worker, the worker is used for converting the Transferable object array into text, and the text is saved in an indexedDB according to the key value; a main thread obtains a key value array, and queries corresponding data from the indexedDB according to the key value array.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to a front-end big data computing processing method and system. BACKGROUND

[0002] In the prior art, part of the business computing is processed in the front end, but the particularity of the front end is single-threaded execution, so the performance problem is more likely to occur than the server. Although the js language is single-threaded, the browser provides a multi-threaded solution for us, and we can process large data computing or some streaming computing by using the multi-threaded characteristics of the browser. In addition to the computing performance of large data, the problem of browser tab crash caused by high thread memory consumption also needs to be handled. This problem is mainly realized by the memory address interaction of multi-thread, and finally the data integration problem is handled by indexedDB storage. However, the prior art has the following technical problems: although there are multi-thread and indexedDB storage, the capacity is still limited, the computing power is mainly limited by the upper limit of the browser, and the storage is mainly limited by the local disk capacity. The current local solution cannot solve the problem of too high memory occupation. SUMMARY

[0003] One of the purposes of the present application is to provide a front-end big data computing processing method and system, which realizes multi-process parallel computing through Web Worker provided by the browser, and optimizes the problem of multiple copies of objects in the ordinary work scheme by transferring the data ownership through the transferList between works.

[0004] Another purpose of the present application is to provide a front-end big data computing processing method and system, which realizes data persistence by using the storage capacity of indexedDB while performing parallel computing by using multiple works, and releases the memory of the work after optimization, so as to better optimize the storage memory.

[0005] Another purpose of the present application is to provide a front-end big data computing processing method and system, which utilizes the worker characteristics of the browser to enable large-scale computing to be performed simultaneously without causing memory overflow due to multiple copies of objects, and utilizes the characteristics of indexedDB local storage to reduce the tab memory occupation as much as possible, so that the garbage collection mechanism is more smooth, and the amount of data obtained by the main thread and the data to be stored is smaller.

[0006] In order to achieve at least one of the above purposes, the present application further provides a front-end big data computing processing method, which comprises:

[0007] Start a browser main thread, obtain to-be-processed data according to the browser main thread, preset a compression algorithm, compress the to-be-processed data according to the compression algorithm, and generate a separate slice;

[0008] Preset a slice offset for the separate slice, set a key value for each slice, generate a corresponding worker for each slice, and convert slice data into a Transferable object array;

[0009] Each Transferable object array is allocated to a corresponding worker, the worker is used to convert the Transferable object array into text, and the text is saved in an indexedDB according to the key value;

[0010] The main thread obtains a key value array, and queries corresponding data from the indexedDB according to the key value array.

[0011] According to one of the preferred embodiments of the present application, after the slice is converted into a Transferable object array, an object ownership is further passed to each worker, and when the object ownership is transferred to the corresponding worker, the object is in an unusable state in the original context state.

[0012] According to another preferred embodiment of the present application, the object passing the ownership needs to add an instance object of an Array Buffer or a MessagePort or an ImageBitMap type, and null cannot be transmitted in the Transferable object array.

[0013] According to another preferred embodiment of the present application, after the Transferable object array is converted into text and stored, a callback event of each worker is listened to through the main thread, and data is processed according to the callback event.

[0014] According to another preferred embodiment of the present application, each callback event is processed in sequence, all processed data is stored in the indexedDB when all data processing is completed, data in the cache is recalled, and the main thread memory always occupies the space before worker thread calculation, so as to optimize the browser memory occupation and the main thread space.

[0015] According to another preferred embodiment of the present application, after the Transferable object array after slicing is converted into text data in the worker, the text data is saved according to the set key value and the indexedDB characteristics, the key value is used to identify the text data of the corresponding slice, and a key value pair is generated and saved in the indexedDB.

[0016] According to another preferred embodiment of the present invention, the main thread obtains a key array and queries the corresponding key values ​​according to the key array and the key-value query method provided by IndexedDB, in order to obtain all the stored data.

[0017] According to another preferred embodiment of the present invention, the main thread extracts the sliced ​​data portion stored in indexedDB into memory for processing based on the obtained key array, thereby improving the efficiency of data processing.

[0018] To achieve at least one of the above-mentioned objectives, the present invention further provides a front-end big data computing and processing system, wherein the system executes the aforementioned front-end big data computing and processing method.

[0019] The present invention further provides a computer-readable storage medium storing a computer program, the computer program being executable by a processor to provide the aforementioned front-end big data computing and processing method. Attached Figure Description

[0020] Figure 1 The diagram shown is a flowchart of a front-end big data computing and processing method according to the present invention. Detailed Implementation

[0021] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0022] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0023] Please combine Figure 1This invention discloses a front-end big data processing method and system. The method includes: slicing acquired data using a main thread; identifying the slices; converting the sliced ​​data into an array of Transferable objects; further converting the Transferable object array into text data; storing the sliced ​​text data in an IndexedDB database; and using the IndexedDB database to query and extract the sliced ​​data for processing. The storage functionality of the IndexedDB database reduces the memory usage of the main thread, thereby optimizing browser memory and the main thread.

[0024] Specifically, this invention uses streaming compression processing as an example: the system pre-defines compression and decompression algorithms at the front and back ends to perform data compression and decompression operations. The compression algorithms include, but are not limited to, the LZW algorithm and the Huffman compression algorithm. Furthermore, it pre-defines offset information for each individual segment, where the segment offset information is used to fragment the IP packets of the data into smaller IP packets, and each fragmented IP packet has a corresponding corresponding fragmented IP packet. After the data acquired from the main line is fragmented and compressed according to the pre-determined compression algorithm, a key needs to be configured for each fragmented compressed data. This key serves as a unique identifier and is used to construct key-value pairs. After completing the fragmented compression, a worker is generated for the fragmented compressed data, and the fragmented compressed data is converted into an array of Transferable objects. This array of Transferable objects is implemented using the Transferable interface, which includes: returning an object representing the data to be transmitted; and returning an array of DataFlavor objects to provide the data's flavor, and indicating whether the object supports the specified flavor. After creating the worker for each slice of compressed data, the array of converted Transferable objects is further input into the worker for data conversion, transforming the Transferable objects into text data. The methods for converting the Transferable objects into text data include, but are not limited to, the `value` function and the `text` function. The `value` function can convert numeric data into text, and the `text` function can convert object data into text data in text format. It should be noted that the functions and algorithms for converting the array of Transferable objects into text are merely illustrative examples; this invention can be implemented with any other text conversion functions or algorithms agreed upon with the front-end.

[0025] It should be noted that after the main thread converts the sliced ​​compressed data into an array of Transferable objects and generates a worker for each slice, the present invention further passes ownership of the Transferable object to each worker. The Transferable interface represents an object that can be passed between different executable contexts, such as between the main thread and the worker.

[0026] This is an abstract interface; no object belongs to this type. It doesn't define any methods or properties; it's simply a label indicating that an object is available in specific situations, such as when passed to a Worker via the Worker.postMessage() method.

[0027] After transferring ownership of an object to the worker via the Transferable object array, the original context of the object becomes unusable; it only becomes usable after being transferred to the worker. The objects that can be transferred are instances of ArrayBuffer, MessagePort, or ImageBitMap types, and null values ​​cannot be added to the transferList array. Each array generates a transferList array, and the object data is converted into text data within the worker using the agreed-upon text conversion algorithm. After the text data conversion is complete, it is stored locally using the indexedDB and the generated key values. The key value serves as the query key, and the corresponding key-value pair stores the corresponding text data. Local storage is achieved by constructing key-value pairs. It should be noted that the indexedDB is a browser database and also a document-oriented database, fully integrated into a sandbox environment.

[0028] Because each worker thread has a callback operation set up, and the main thread listens for the callback events of each worker thread, the data processing status can be uniformly managed and monitored through these callback events. Once the main thread detects that all worker threads have finished processing data, it stores the processed data in the corresponding IndexedDB, thus ensuring that cached data is reclaimed. Therefore, the main thread's memory usage is always only the amount of memory used before the other workers' computations, which better optimizes the memory usage of both the main thread and the browser. Since the text data is in a sliced ​​state, when the main thread obtains the key value and queries the IndexedDB to retrieve the corresponding text data, it can retrieve a portion of the data separately, thus avoiding loading all the data into memory and reducing memory processing pressure.

[0029] Therefore, the above technical solution can solve the following technical problems: It improves computational efficiency by using Web Workers provided by the browser to achieve multi-process parallel computing; it optimizes the problem of objects being copied multiple times, which is a common issue with ordinary worker solutions, by passing a transferList between workers to transfer data ownership. Furthermore, the above technical solution can achieve the following technical effects: It utilizes the storage capabilities of IndexedDB to achieve data persistence while simultaneously utilizing multiple workers for parallel computation. After computation, it releases the worker from memory usage, achieving optimal space optimization. It also leverages the local storage characteristics of IndexedDB to minimize tab memory usage, making garbage collection smoother and reducing the amount of data the main thread needs to retrieve and store.

[0030] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0031] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0032] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.

Claims

1. A front-end big data computing and processing method, characterized in that, The method includes: Start the browser's main thread, obtain the data to be processed according to the browser's main thread, preset a compression algorithm, compress the data to be processed according to the compression algorithm, and generate separate slices; The individual slices are preset with a slice offset, and a key value is set for each slice. A corresponding worker is generated for each slice, and the slice data is converted into an array of Transferable objects. Each Transferable object array is assigned to a corresponding worker, which converts the Transferable object array into text and stores it in the indexedDB according to the key value. The main thread retrieves an array of key values ​​and queries the corresponding data from the indexedDB based on the array of key values. The array of Transferable objects is implemented by the Transferable interface. The implementation of the Transferable interface includes: returning an object, which is the data to be transferred; returning an array of DataFlavor objects to provide the data's flavor; and returning whether the object supports the specified flavor.

2. The front-end big data computing and processing method according to claim 1, characterized in that, After converting the slice into an array of Transferable objects, the ownership of the objects is further passed to each worker. Once the ownership of the objects is transferred to the corresponding worker, the objects are in an unavailable state in the original context.

3. The front-end big data computing and processing method according to claim 2, characterized in that, The object to which ownership is transferred needs to be an instance of type ArrayBuffer, MessagePort, or ImageBitMap, and null cannot be passed into the Transferable object array.

4. The front-end big data computing and processing method according to claim 1, characterized in that, After converting the array of Transferable objects into text and storing it, the main thread listens for the callback events of each worker and processes the data according to the callback events.

5. The front-end big data computing and processing method according to claim 4, characterized in that, Each callback event is processed sequentially. Once all data has been processed, all processed data is stored in the indexedDB, and cached data is retrieved. This ensures that the main thread memory always occupies the space before the worker thread's computation, thereby optimizing browser memory usage and main thread space.

6. The front-end big data computing and processing method according to claim 1, characterized in that, After the sliced ​​Transferable object array is converted into text data inside the worker, the text data is saved according to the set key value and indexedDB characteristics, wherein the key value is used to identify the text data of the corresponding slice, and key-value pairs are generated and saved in the indexedDB.

7. The front-end big data computing and processing method according to claim 1, characterized in that, The main thread obtains the key array and queries the corresponding key values ​​according to the key array and the key-value query method provided by IndexedDB, in order to retrieve all the saved data.

8. The front-end big data computing and processing method according to claim 1, characterized in that, The main thread extracts the sliced ​​data stored in IndexedDB into memory for processing based on the obtained key array, thereby improving the efficiency of data processing.

9. A front-end big data computing and processing system, characterized in that, The system executes a front-end big data computing and processing method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that can be executed by a processor according to any one of claims 1-8, a front-end big data computing and processing method.

Images