Front-end data processing method, system, terminal and storage medium

Abstract

The present invention relates to the field of server technology, and specifically provides a front-end data processing method, system, terminal and storage medium, including: slicing a large file to be processed, the large file being a file whose size exceeds a set threshold, and processing the slicing data in parallel; collecting the browsing behavior data of the user, and predicting the demand data in the large file data and derived data based on the browsing behavior data; loading the demand data and calculating and rendering the demand data. The present invention divides the large file into pieces and only loads the demand data through slicing and demand prediction based on user behavior, thereby reducing the difficulty of large file processing and improving the fluency of the display interface.

Description

Technical Field

[0001] The present invention belongs to the technical field of servers, and in particular relates to a front-end data processing method, system, terminal and storage medium. Background Art

[0002] In today's Web development field, with the rapid growth of data volume, Web front-ends face a series of severe technical challenges when processing large-scale data sets. Among them, the most significant problem is that processing and rendering large data sets often cause obvious lags on the front-end page and a significant decrease in response speed, which greatly affects the overall user experience.

[0003] Specifically, when the data size reaches a certain level, the front-end page will take longer to load, process and render the data, which will cause users to experience significant delays when operating the page. In addition, the complexity and diversity of large data sets also make the rendering of the front-end page more difficult, and rendering errors or incomplete rendering are prone to occur.

[0004] What's more serious is that if the front-end tries to load and process all the data at once, it will not only take up a lot of memory resources, but may also cause the browser to crash due to insufficient memory. This situation is very bad for users because they may lose unsaved work or need to spend more time reloading and processing data. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a front-end data processing method, system, terminal and storage medium to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a front-end data processing method, comprising: Slice the large file to be processed, the large file being a file whose size exceeds a set threshold, and process the sliced ​​data in parallel; Collecting user browsing behavior data, and predicting demand data in large file data and derived data based on the browsing behavior data; The demand data is loaded and the demand data is calculated and rendered.

[0007] In an optional implementation, a large file to be processed is fragmented, where the large file is a file whose size exceeds a set threshold, and the fragmented data is processed in parallel, including: Get the size of the file to be processed. If the file size exceeds the set threshold, the file to be processed is determined to be a large file. Splitting slice data of corresponding sizes from the large file based on the set slice size, and calculating the hash value of the allocated data; Transmitting the shard data and the hash value to the backend server, wherein the hash value is used to verify the integrity of the shard data; Receive the verification result returned by the backend server. If the verification result is failed, resend the shard data and hash value. If the verification result is passed, continue to split the shard data from the remaining data of the large file until the large file is completely split into shard data.

[0008] In an optional implementation, collecting the browsing behavior data of the user, and predicting the demand data in the large file data and the derived data based on the browsing behavior data, includes: Collecting user browsing behavior data, including mouse click behavior, mouse page turning behavior and sliding speed, and mouse stop position and stop time; Record the area and data type corresponding to the mouse click behavior, and the number of clicks of the statistical data type; Monitor the mouse page turning behavior, collect the current display area and the corresponding data type, and the page turning speed, and calculate the average page turning speed of each data type and the slope of the page turning speed over time; Monitor the area where the mouse stays and the time it stays, and count the data type corresponding to the stay area and the stay time corresponding to the data type; Calculate the interest of a data type based on the number of clicks, average page turning speed and slope, and dwell time corresponding to the data type; Get the type, display content and mouse behavior of the current display interface; If the type is a secondary interface that only displays the corresponding data type, and the secondary interface is an interface, the data volume of the required data is set based on the page turning speed, and the data after the displayed content is extracted from the corresponding data type based on the data volume to obtain the required data; If the type is a first-level interface containing multiple windows, the window has a corresponding relationship with the data type and the window displays summary data of the corresponding data type, then part of the data type with the highest interest among all the display contents of the first-level interface and the data types corresponding to all windows contained in the current display content will be used as the demand data.

[0009] In an optional implementation, loading the demand data and calculating and rendering the demand data includes: Splitting the rendering task of the demand data into multiple incremental tasks; A plurality of the incremental tasks are executed during the animation cycle of the page.

[0010] In a second aspect, the present invention provides a front-end data processing system, comprising: A file segmentation module, used to segment large files to be processed, wherein the large files are files whose size exceeds a set threshold, and process the segmented data in parallel; A demand prediction module, used to collect the browsing behavior data of users, and predict the demand data in the large file data and the derived data based on the browsing behavior data; The data processing module is used to load the demand data and calculate and render the demand data.

[0011] In an optional implementation, the file segmentation module includes: A file judgment unit, used to obtain the size of the file to be processed, and if the file size exceeds a set threshold, the file to be processed is determined to be a large file; A shard processing unit, used for splitting shard data of corresponding size from the large file based on a set shard size, and calculating a hash value of the allocated data; A transmission management unit, used to transmit the shard data and the hash value to the backend server, wherein the hash value is used to verify the integrity of the shard data; The step-by-step execution unit is used to receive the verification result returned by the back-end server. If the verification result is failed, the shard data and hash value are resent. If the verification result is passed, the shard data is continued to be split from the remaining data of the large file until the large file is completely split into shard data.

[0012] In an optional implementation, the demand forecasting module includes: A data collection unit is used to collect the browsing behavior data of the user, wherein the browsing behavior data includes the mouse clicking behavior, the mouse page turning behavior and sliding speed, and the mouse staying position and staying time; Click analysis unit, used to record the area corresponding to the mouse click behavior and the corresponding data type, and the number of clicks of statistical data type; The page turning analysis unit is used to monitor the mouse page turning behavior, collect the current display area and the corresponding data type, and the page turning speed, and calculate the average page turning speed of each data type and the slope of the page turning speed over time; A dwell analysis unit, used to monitor the area where the mouse is located and the dwell time, and to count the data type corresponding to the dwell area and the dwell time corresponding to the data type; An interest calculation unit, used to calculate the interest of a data type based on the number of clicks, average page turning speed and slope, and dwell time corresponding to the data type; A state acquisition unit, used to obtain the type, display content and mouse behavior of the current display interface; A first prediction unit is used for, if the type is a secondary interface that only displays a corresponding data type, and the secondary interface is an interface, setting a data volume of the required data based on a page turning speed, and extracting data after the displayed content from the corresponding data type based on the data volume to obtain the required data; The second prediction unit is used to use part of the data type with the highest interest among all the display contents of the first-level interface and the data types corresponding to all windows contained in the current display content as the demand data if the type is a first-level interface including multiple windows, the windows have a corresponding relationship with the data types and the windows display summary data of the corresponding data types.

[0013] In an optional embodiment, the data processing module includes: A task splitting unit, used for splitting the rendering task of the demand data into multiple incremental tasks; The rendering execution unit is used to execute a plurality of the incremental tasks within the animation cycle of the page.

[0014] In a third aspect, a terminal is provided, including: A memory, used for storing a front-end data processing program; The processor is used to implement the steps of the front-end data processing method provided in the first aspect when executing the front-end data processing program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, on which a front-end data processing program is stored. When the front-end data processing program is executed by a processor, the steps of the front-end data processing method provided in the first aspect are implemented.

[0016] The beneficial effect of the present invention is that the front-end data processing method, system, terminal and storage medium provided by the present invention divide and process large files through slicing and demand prediction based on user behavior, and only load the required data, thereby reducing the difficulty of large file processing and improving the fluency of the display interface.

[0017] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0020] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0021] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0024] The front-end data processing method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the front-end data processing system runs in the computer device.

[0025] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a front-end data processing system. According to different requirements, the order of the steps in the flow chart may be changed, and some may be omitted.

[0026] like Figure 1 As shown, the method includes: Step 110, fragmenting the large file to be processed, wherein the large file is a file whose size exceeds a set threshold, and processing the fragmented data in parallel; Step 120, collecting the browsing behavior data of the user, and predicting the demand data in the large file data and the derived data based on the browsing behavior data; Step 130: load the demand data and calculate and render the demand data.

[0027] To facilitate understanding of the present invention, the front-end data processing method provided by the present invention is further described below based on the principle of the front-end data processing method of the present invention and in combination with the process of processing the front-end data in the embodiment.

[0028] Specifically, the front-end data processing method includes: S1. Slice a large file to be processed, where the large file is a file whose size exceeds a set threshold, and process the sliced ​​data in parallel.

[0029] Obtain the size of the file to be processed, and if the file size exceeds a set threshold, determine that the file to be processed is a large file; split slice data of corresponding size from the large file based on the set slice size, and calculate the hash value of the allocated data; transmit the slice data and the hash value to the back-end server, and the hash value is used to verify the integrity of the slice data; receive the verification result returned by the back-end server, and if the verification result is that the verification fails, resend the slice data and the hash value, and if the verification result is that the verification passes, continue to split the slice data from the remaining data of the large file until the large file is completely split into slice data.

[0030] After receiving all the shard data, the backend server integrates all the shard data into a large file and processes the large file according to the preset processing logic, where the preset processing logic includes data classification, data statistics, data sorting, drawing charts, etc.

[0031] In a specific example, the following steps are included: S101. Get the size of the file to be processed The system detected a file named large_data_file.csv to be processed. Get the size of the file: 10GB.

[0032] S102. Determine large files Set the threshold to 5GB. Since the file size of large_data_file.csv (10GB) exceeds the threshold, it is considered a large file.

[0033] S103. Split large files Set the shard size to 100MB. Split large_data_file.csv into 100 shards of 100MB each.

[0034] S104. Calculate hash value Calculate the SHA-256 hash value for each shard data. For example: Shard 1: data_chunk_1.csv, hash value: hash1 Shard 2: data_chunk_2.csv, hash value: hash2 ... Shard 100: data_chunk_100.csv, hash value: hash100 S105. Transmit shard data and hash value to backend server The client sends each shard data and the corresponding hash value to the backend server through an HTTP POST request.

[0035] Request example: for i in range(1, 101): send_to_server(f"data_chunk_{i}.csv", f"hash{i}").

[0036] S106. Verify shard data After receiving the shard data and hash value, the backend server recalculates the hash value for the shard data and compares it with the transmitted hash value.

[0037] If the hash values ​​do not match, the server responds with "Verification Failed".

[0038] If the hash values ​​match, the server responds with "Verification Passed".

[0039] S107. Processing verification results The client receives the verification result: If a shard fails verification, resend the shard data and hash value.

[0040] If all shards pass verification, continue processing the remaining shard data until all shard data is sent.

[0041] S108. Backend server integrates shard data After all the shard data (data_chunk_1.csv to data_chunk_100.csv) are received and verified, the backend server integrates all the shard data into a complete large file large_data_file_reconstructed.csv.

[0042] S109. Processing large files The backend server processes the data shards of large files in parallel according to the preset processing logic. The processing method is as follows: Data classification: Data is classified and stored according to specific fields.

[0043] Data statistics: Calculate the statistical values ​​of certain fields (such as average, sum, etc.).

[0044] Data Sorting: Sort the data for further analysis.

[0045] Draw charts: Generate various charts (such as bar charts, line charts, etc.) based on the processing results.

[0046] In this way, the metadata in the large file and the derivative data generated by the backend server are obtained.

[0047] In order to rationally plan data sharding, in another embodiment of the present invention, data allocation is dynamically adjusted. According to device performance and network conditions, parameters such as shard size, lazy calculation threshold, virtual scrolling cache size, etc. are dynamically adjusted to ensure efficient operation in different environments. The front-end system performance is monitored in real time, and the data processing strategy is dynamically adjusted to ensure that the system remains stable under high-load scenarios.

[0048] Specifically, the following steps are included: Use the API to obtain device performance information: use the browser's navigator.hardwareConcurrency to obtain the number of CPU cores, navigator.deviceMemory to obtain the device memory size, and window.innerWidth and window.innerHeight to obtain screen resolution and other information.

[0049] Detect network status: Use navigator.connection (or the deprecated navigator.mozConnection, navigator.webkitConnection) to obtain network type (such as WiFi, 4G, 3G, etc.) and bandwidth information. You can also use third-party libraries such as network-information to obtain more detailed network status.

[0050] User interaction feedback: Indirectly evaluate the processing power of the current device through the response time of user interactions (such as button clicks, page scrolling, etc.).

[0051] Define shard size: Set the shard size for data loading and processing based on device performance and network conditions. For example, on high-performance devices and high-speed networks, you can increase the shard size to reduce the number of requests; on low-performance devices or low-speed networks, you can reduce the shard size to reduce the burden of single processing.

[0052] Set lazy evaluation threshold: Determine when lazy evaluation is triggered (i.e. delaying evaluation until it is really needed). This can be adjusted dynamically based on the device's CPU usage and memory footprint.

[0053] Configure virtual scroll cache size: Dynamically adjust the number of cache items in the virtual scroll list according to the screen size and scroll speed. When scrolling at high speed, you can increase the cache to reduce scrolling lag; when scrolling at low speed or still, you can reduce the cache to save memory.

[0054] Write performance monitoring functions: use performance.now() or Date.now() to monitor function execution time, use PerformanceObserver to monitor resource loading time, etc.

[0055] Implement parameter adjustment algorithms: Based on the collected performance data, implement an algorithm to dynamically adjust parameters such as shard size, lazy calculation threshold, and virtual scroll cache size. This may require some experiments and tuning to find the best balance.

[0056] Trigger adjustment mechanism: When performance data changes significantly (such as network type switching, device entering low power mode, etc.), the parameter adjustment mechanism is triggered.

[0057] Integrated performance monitoring tools: Use performance monitoring tools such as Sentry, New Relic, Google Analytics, etc. to track the performance indicators of the front-end system in real time.

[0058] Set up performance alerts: Configure performance monitoring tools to send alerts when key performance indicators fall below thresholds.

[0059] Analyze performance bottlenecks: Check performance monitoring reports regularly to identify and resolve performance bottlenecks.

[0060] Implement data loading strategy: Dynamically adjust data loading strategy based on network conditions and shard size. For example, when the network is poor, you can reduce the amount of data loaded at one time and increase the number of retries when loading fails.

[0061] Optimize data processing flow: Dynamically adjust data processing flow according to device performance. For example, on low-performance devices, the complexity and frequency of data processing can be reduced.

[0062] Implement degradation strategy: In high-load scenarios, implement degradation strategy to sacrifice some functions to ensure system stability. For example, you can turn off some non-core functions or reduce image quality to reduce resource consumption.

[0063] S2. Collecting the browsing behavior data of the user, and predicting the demand data in the large file data and the derived data based on the browsing behavior data.

[0064] Collecting user browsing behavior data, the browsing behavior data includes mouse click behavior, mouse page turning behavior and sliding speed, and mouse stay position and stay time; recording the area corresponding to the mouse click behavior and the corresponding data type, and counting the number of clicks of the data type; monitoring the mouse page turning behavior, collecting the current display area and the corresponding data type, and the page turning speed, and counting the average page turning speed of each data type and the slope of the page turning speed over time; monitoring the area where the mouse stays and the stay time, counting the data type corresponding to the stay area, and the stay time corresponding to the data type; calculating the interest of the data type based on the number of clicks, average page turning speed and slope, and stay time corresponding to the data type; obtaining the type, display content and mouse behavior of the current display interface; if the type is a secondary interface that only displays the corresponding data type, and the secondary interface is an interface, then setting the data volume of the required data based on the page turning speed, and extracting the data after the display content from the corresponding data type based on the data volume to obtain the required data; if the type is a primary interface containing multiple windows, the window has a corresponding relationship with the data type and the window displays summary data of the corresponding data type, then taking part of the data of the data type with the highest interest among all the display contents of the primary interface and the data types corresponding to all windows contained in the current display content as the required data.

[0065] In a specific example, the demand data forecasting method is as follows: S201. Data Collection S201.1 Collecting mouse click behavior Implementation: Use JavaScript event listeners (such as click events) to capture the user's mouse click behavior.

[0066] Recorded content: coordinates of the mouse click (relative to the page or specific element), click timestamp, and data type corresponding to the click location (such as news, products, videos, etc.).

[0067] S201.2 Collect mouse page turning behavior Implementation method: Listen to scroll events (such as scroll events) or paging button click events.

[0068] Recorded content: currently displayed page area (can be calculated by scrolling position), page turning timestamp, data type before and after page turning, and page turning speed (obtained by calculating the time difference before and after page turning and the scrolling distance).

[0069] S201.3 Collect sliding speed and mouse position and dwell time Implementation: Use event listeners such as mousemove and mouseover to capture sliding behavior, and events such as mouseenter and mouseleave to capture dwelling behavior.

[0070] Recorded content: sliding speed (obtained by calculating the distance and time difference of mouse movement), stop position coordinates, stop timestamp, and data type corresponding to the stop position.

[0071] S202. Data statistics and analysis S202.1 Counting clicks Implementation: Maintain a counter of the number of clicks on the data type and update it every time the mouse is clicked.

[0072] S202.2 Statistical average page turning speed and the slope of page turning speed over time Implementation method: For each data type, maintain a list of page turning speed records and calculate the average value. At the same time, record the time series of page turning speed and use linear regression and other methods to calculate the slope.

[0073] S202.3 Statistics of stay time Implementation method: For each data type, maintain an accumulator of the dwell time, which is updated every time the mouse stays.

[0074] S202.4 Calculating the Interest of Data Types Implementation: Calculate interest based on factors such as number of clicks, average page turning speed (and its slope if time changes are considered), dwell time, etc. using weighted summation or machine learning models.

[0075] S203. Demand data extraction S203.1 Get the type and content of the current display interface Implementation method: Get the type and display content of the current interface through state management or DOM operation of the front-end framework (such as React, Vue, etc.).

[0076] S203.2 Determine the interface type and extract required data Secondary interface processing: Implementation method: If the current interface is a secondary interface that only displays the corresponding data type, dynamically adjust the amount of data to be loaded according to the page turning speed.

[0077] Data extraction: Request more data of the corresponding data type from the server to meet the user's possible page turning needs.

[0078] First level interface processing: Implementation method: If the current interface is a first-level interface containing multiple windows, and the windows have a corresponding relationship with the data types, extract part of the data according to the data type with the highest interest.

[0079] Data extraction: Request partial data from the server for the most interesting data type to update the summary information of the corresponding window.

[0080] S204. Data Request and Update Implementation method: Use the front-end HTTP client (such as Axios, Fetch API, etc.) to send data requests to the server.

[0081] S3: Load the demand data and calculate and render the demand data.

[0082] S301. Split the rendering task of the demand data into multiple incremental tasks.

[0083] First, split the original rendering task (which may be rendering a large data set or a complex UI) into multiple smaller incremental tasks. These incremental tasks should be completed in a short time to ensure that they can be executed within the animation cycle of the page.

[0084] step: Identify separable parts: Identify the parts of the rendering task that can be executed independently, such as rendering list items, drawing charts, etc.

[0085] Divide incremental tasks: further divide the divisible parts into multiple incremental tasks, each task is responsible for rendering a part of the data.

[0086] Manage task queues: Create a task queue to manage these incremental tasks and ensure that they can be executed in order.

[0087] S302: Utilize the request Animation Frame API to execute the plurality of incremental tasks within the animation cycle of the page.

[0088] Request Animation Frame is an API provided by the browser to call a specified callback function before the next repaint. It allows you to synchronize the execution of your code with the browser's rendering cycle, thereby optimizing performance and reducing lag.

[0089] step: Define callback function: Write a callback function that will execute an incremental task and remove the task from the task queue.

[0090] Call request Animation Frame: In the callback function, check whether there are any remaining tasks in the task queue. If so, call request Animation Frame again and pass in the callback function to execute the next incremental task in the next animation cycle.

[0091] Execute incremental tasks: In the callback function, execute the current incremental task and update the UI (such as DOM).

[0092] In addition, in one embodiment of the present invention, in large data display scenarios such as tables and lists, virtual scrolling technology is used to render only the data in the user's visible area, and the rest of the data remains virtual. According to the scrolling speed and direction, the data that is about to enter the visible area is preloaded to avoid blank areas or delayed loading when the user scrolls. The virtual DOM technology is used to perform lazy rendering of the data area that is not visible to the user, effectively improving the rendering efficiency.

[0093] The update methods of DOM data include: monitoring data changes through Proxy, updating DOM only for changed data, avoiding full refresh. Tracking the dependency between data and page elements, accurately locating the page update requirements caused by data changes, and improving update efficiency.

[0094] In some embodiments, the front-end data processing system may include a plurality of functional modules composed of computer program segments. The computer programs of the various program segments in the front-end data processing system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Function of front-end data processing.

[0095] In this embodiment, the front-end data processing system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules of the system 200 may include: a file segmentation module 210, a demand forecasting module 220 and a data processing module 230. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0096] A file segmentation module, used to segment large files to be processed, wherein the large files are files whose size exceeds a set threshold, and process the segmented data in parallel; A demand prediction module, used to collect the browsing behavior data of users, and predict the demand data in the large file data and the derived data based on the browsing behavior data; The data processing module is used to load the demand data and calculate and render the demand data.

[0097] Optionally, as an embodiment of the present invention, the file segmentation module includes: A file judgment unit, used to obtain the size of the file to be processed, and if the file size exceeds a set threshold, the file to be processed is determined to be a large file; A shard processing unit, used for splitting shard data of corresponding size from the large file based on a set shard size, and calculating a hash value of the allocated data; A transmission management unit, used to transmit the shard data and the hash value to the backend server, wherein the hash value is used to verify the integrity of the shard data; The step-by-step execution unit is used to receive the verification result returned by the back-end server. If the verification result is failed, the shard data and hash value are resent. If the verification result is passed, the shard data is continued to be split from the remaining data of the large file until the large file is completely split into shard data.

[0098] Optionally, as an embodiment of the present invention, the demand forecasting module includes: A data collection unit is used to collect the browsing behavior data of the user, wherein the browsing behavior data includes the mouse clicking behavior, the mouse page turning behavior and sliding speed, and the mouse staying position and staying time; Click analysis unit, used to record the area corresponding to the mouse click behavior and the corresponding data type, and the number of clicks of statistical data type; The page turning analysis unit is used to monitor the mouse page turning behavior, collect the current display area and the corresponding data type, and the page turning speed, and calculate the average page turning speed of each data type and the slope of the page turning speed over time; A dwell analysis unit, used to monitor the area where the mouse is located and the dwell time, and to count the data type corresponding to the dwell area and the dwell time corresponding to the data type; An interest calculation unit, used to calculate the interest of a data type based on the number of clicks, average page turning speed and slope, and dwell time corresponding to the data type; A state acquisition unit, used to obtain the type, display content and mouse behavior of the current display interface; A first prediction unit is used for, if the type is a secondary interface that only displays a corresponding data type, and the secondary interface is an interface, setting a data volume of the required data based on a page turning speed, and extracting data after the displayed content from the corresponding data type based on the data volume to obtain the required data; The second prediction unit is used to use part of the data type with the highest interest among all the display contents of the first-level interface and the data types corresponding to all windows contained in the current display content as the demand data if the type is a first-level interface including multiple windows, the windows have a corresponding relationship with the data types and the windows display summary data of the corresponding data types.

[0099] Optionally, as an embodiment of the present invention, the data processing module includes: A task splitting unit, used for splitting the rendering task of the demand data into multiple incremental tasks; The rendering execution unit is used to execute a plurality of the incremental tasks within the animation cycle of the page.

[0100] Figure 3 The present invention provides a schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the front-end data processing method provided in an embodiment of the present invention.

[0101] The terminal 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0102] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.

[0103] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0104] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0105] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0106] Therefore, the present invention divides large files into pieces through sharding and demand prediction based on user behavior, and only loads the required data, thereby reducing the difficulty of large file processing and improving the smoothness of the display interface. The technical effects that can be achieved by this embodiment can be found in the description above and will not be repeated here.

[0107] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.

[0108] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0109] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0110] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0112] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A front-end data processing method, characterized in that: include: Slice the large file to be processed, the large file being a file whose size exceeds a set threshold, and process the sliced ​​data in parallel; Collecting user browsing behavior data, and predicting demand data in large file data and derived data based on the browsing behavior data; The demand data is loaded and the demand data is calculated and rendered.

2. The method according to claim 1, characterized in that Slice a large file to be processed, the large file being a file whose size exceeds a set threshold, and process the sliced ​​data in parallel, including: Get the size of the file to be processed. If the file size exceeds the set threshold, the file to be processed is determined to be a large file. Splitting slice data of corresponding sizes from the large file based on the set slice size, and calculating the hash value of the allocated data; Transmitting the shard data and the hash value to the backend server, wherein the hash value is used to verify the integrity of the shard data; Receive the verification result returned by the backend server. If the verification result is failed, resend the shard data and hash value. If the verification result is passed, continue to split the shard data from the remaining data of the large file until the large file is completely split into shard data.

3. The method according to claim 1, characterized in that Collecting user browsing behavior data, and predicting demand data in large file data and derived data based on the browsing behavior data, including: Collecting user browsing behavior data, including mouse click behavior, mouse page turning behavior and sliding speed, and mouse stop position and stop time; Record the area and data type corresponding to the mouse click behavior, and the number of clicks of the statistical data type; Monitor the mouse page turning behavior, collect the current display area and the corresponding data type, and the page turning speed, and calculate the average page turning speed of each data type and the slope of the page turning speed over time; Monitor the area where the mouse stays and the time it stays, and count the data type corresponding to the stay area and the stay time corresponding to the data type; Calculate the interest of a data type based on the number of clicks, average page turning speed and slope, and dwell time corresponding to the data type; Get the type, display content and mouse behavior of the current display interface; If the type is a secondary interface that only displays the corresponding data type, and the secondary interface is an interface, the data volume of the required data is set based on the page turning speed, and the data after the displayed content is extracted from the corresponding data type based on the data volume to obtain the required data; If the type is a first-level interface containing multiple windows, the window has a corresponding relationship with the data type and the window displays summary data of the corresponding data type, then part of the data type with the highest interest among all the display contents of the first-level interface and the data types corresponding to all windows contained in the current display content will be used as the demand data.

4. The method according to claim 1, characterized in that Loading the demand data and calculating and rendering the demand data includes: Splitting the rendering task of the demand data into multiple incremental tasks; A plurality of the incremental tasks are executed during the animation cycle of the page.

5. A front-end data processing system, characterized in that: include: A file segmentation module, used to segment large files to be processed, wherein the large files are files whose size exceeds a set threshold, and process the segmented data in parallel; A demand prediction module, used to collect the browsing behavior data of users, and predict the demand data in the large file data and the derived data based on the browsing behavior data; The data processing module is used to load the demand data and calculate and render the demand data.

6. The system according to claim 5, characterized in that The file segmentation module includes: A file judgment unit, used to obtain the size of the file to be processed, and if the file size exceeds a set threshold, the file to be processed is determined to be a large file; A shard processing unit, used for splitting shard data of corresponding size from the large file based on a set shard size, and calculating a hash value of the allocated data; A transmission management unit, used to transmit the shard data and the hash value to the backend server, wherein the hash value is used to verify the integrity of the shard data; The step-by-step execution unit is used to receive the verification result returned by the back-end server. If the verification result is failed, the shard data and hash value are resent. If the verification result is passed, the shard data is continued to be split from the remaining data of the large file until the large file is completely split into shard data.

7. The system according to claim 5, characterized in that The demand forecasting module comprises: A data collection unit is used to collect the browsing behavior data of the user, wherein the browsing behavior data includes the mouse clicking behavior, the mouse page turning behavior and sliding speed, and the mouse staying position and staying time; Click analysis unit, used to record the area corresponding to the mouse click behavior and the corresponding data type, and the number of clicks of statistical data type; The page turning analysis unit is used to monitor the mouse page turning behavior, collect the current display area and the corresponding data type, and the page turning speed, and calculate the average page turning speed of each data type and the slope of the page turning speed over time; A dwell analysis unit, used to monitor the area where the mouse is located and the dwell time, and to count the data type corresponding to the dwell area and the dwell time corresponding to the data type; An interest calculation unit, used to calculate the interest of a data type based on the number of clicks, average page turning speed and slope, and dwell time corresponding to the data type; A state acquisition unit, used to obtain the type, display content and mouse behavior of the current display interface; A first prediction unit is used for, if the type is a secondary interface that only displays a corresponding data type, and the secondary interface is an interface, setting a data volume of the required data based on a page turning speed, and extracting data after the displayed content from the corresponding data type based on the data volume to obtain the required data; The second prediction unit is used to use part of the data type with the highest interest among all the display contents of the first-level interface and the data types corresponding to all windows contained in the current display content as the demand data if the type is a first-level interface including multiple windows, the windows have a corresponding relationship with the data types and the windows display summary data of the corresponding data types.

8. The system according to claim 5, characterized in that The data processing module comprises: A task splitting unit, used for splitting the rendering task of the demand data into multiple incremental tasks; The rendering execution unit is used to execute a plurality of the incremental tasks within the animation cycle of the page.

9. A terminal, characterized in that: include: A memory, used for storing a front-end data processing program; A processor is used to implement the steps of the front-end data processing method as described in any one of claims 1 to 4 when executing the front-end data processing program.

10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a front-end data processing program, and when the front-end data processing program is executed by the processor, the steps of the front-end data processing method according to any one of claims 1 to 4 are implemented.

Images