Data processing system and method

Abstract

The invention discloses a data processing system which comprises a data display and export module, a background message processing module, a thread pool configuration and processing module, a message notification module and a file downloading module. The method for using the data processing system comprises the steps that S1, a user initiates a request to a background on a front-end interface, and a background service generates a task entity according to the request after receiving the request; s2, processing the received task entities in parallel by a plurality of working threads created by the thread pool; and S3, after the front-end interface receives the notification message of the background service, guiding the user to store the file. According to the data processing method, by adopting a mode of combining asynchronous multi-thread export with message notification, the pressure on disk I / O operation during export of a large amount of data can be effectively reduced, and the problem of high consumption of server resources can be solved, so that the response speed and the overall stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system and method. Background Art

[0002] In the daily operation of Web management systems, the data export function plays a vital role. It allows users to export data in the system to local computers in various formats, such as Excel, Word documents, PDF, and plain text files. Excel has become a widely used tool due to its powerful data processing capabilities and wide compatibility. Traditional data export methods usually adopt a synchronous method, that is, after the user initiates an export request on the front-end interface, the back-end service responds immediately and performs a series of operations: querying the required data from the database, loading the data into the memory for processing, then writing the processed data to a file, and finally returning the file to the front-end, and the user downloads the file to the user's local disk through the browser.

[0003] However, as the size of the database continues to expand, when faced with the need to export large amounts of data, the intensive demand for disk I / O operations (input / output operations) and the large amount of server memory resources occupied have restricted the system's response speed and stability. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned background technical problems and to provide a data processing system and method.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A data processing system comprises: a data display and export module, which is used to build an interactive front-end interface to visually display, conditionally filter and export data; the data display and export module is connected to a background message processing module, which is used to convert the data export request initiated by the front-end interface into a message and write it into a message queue for storage, and after receiving the message, the background service converts several messages into several task entities one by one; the background message processing module is connected to a thread pool configuration and processing module, which is used to pre-create multiple working threads and asynchronously process the received several task entities through the multiple working threads; the thread pool configuration and processing module is connected to a message notification module, which is used to monitor the status and progress of the multiple working threads and generate message notifications; the message notification module is connected to a file download module, which is used to guide users to save files locally.

[0007] A data processing method, the method using a data processing system, the method comprising:

[0008] S1. The user initiates a request to the backend in the frontend interface. After receiving the request, the backend service generates a task entity according to the request.

[0009] S2. Several worker threads created by the thread pool process several received task entities in parallel;

[0010] S3. After the front-end interface receives the notification message from the background service, it guides the user to save the file.

[0011] As a further solution of the present invention: in the step S1, it includes:

[0012] S11. After receiving the request, the background service performs parameter verification and data processing. After the processing is completed, the relevant parameters are encapsulated into a message, sent to the message queue, and confirmation information is fed back to the front-end interface.

[0013] As a further solution of the present invention: in the step S1, it includes:

[0014] S12. The background service monitors the message queue, consumes messages from the message queue, converts them into task entities, and submits them to the thread pool.

[0015] As a further solution of the present invention: in the step S2, it includes:

[0016] S21. Query the data records that meet the conditions from the database and load them into the memory.

[0017] As a further solution of the present invention: in the step S2, it includes:

[0018] S22. Divide the data to be exported in the memory into data blocks, which are written by several sub-threads in the working thread.

[0019] As a further solution of the present invention: in the step S2, it includes:

[0020] S23. After all child threads have completed writing, the data is persisted to disk and an export file is generated.

[0021] As a further solution of the present invention: in the step S21, the data loaded into the memory needs to be logically processed, and the logical processing includes: data cleaning, data conversion, and data aggregation.

[0022] Beneficial effects of the present invention:

[0023] In the present invention, by adopting the method of asynchronous multi-threaded export combined with message notification, the pressure on disk I / O operations when exporting large amounts of data and the high consumption of server resources can be effectively reduced, thereby improving the response speed and overall stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below in conjunction with the accompanying drawings.

[0025] Figure 1 It is a schematic diagram of the structure of the data processing system of the present invention;

[0026] Figure 2 It is a flow chart of the data processing method of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 As shown, the present invention is a data processing method, comprising:

[0029] Data display and export module: The data display and export module is used to build an interactive front-end interface, visualize the data, and is equipped with a conditional filtering function to facilitate users to easily export the required data.

[0030] The data display and export module is connected to the background message processing module. The background message processing module is used to convert the data export request initiated by the front-end interface into a message and write it into the message queue for storage. After receiving the message, the background service converts several messages into several task entities one by one.

[0031] Specifically, the background service can integrate the message middleware and continuously monitor the message queue. After receiving the data export request initiated by the front-end interface, it will immediately convert the request into a message and store it. Subsequently, the message consumer consumes the message from the message queue and converts it into a task entity.

[0032] The background message processing module is connected to the thread pool configuration and processing module, and the thread pool configuration and processing module is used to pre-create multiple working threads and asynchronously process several received task entities through multiple working threads.

[0033] The working thread divides the data that needs to be exported by each task entity into several parts, uses a multi-threading mechanism to process each part in parallel, and uses the computing power of multi-core processors to reduce the total data processing time to ensure that the task entity can be responded to and executed quickly.

[0034] The thread pool configuration and processing module is connected to the message notification module, which is used to monitor the status and progress of multiple working threads and generate message notifications.

[0035] The message notification module is connected to the file download module. When the message notification module detects that the task entity processing is completed, it sends a message notification to the file download module. After receiving the message notification, the file download module will provide a file download page to guide the user to save the file locally.

[0036] See also Figure 2 As shown, in another embodiment, a method for data processing includes:

[0037] S1. The user initiates an HTTP request to the backend in the front-end interface through the data display and export module. The backend message processing module, as a backend service, generates a task entity according to the request after receiving the request.

[0038] The specific steps include:

[0039] S11. After receiving the request, the background message processing module performs parameter verification and data processing. After the processing is completed, the relevant parameters are encapsulated into a message and sent to the message queue. After the message is successfully added to the queue, the background service will immediately feedback confirmation information to the front-end interface, indicating that the request has been received and is preparing to execute the export task. At this time, although the background service will respond to the request, it will not execute it immediately. Instead, the task will be added to the message queue for asynchronous processing.

[0040] Asynchronous processing is a programming pattern that allows a program to continue performing other tasks while waiting for an operation to complete, rather than blocking the current thread.

[0041] S12. The background service monitors the message queue, consumes messages from the message queue, converts them into task entities, and submits them to the thread pool.

[0042] S2. The thread pool configuration and processing module pre-configures the thread pool. Specifically, the thread pool configuration and processing module initializes thread pool parameters based on business needs and server resource conditions, and pre-creates multiple worker threads. The multiple worker threads are respectively recorded as thread A, thread B, thread C... so that several worker threads can process several received task entities in parallel to ensure that the task entities can be responded to and executed quickly.

[0043] The specific steps include:

[0044] S21, the thread pool configuration and processing module queries the qualified data records from the database and loads them into the memory.

[0045] The data loaded into the memory needs to be processed logically, including data cleaning, data conversion, and data aggregation.

[0046] Data cleaning requires identifying and processing missing values. You can choose to delete, fill, or replace them with default values. You also need to check and remove duplicate data records, and then convert the data into a suitable format (including but not limited to date format and numeric type).

[0047] Data conversion requires data standardization and feature engineering. Data standardization can make the data conform to predetermined specifications. Feature engineering requires extracting and constructing new features based on analysis requirements, including but not limited to normalization and encoding.

[0048] Data aggregation can be done by grouping operations to categorize and aggregate data, such as by region, category, or time period.

[0049] S22. The data to be exported in the memory is divided into data blocks, which are written by several sub-threads in the working thread. The writing efficiency can be improved by dividing the data into blocks.

[0050] Specifically, a number of sub-threads can be created in each working thread through the thread pool configuration and processing module. For example, thread A has two sub-threads, thread a1 and thread a2, and each sub-thread writes a data block.

[0051] S23. After all child threads have completed writing, the data is exported and persisted to disk, generating an export file.

[0052] S3. After the front-end interface receives the notification message from the background service, it guides the user to save the file.

[0053] Specifically, the background message notification module monitors the status and progress of multiple working threads and generates message notifications. After receiving the message notifications, the file download module provides a file download page to guide the user to save the file locally, thereby completing the export of the entire data.

[0054] In this application, through asynchronous processing, during the task export process, users can handle other businesses, reduce the user's waiting time, and thus improve the user experience. When a new message is added to the message queue, the background service takes out the message from the message queue and converts it into a task entity, which is then processed by the thread pool. Each worker thread in the thread pool executes the task entity separately, so that multiple task entities can be processed in parallel. When executing the task entity, the thread divides the data into several data blocks, and uses a multi-threading mechanism to process each data block in parallel. This method can make full use of the computing power of multi-core processors, significantly reduce the total time of data processing, reduce the pressure of disk I / O operations, and improve the overall stability of the system. In addition, since the data export operation is performed in the child thread, it will not block the operation of the worker thread or other services, ensuring the normal operation and response speed of the system.

[0055] It is understandable that the status and progress of each work thread can be monitored through the message notification module, so that the user can be informed immediately through message notification after the task entity is completed, thereby enhancing the controllability and transparency of the task.

[0056] In the present invention, by adopting asynchronous multi-threaded export combined with message notification, the problem of high consumption of server resources when exporting large amounts of data can be effectively solved. When processing large-scale data sets, it can not only improve the user experience, but also maintain the response speed and overall stability of the system.

[0057] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A data processing system, characterized in that: include: Data display and export module, which is used to build an interactive front-end interface to visualize data, filter conditions and export data; The data display and export module is connected to the background message processing module, and the background message processing module is used to convert the data export request initiated by the front-end interface into a message and write it into a message queue for storage. After receiving the message, the background service converts several messages into several task entities one by one; The background message processing module is connected to the thread pool configuration and processing module, and the thread pool configuration and processing module is used to pre-create multiple working threads and asynchronously process the received task entities through the multiple working threads; The thread pool configuration and processing module is connected to a message notification module, and the message notification module is used to monitor the status and progress of the plurality of working threads and generate message notifications; The message notification module is connected to a file download module, and the file download module is used to guide the user to save the file locally.

2. A data processing method, characterized in that: The method uses the data processing system of claim 1, and the method comprises: S1. The user initiates a request to the backend in the frontend interface. After receiving the request, the backend service generates a task entity according to the request. S2. Several worker threads created by the thread pool process several received task entities in parallel; S3. After the front-end interface receives the notification message from the background service, it guides the user to save the file.

3. The method according to claim 2, characterized in that: In the step S1, it includes: S11. After receiving the request, the background service performs parameter verification and data processing. After the processing is completed, the relevant parameters are encapsulated into a message, sent to the message queue, and confirmation information is fed back to the front-end interface.

4. The method according to claim 3, characterized in that: In the step S1, it includes: S12. The background service monitors the message queue, consumes messages from the message queue, converts them into task entities, and submits them to the thread pool.

5. The method according to claim 2, characterized in that: In the step S2, it includes: S21. Query the data records that meet the conditions from the database and load them into the memory.

6. The method according to claim 5, characterized in that: In the step S2, it includes: S22. Divide the data to be exported in the memory into data blocks, which are written by several sub-threads in the working thread.

7. The method according to claim 6, characterized in that: In the step S2, it includes: S23. After all child threads have completed writing, the data is persisted to disk and an export file is generated.

8. The method according to claim 5, characterized in that: In the step S21, the data loaded into the memory needs to be logically processed, and the logical processing includes: data cleaning, data conversion, and data aggregation.

Images