An intelligent asynchronous preprocessing query data method

Abstract

The present application provides an intelligent asynchronous pre-processing query data method, which relates to the field of data processing and storage technology, including: when a query request is initiated, a main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows displayed on a single page, the first page of data is acquired and organized, the task information is recorded in a memory cache, the memory cache is managed, and when the main task is completed, the asynchronous task is started; when a request to switch page numbers is initiated, the memory cache is checked, if the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; otherwise, the data is mined again and asynchronous task processing is started according to the mining status; when an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again according to the cache identifier, and once the data is ready, it can be exported from the database.

Description

Technical Field

[0001] The present application belongs to the field of data processing and storage technology, and specifically relates to an intelligent asynchronous pre-processing query data method. Background Art

[0002] With the rapid progress of information technology, especially the widespread application of big data technology in recent years, the amount of data has shown an exponential growth trend. Especially in the field of financial accounting, the query statistics and export filing requirements of financial statements have become more and more frequent, which places extremely high demands on the performance of data processing systems. The traditional data processing method is usually to obtain all relevant data in full after receiving the query request, and transmit all of them to the front end for display. This method can still cope with relatively small data volumes. For example, when the data volume is at the level of 50,000 to 100,000, it is acceptable for high-performance hardware and network environments. However, when the data volume surges to the level of millions or even tens of millions, the defects of this method are exposed: the query performance is seriously degraded, not only the response time is prolonged, but also the front-end system (such as the browser) may crash.

[0003] To alleviate this problem, the industry currently generally adopts a paging processing mode, which divides the data into multiple small blocks and only displays a part of it (usually the first page) at a time. Although this improves the user experience to a certain extent, when users need to view subsequent pages or export all data, they still face the problem of slow loading. The performance bottleneck of traditional synchronous query processing methods is increasingly obvious when processing large-scale data sets, and it is difficult to meet the requirements of fast response and high efficiency in modern data processing. Summary of the invention

[0004] The present application provides an intelligent asynchronous pre-processing query data method to solve the problem that the traditional synchronous query processing method has performance bottlenecks when facing large-scale data sets and is difficult to meet the needs of fast response and efficient processing.

[0005] The technical solution adopted in this application is:

[0006] The present application embodiment provides an intelligent asynchronous pre-processing query data method, comprising:

[0007] When a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows displayed on a single page, the first page of data is obtained and organized, the task information is recorded in the memory cache, the memory cache is managed, and when the main task is completed, the asynchronous task is started;

[0008] When a request to switch the page number is initiated, the memory cache is checked. If the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; otherwise, the data is mined again and asynchronous task processing is started according to the mining status;

[0009] When an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again based on the cache identifier. Once the data is ready, it can be exported from the database.

[0010] The intelligent asynchronous preprocessing query data method provided by the present application also includes the following additional technical features: when a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows displayed on a single page, the first page of data is obtained and organized, the task information is recorded in the memory cache, and the memory cache is managed, specifically:

[0011] The user initiates a query request, and the backend generates a globally unique identifier as the main task identifier according to the request. The backend calculates the total number of rows of data that meet the query conditions, and calculates the starting row number, the ending row number, and the total number of pages of the first page of data based on the total number of rows and the preset number of rows to be displayed on a single page.

[0012] According to the sorting rule specified by the user, the data rows between the starting row number and the ending row number are intercepted and inserted into the global temporary table.

[0013] The task queried this time is recorded in the memory cache of the server, and the recorded information includes the main task identifier, auxiliary task identifier, auxiliary task completion identifier and other related information.

[0014] Ensure that the memory cache does not store actual query data to prevent memory overflow, and set the destruction time of the memory cache.

[0015] According to an embodiment of the present application, when the main task is completed, the asynchronous task is started, specifically: after the main task is completed, the backend returns the main task identifier, the total data volume and the first page data for display;

[0016] Start a concurrent thread to execute the auxiliary task, and set the auxiliary task completion flag in the memory cache to being obtained asynchronously;

[0017] The auxiliary task performs complete data mining, and organizes and processes the mined data and stores it in a database real table;

[0018] When the auxiliary task is completed, the auxiliary task completion flag in the memory cache is updated to completed.

[0019] According to an embodiment of the present application, when a request to switch the page number is initiated, the memory cache is checked. If the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; otherwise, the data is mined again and asynchronous task processing is started according to the mining status, specifically:

[0020] The front end initiates a request to switch the page number, and the back end checks whether the memory cache of the server exists according to the main task identifier. If the memory cache does not exist or the auxiliary task completion identifier is 0 or 1, the data of the specified page is re-mined. If the auxiliary task completion identifier is 2, the data of the specified page is filtered out from the database cache. If the auxiliary task completion identifier is 0, the auxiliary task is started to asynchronously organize data.

[0021] According to an embodiment of the present application, when an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again according to the cache identifier. Once the data is ready, it can be exported from the database, specifically:

[0022] The front end sends an export request to the back end, carrying the main task identifier, checks the memory cache, and determines whether it is necessary to re-mine the complete data according to the state of the memory cache, or directly obtain the complete data from the database cache and export it.

[0023] According to an embodiment of the present application, the method further includes: refreshing data, specifically:

[0024] The front end generates a new main task ID, the back end generates a new auxiliary task ID, and starts to re-execute the query task.

[0025] According to an embodiment of the present application, the method further includes: releasing resources, specifically:

[0026] The memory cache is created with a destruction time set, and the memory is automatically released when the time is reached. Expired task data caches are checked and cleared through asynchronous threads, and a scheduled task is set to completely clear the database cache at 2 a.m. every day.

[0027] A computer program product comprising instructions, when executed on a device, enables the device to execute the steps in a method for implementing intelligent asynchronous pre-processing query data.

[0028] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in an intelligent asynchronous preprocessing query data method.

[0029] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the intelligent asynchronous pre-processing query data method are implemented.

[0030] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0031] 1. Improve data query efficiency: In large-scale data processing and query scenarios, traditional synchronous query methods may cause system resources to be occupied for a long time, thereby affecting query efficiency. Intelligent asynchronous pre-processing query data methods can use asynchronous processing mechanisms to enable data query requests to be processed in parallel, thereby improving overall query efficiency.

[0032] 2. Optimize the data processing process: Raw data often has problems such as clutter, duplication and incompleteness, which will directly affect the accuracy and efficiency of query results. The intelligent asynchronous pre-processing query data method includes pre-processing steps such as intelligent cleaning, deduplication and completion of raw data, thereby optimizing the data processing process and providing a clean, accurate and concise data foundation for subsequent query operations.

[0033] 3. Reduce system resource consumption: The asynchronous processing mechanism allows the system to continue to receive and process other requests while processing a query request, thus avoiding the idle waiting of system resources in the traditional synchronous query method. In addition, the intelligent pre-processing strategy can also dynamically adjust the allocation of processing resources according to the characteristics of the data and the needs of the query, further reducing the consumption of system resources.

[0034] 4. Improve user experience: Efficient query efficiency and accurate data processing results can significantly improve user experience. Users can obtain the required query results faster, and the results are more accurate and reliable, thereby increasing user satisfaction and trust in the system.

[0035] 5. Support complex query requirements: In actual applications, users may need to perform complex query operations involving multiple data sources and complex logical judgments. The intelligent asynchronous pre-processing query data method provides powerful query capabilities, supports complex query statements and logical judgments, and thus meets the diverse query needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 A partial flow chart of an intelligent asynchronous pre-processing query data method provided in an embodiment of the present application;

[0038] Figure 2 A partial flow chart of an intelligent asynchronous pre-processing query data method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the overall concept of the present application, a detailed description is given below in an illustrative manner in conjunction with the accompanying drawings.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the present application and the features in each embodiment may be combined with each other without conflict.

[0041] In the present application, unless otherwise clearly specified and limited, a first feature "above" or "below" a second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0042] Example 1

[0043] like Figure 1 and Figure 2 As shown, ( Figure 1 and Figure 2 Combined into a complete method flow chart), an intelligent asynchronous pre-processing query data method, including:

[0044] When a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined based on the total number of rows and the pre-set number of rows to be displayed on a single page, the first page of data is obtained and organized, the task information is recorded in the memory cache, and the memory cache is managed. When the main task is completed, the asynchronous task is started.

[0045] Specifically, a query request is initiated: the front-end user sends a query request, and after receiving the request, the back-end generates a globally unique task identifier (GUID) as the main task identifier (task1).

[0046] Mining total number of rows: The backend calculates the total number of rows (totalCount) that meet the query conditions based on the query conditions.

[0047] Determine the paging parameters: According to the total number of rows (totalCount) and the preset number of rows displayed on a single page (perPageCount), calculate the starting row number (startIndex) of the first page of data as 1, and the ending row number (endIndex) as the number of rows displayed on a single page. If the total number of rows can divide the number of rows displayed on a single page, the total number of pages (pageCount) is the total number of rows divided by the number of rows displayed on a single page; if it cannot be divided evenly, the total number of pages is the total number of rows divided by the number of rows displayed on a single page, rounded up and added to 1.

[0048] Get and organize the first page of data: According to the sorting rules specified by the user, intercept the data rows between the starting row number and the ending row number, that is, the first page of data, and insert this data into the global temporary table for preliminary organization and processing, such as converting the foreign key field of the master data into information such as number and name.

[0049] Record task information to memory cache: Record the task information of this query to the memory cache of the server, including the main task ID (task1), auxiliary task ID (task2), auxiliary task completion ID (initial value is set to 0 to indicate unexecuted), and other related information. The memory cache does not store actual query data to avoid memory overflow, and sets a destruction time to automatically release memory space.

[0050] Manage memory cache: If the system supports load balancing, the memory cache should support distributed services to ensure consistency in a multi-server environment.

[0051] Completing the main task: After completing the above operations, the main task ends, and the backend sends the main task identifier (task1), the total amount of data, and the first page of data back to the frontend, which displays the first page of data accordingly.

[0052] Start asynchronous task: After the main task is completed, the backend starts a concurrent thread to execute the auxiliary task (task2). The auxiliary task is responsible for mining the complete data and storing the mined data in the database real table after organization and processing. Before starting the auxiliary task, set the auxiliary task completion flag in the memory cache to 1 (indicating asynchronous acquisition) to avoid repeated mining. When the auxiliary task is completed, update the auxiliary task completion flag in the memory cache to 2 (indicating completion).

[0053] For example, to issue a query request:

[0054] The user submits a query request through the front-end interface, for example, he wishes to query the detailed information of a financial statement.

[0055] After receiving the request, the backend generates a globally unique task identifier (for example, a 36-bit string generated by a GUID) as the main task identifier (task1).

[0056] Mining total rows:

[0057] The backend calculates the total number of rows (totalCount) of data that meet the query conditions based on the user's query conditions. For example, if the user wants to query all transaction records in 2023, the system calculates that there are 1 million records in total.

[0058] Determine the paging parameters:

[0059] Based on the calculated total number of rows (1 million) and the preset number of rows to be displayed on a single page (for example, 50 records per page), the starting row number (startIndex) of the first page of data is calculated to be 1, and the ending row number (endIndex) is calculated to be 50.

[0060] Calculate the total number of pages (pageCount). Since the total number of rows may not be divisible by the number of rows displayed on a single page, the total number of pages should be the ceiling value of the total number of rows divided by the number of rows on a single page (that is, the smallest integer greater than or equal to the value). The formula is: pageCount = ceil (1 million / 50) = 20000.

[0061] Get and organize the first page of data:

[0062] According to the user's sorting rules (for example, sorting by transaction date), intercept the data rows from row 1 to row 50 and insert the data into the global temporary table.

[0063] Organize and process these data, such as converting the transaction type code into the name of the transaction type.

[0064] Record task information to memory cache:

[0065] The relevant information of this query is recorded in the memory cache of the server, including: main task identifier (task1), auxiliary task identifier (task2), auxiliary task completion identifier (the initial value is set to 0 to indicate that it has not been executed), and other relevant information (such as query filter items, total data volume, etc.).

[0066] Ensure that actual query data is not stored in the memory cache to prevent memory overflow, and set the destruction time of the memory cache.

[0067] Managing memory cache:

[0068] If the system supports load balancing, the memory cache should support distributed services to ensure data consistency across multiple servers.

[0069] Complete the main task:

[0070] After completing the above operations, the main task ends, and the backend sends the main task identifier (task1), the total data volume (1 million), and the first page of data back to the frontend, which then displays the first page of data.

[0071] Start asynchronous tasks:

[0072] After the main task is completed, a concurrent thread is started to execute the auxiliary task (task2), which is responsible for mining the complete data.

[0073] Before starting the auxiliary task, set the auxiliary task completion flag in the memory cache to 1 (indicating that it is being acquired asynchronously) to prevent repeated mining.

[0074] The auxiliary task performs complete data mining and organizes and processes the mined data and stores it in the database real table.

[0075] When the auxiliary task is completed, the auxiliary task completion flag in the memory cache is updated to 2 (indicating completion).

[0076] Furthermore, intelligent prediction and preloading can be performed: machine learning algorithms can be used to analyze user behavior habits, predict the user's next possible operation (such as turning pages, exporting, etc.), and load relevant data into the cache in advance to further improve the user experience. For example, if a user frequently views the first page and then the second page, the system can predict the user's next action and load the data for the second page in advance.

[0077] Furthermore, it can be customized: users are allowed to customize query parameters and display preferences, such as sorting rules, the number of rows displayed per page, etc., and these preferences are stored in the user configuration to enhance the personalized experience. Users can choose different view modes, and the system organizes the presentation of data according to the user's preferences.

[0078] Furthermore, real-time data analysis and feedback can be performed: query requests can be monitored in real time, and indicators such as user query response time and error rate can be collected to optimize system performance. Opinions on query experience can be collected through user feedback mechanisms, and algorithms can be adjusted or functions can be added in a timely manner.

[0079] Furthermore, data security and permission management can be strengthened: data security measures can be strengthened to ensure encryption of sensitive data during transmission. Fine-grained permission management can be implemented to ensure that users can only access the data they are authorized to protect data privacy.

[0080] When a request to switch pages is initiated, the memory cache is checked. If the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; otherwise, the data is mined again and asynchronous task processing is started according to the mining status.

[0081] Specifically, a request is received: the front end sends a request for switching pages to the back end, and the request includes a page identifier that the user wants to view (for example, the user wants to jump from the first page to the second page).

[0082] Check the memory cache: After receiving the request, the backend first checks whether the memory cache on the server exists based on the main task identifier (task1) transmitted.

[0083] Determine the cache status:

[0084] If the memory cache exists and the [Auxiliary Task Completion Flag] is 2 (indicating completion), this means that the background has completed the mining of the complete data and cached the data in the database.

[0085] In this case, the system directly obtains the data of the corresponding page from the database and returns it to the front-end for display. For example, if the user requests the second page of data, the system will filter the corresponding data from the database cache based on the starting row number (startIndex) and the ending row number (endIndex).

[0086] Processing unfinished state:

[0087] If the memory cache exists, but the [Auxiliary Task Completion Flag] is 0 (not executed) or 1 (in progress), this indicates that the background has not yet completed the mining of the complete data, or mining is in progress.

[0088] At this point, the system needs to mine data again. Specifically, the system will start a new main task (task1) to mine the data of the specified page and display it to the user.

[0089] If the mining status is 0, the system will also start an asynchronous task (auxiliary task task2) to continue mining the complete data and prepare for subsequent requests.

[0090] Enable asynchronous task processing:

[0091] If the auxiliary task completion flag is found to be 0 (indicating that it has not been executed), the system will intelligently start the auxiliary task to asynchronously organize data, that is, re-mine the data and start the auxiliary task.

[0092] The auxiliary task will organize and process the mined data and store it in the database real table so that the data can be directly used in future requests.

[0093] For example, suppose a user is using a financial statement query system that allows users to browse data in pages. The user initially requests the first page of data, the system successfully displays the first page, and starts an asynchronous task to mine the complete data in the background.

[0094] The user first requests the first page of data

[0095] User initiated request: The user requests to view the first page of financial statement data.

[0096] System Response:

[0097] The backend calculates the total number of rows (e.g. 100000 rows).

[0098] According to the fact that 50 lines are displayed per page (perPageCount=50), the starting line number (startIndex=1) and the ending line number (endIndex=50) of the first page are determined.

[0099] Get and organize the first page of data and display it to the user.

[0100] At the same time, start the asynchronous task to mine the complete data, and set the auxiliary task flag in the memory cache to 1 (indicating that it is being acquired asynchronously).

[0101] The user switches to the second page

[0102] User initiates request: The user clicks the navigation button and requests to view the second page of data.

[0103] System Response:

[0104] The backend checks the memory cache and finds the record corresponding to the main task identifier (task1).

[0105] The [Auxiliary Task Completion Flag] in the memory cache is 1 (indicating that it is being acquired asynchronously), which means that the complete data mining has not yet been completed.

[0106] The system re-mines the data on the second page.

[0107] Calculate the starting row number (startIndex=51) and ending row number (endIndex=100) of the second page.

[0108] Get and organize the second page of data and display it to the user.

[0109] The system checks the [auxiliary task completion flag], if it is 0, the auxiliary task asynchronously organizes data.

[0110] The secondary task is completed and the user switches to the second page again

[0111] Auxiliary task completion: The asynchronous task mining is completed, and the [Auxiliary task completion flag] in the memory cache is updated to 2 (indicating completion).

[0112] User requests again: The user clicks the navigation button again to request to view the second page of data.

[0113] System Response:

[0114] The backend checks the memory cache and finds the record corresponding to the main task identifier (task1).

[0115] The [Auxiliary Task Completion Flag] in the memory cache is 2 (indicating completion), which means that the complete data has been cached in the database.

[0116] Get the data for the second page directly from the database.

[0117] Use the starting row number (startIndex=51) and ending row number (endIndex=100) of the second page as filtering conditions to filter out the data of the second page from the database cache.

[0118] Organize and present the second page of data to the user.

[0119] From this example, we can see that when the user switches pages, the system first checks the status of the memory cache. If the cache exists and is complete, the system directly obtains the data of the corresponding page from the database; if the cache is not complete, the data is mined again and asynchronous task processing is started according to the mining status. This ensures the timeliness of the data and avoids repeated mining, improving the system's response speed and user experience.

[0120] Furthermore, intelligent prediction and preloading can be performed: by analyzing user behavior patterns, possible user operations (such as page turning, exporting, etc.) can be predicted, and relevant data can be loaded into the cache in advance to further speed up the response. For example, if the system observes that users often flip through multiple pages in succession, the data of the next few pages can be preloaded when the user requests the current page.

[0121] Furthermore, data preheating can be performed: for popular or frequently accessed data, it can be preloaded into the cache to improve response speed even without user requests. Use machine learning algorithms to analyze which data is most frequently accessed and keep this data in the memory cache.

[0122] Furthermore, it is possible to update incrementally: if data changes, only the affected part of the data is updated instead of re-mining the entire data, saving time and resources. For example, if only a small part of the data has changed, the system only needs to update this part of the data and mark it as the latest in the cache.

[0123] Furthermore, the cache strategy can be adjusted dynamically: according to the system load and user request frequency, the cache strategy can be adjusted dynamically, such as increasing or decreasing the cache validity period. When the system load is low, the cache validity period can be appropriately extended to reduce the frequency of data mining; conversely, when the load is high, the cache validity period can be shortened to ensure the timeliness of the data.

[0124] Furthermore, priority scheduling can be performed: the priority of task execution can be adjusted according to the urgency or importance of the user's request. For VIP users or critical tasks, higher priority can be given to ensure that these requests can be responded to faster.

[0125] Furthermore, a fault recovery mechanism can be designed to ensure that services can be quickly restored when a system failure occurs, such as automatically restarting failed tasks. When a task is found to have failed to be completed on time, it is automatically retried or assigned to other available resources for execution.

[0126] Furthermore, user behavior analysis can be performed: analyzing user query patterns, identifying high-frequency queries, and caching these query results for a long time to reduce the query pressure on the database. Using big data analysis tools, user behavior reports can be provided to help system administrators better understand user needs.

[0127] When an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again based on the cache identifier. Once the data is ready, it can be exported from the database.

[0128] Specifically, initiate an export request:

[0129] The user triggers an export request through the front-end interface, for example, he wants to export the current query results as an Excel or CSV file.

[0130] The request contains the main task identifier (task1), which is used to identify the current query task.

[0131] Check the memory cache status:

[0132] After receiving the export request, the backend first checks whether there are records related to the main task ID (task1) in the memory cache.

[0133] Check the auxiliary task completion flag in the memory cache, which is used to indicate the status of the auxiliary task (complete data mining).

[0134] Determine whether data mining needs to be re-done:

[0135] If the memory cache does not exist, it means that the relevant data has not been queried before or the cache has expired, and the data needs to be mined again.

[0136] If the memory cache exists but the [Auxiliary Task Completion Flag] is 0 (indicating not executed) or 1 (indicating being executed), it means that the complete data is not yet ready and needs to be mined again.

[0137] If the memory cache exists and the [Auxiliary Task Completion Flag] is 2 (indicating completion), it means that the complete data is ready and can be directly obtained from the database.

[0138] Re-mining the data:

[0139] If the data needs to be mined again, the system will start a new main task (task1) to mine the complete data.

[0140] At the same time, if the [auxiliary task completion flag] is 0, the system will start the auxiliary task (task2) to asynchronously process data mining, organize and process the mined data, and store it in the database real table.

[0141] Export data from the database:

[0142] Once the data is ready, that is, the [Auxiliary Task Completion Flag] is 2, the system can export the data from the database.

[0143] Export the complete data to Excel, CSV or other formats according to user needs.

[0144] After the file is generated, a download link can be provided to the user, or the file can be sent directly to the user via email or other means.

[0145] For example, suppose a user is using a financial report query system that allows the user to browse data in pages and supports exporting data in Excel or CSV format. After querying some data, the user wants to export the results of the current query.

[0146] User initiated export request

[0147] User behavior: The user clicks the "Export" button on the interface to initiate an export request.

[0148] Request content: The request contains the main task ID (task1) and other necessary export parameters, such as the file format (for example, Excel or CSV).

[0149] Backend checks memory cache status

[0150] Backend response: After receiving the export request, the backend first checks whether there are records related to the main task ID (task1) in the memory cache.

[0151] Memory cache exists:

[0152] Check the [Sub-task Completion Flag]:

[0153] If the [Auxiliary Task Completion Flag] is 2 (indicating completion), it means that the complete data is ready.

[0154] If the [Auxiliary Task Completion Flag] is 0 (indicating not executed) or 1 (indicating in progress), it means that the complete data is not yet ready.

[0155] Memory cache does not exist:

[0156] This indicates that the relevant data has not been queried before or the cache has expired, and the data needs to be mined again.

[0157] Handling different states in memory cache

[0158] Case 1: Memory cache exists and [Auxiliary task completion flag] is 2

[0159] Memory cache status: The memory cache exists and the [Auxiliary Task Completion Flag] is 2, indicating that the complete data is ready.

[0160] Processing steps:

[0161] Get data from the database: directly use the auxiliary task identifier (task2) as the filtering condition to filter out complete data from the database cache.

[0162] Export data: After the filtered data is organized and processed, it is exported to the file format selected by the user (such as Excel or CSV).

[0163] Provide download link or file: After the file is generated, the system provides a download link or sends it directly to the user via email.

[0164] Case 2: The memory cache exists but the [Auxiliary Task Completion Flag] is 0 or 1

[0165] Memory cache status: The memory cache exists but the [Auxiliary Task Completion Flag] is 0 or 1, indicating that the complete data is not yet ready.

[0166] Processing steps:

[0167] Re-mine data: Restart the main task (task1) and mine the complete data.

[0168] Start auxiliary task: If the auxiliary task completion flag is 0, start the auxiliary task (task2), process data mining asynchronously, and organize and process the mined data and store it in the database real table.

[0169] Waiting for data preparation to complete: After waiting for the auxiliary task to complete, update the [Auxiliary Task Completion Flag] in the memory cache to 2.

[0170] Get data from the database: Once the data is ready, use the auxiliary task identifier (task2) as a filter condition to filter out the complete data from the database cache.

[0171] Export data: After the filtered data is organized and processed, it is exported to the file format selected by the user (such as Excel or CSV).

[0172] Provide download link or file: After the file is generated, the system provides a download link or sends it directly to the user via email.

[0173] Handling the situation where the memory cache does not exist

[0174] Memory cache status: The memory cache does not exist, which means that the relevant data has not been queried before or the cache has expired.

[0175] Processing steps:

[0176] Re-mine data: Start the main task (task1) and mine the complete data.

[0177] Start auxiliary task: Start the auxiliary task (task2), process data mining asynchronously, organize and process the mined data, and store it in the database real table.

[0178] Waiting for data preparation to complete: After waiting for the auxiliary task to complete, update the [Auxiliary Task Completion Flag] in the memory cache to 2.

[0179] Get data from the database: Once the data is ready, use the auxiliary task identifier (task2) as a filter condition to filter out the complete data from the database cache.

[0180] Export data: After the filtered data is organized and processed, it is exported to the file format selected by the user (such as Excel or CSV).

[0181] Provide download link or file: After the file is generated, the system provides a download link or sends it directly to the user via email.

[0182] Through this process, the system can intelligently determine whether it is necessary to mine the data again and efficiently complete the export operation after the data is ready. This not only improves the user experience, but also optimizes the resource utilization of the system.

[0183] Furthermore, it can also support multi-tenancy: in a multi-user environment, it supports multi-tenant isolation to ensure that the data of different users do not interfere with each other. It provides independent cache space for different users to prevent data confusion or conflict.

[0184] Furthermore, security can be enhanced: Strengthen security control of data access to ensure that only authorized users can access specific data. Implement data encryption and access log auditing to improve system security.

[0185] Furthermore, you can also provide export progress feedback: implement a real-time feedback mechanism for export progress to let users know the current status of the export operation and improve transparency. For example, provide a progress bar or percentage to show the completion of the export.

[0186] Furthermore, an asynchronous notification mechanism can be used: after the export is completed, the user is informed of the completion of the export through an asynchronous notification mechanism, such as sending an email, SMS or push notification. The user can receive a notification immediately after the export is completed, without having to repeatedly check the export status.

[0187] The intelligent asynchronous preprocessing query data method provided by the present application also includes the following additional technical features: when a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows displayed on a single page, the first page of data is obtained and organized, the task information is recorded in the memory cache, and the memory cache is managed, specifically:

[0188] The user initiates a query request, and the backend generates a globally unique identifier as the main task identifier based on the request. The backend calculates the total number of rows that meet the query conditions, and calculates the starting row number, ending row number, and total number of pages of the first page of data based on the total number of rows and the preset number of rows to be displayed on a single page.

[0189] According to the sorting rule specified by the user, the data rows between the starting row number and the ending row number are intercepted and inserted into the global temporary table.

[0190] The task queried this time is recorded in the server's memory cache. The recorded information includes the main task ID, auxiliary task ID, auxiliary task completion ID, and other related information.

[0191] Make sure that the memory cache does not store actual query data to prevent memory overflow, and set the destruction time of the memory cache.

[0192] Specifically, the user initiates a query request:

[0193] The user submits a query request through the front-end interface. This request usually includes specific query conditions, such as filtering conditions, sorting methods, etc.

[0194] Generate a unique identifier:

[0195] After receiving the query request, the backend generates a globally unique identifier (GUID), which is used as the identifier of the main task (task1) to track and manage the query task.

[0196] Total number of rows of mining data:

[0197] The backend calculates the total number of rows (totalCount) that meet the query conditions. This is to prepare for subsequent paging and ensure that the total number of pages and the data range of each page can be calculated correctly.

[0198] Determine the paging parameters:

[0199] According to the total number of rows (totalCount) and the preset number of rows displayed on a single page (perPageCount), the starting row number (startIndex) of the first page of data is calculated to be 1, and the ending row number (endIndex) is the number of rows displayed on a single page.

[0200] If the total number of rows can be divided evenly by the number of rows displayed on a single page, then the total number of pages (pageCount) is the total number of rows divided by the number of rows displayed on a single page; if it cannot be divided evenly, then the total number of pages is the total number of rows divided by the number of rows displayed on a single page, removing the remainder and adding 1.

[0201] Get and organize the first page of data:

[0202] According to the sorting rules configured by the user, extract the data rows between the starting row number (startIndex) and the ending row number (endIndex).

[0203] Inserting these data rows into the global temporary table takes less time because the number of data rows inserted is limited.

[0204] Organize and process the data in the global temporary table, for example, convert the foreign key fields of the master data into corresponding numbers and names.

[0205] Record task information to memory cache:

[0206] The relevant information of this query is recorded in the memory cache of the server, including the main task identifier (task1), the auxiliary task identifier (task2), the auxiliary task completion identifier (0 means not executed, 1 means being obtained asynchronously, and 2 means completed) and other relevant information, such as the filter items of the query data, the total data volume, etc.

[0207] It should be noted that the actual query data should not be stored in the memory cache to prevent memory overflow. The memory cache is mainly used to manage the status information of the task.

[0208] Memory cache management:

[0209] When creating a memory cache, set a destruction time. When this time is reached, the memory space is automatically released to prevent the memory from being consumed indefinitely.

[0210] The purpose of this is to avoid memory leaks and ensure that the system can run stably in a high-concurrency environment.

[0211] According to an embodiment of the present application, when the main task is completed, the asynchronous task is started, specifically: after the main task is completed, the backend returns the main task identifier, the total data volume and the first page of data for display;

[0212] Start concurrent threads to execute auxiliary tasks, and set the auxiliary task completion flag in the memory cache to being obtained asynchronously;

[0213] The auxiliary task performs complete data mining and organizes and processes the mined data and stores it in the database real table;

[0214] When the auxiliary task is completed, the auxiliary task completion flag in the memory cache is updated to completed.

[0215] Specifically, when the main task is completed, the asynchronous task is started, specifically: after the main task is completed, the backend returns the main task identifier, the total data volume and the first page of data for display;

[0216] Start concurrent threads to execute auxiliary tasks, and set the auxiliary task completion flag in the memory cache to being obtained asynchronously;

[0217] The auxiliary task performs complete data mining and organizes and processes the mined data and stores it in the database real table;

[0218] When the auxiliary task is completed, the auxiliary task completion flag in the memory cache is updated to completed.

[0219] According to an embodiment of the present application, when a request to switch the page number is initiated, the memory cache is checked. If the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; otherwise, the data is mined again and asynchronous task processing is started according to the mining status, specifically:

[0220] The front-end user initiates a request to switch the page number. The back-end checks whether the server memory cache exists based on the main task identifier. If the memory cache does not exist or the auxiliary task completion identifier is 0 or 1, the data of the specified page is mined again. If the auxiliary task completion identifier is 2, the data of the specified page is filtered out from the database cache. If the auxiliary task completion identifier is 0, the auxiliary task is started to asynchronously organize data.

[0221] Specifically, the user initiates a request to switch pages:

[0222] The user triggers the page switching operation through the front-end interface (such as a web page or application), for example, switching from page 1 to page 2 or returning from the last page to page 1.

[0223] The request contains the main task identifier (task1) and other possible parameters, such as the target page number (pageIndex).

[0224] Backend checks memory cache:

[0225] After the backend receives the request to switch pages, it first checks whether the server memory cache exists based on the main task identifier (task1) received.

[0226] If the memory cache exists, check the status of the [auxiliary task completion flag] in the cache.

[0227] If the memory cache does not exist, it means that the query task has never been executed before, or the previous cache has expired and the data needs to be mined again.

[0228] Process data based on cache status:

[0229] The memory cache exists and the [Auxiliary Task Completion Flag] is 2:

[0230] Indicates that full data mining has been completed and the data of the specified page can be obtained directly from the database cache.

[0231] Calculate the starting row number (startIndex) and ending row number (endIndex) of the specified page.

[0232] Filter the data of the corresponding page from the database and return it to the front end.

[0233] The memory cache exists and the [Auxiliary Task Completion Flag] is 0 or 1:

[0234] Indicates that full data mining has not been completed or is in progress.

[0235] The data of the specified page needs to be mined again.

[0236] If the auxiliary task completion flag is 0, you also need to start the auxiliary task (task2) to asynchronously organize data to ensure that the cache status can be updated in time after data mining is completed.

[0237] Memory cache does not exist:

[0238] Re-mine the data of the specified page.

[0239] Execute the complete query process, including data mining, organization and processing, and recording to memory cache.

[0240] Enable auxiliary tasks to organize data asynchronously:

[0241] If the auxiliary task completion flag is 0, it means that asynchronous mining has not started or is in progress. At this time, the auxiliary task (task2) needs to be started to complete the data mining work.

[0242] Set the auxiliary task completion flag in the memory cache to 1 (indicating that it is being retrieved asynchronously) to prevent repeated opening of auxiliary tasks.

[0243] After the mining is completed, the data is stored in the database and the [auxiliary task completion flag] in the memory cache is updated to 2 (indicating completion).

[0244] According to one embodiment of the present application, when an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again based on the cache identifier. Once the data is ready, it can be exported from the database, specifically:

[0245] The front end sends an export request to the back end, carrying the main task identifier, checks the memory cache, and decides whether to re-mine the complete data based on the status of the memory cache, or directly obtain the complete data from the database cache and export it.

[0246] Specifically, the user initiates an export request:

[0247] The user triggers the export operation through the front-end interface (such as a web page or application) and requests to export the current query results to a file (such as Excel or CSV format).

[0248] The request contains the main task ID (task1), which is an identifier used to uniquely identify the current query task.

[0249] The backend receives the request and checks the memory cache:

[0250] After receiving the export request, the backend first checks whether the server memory cache exists based on the main task identifier (task1) transmitted.

[0251] If the memory cache exists, the status of the [auxiliary task completion flag] in the cache is further checked.

[0252] If the memory cache does not exist, it means that the query task has never been executed before, or the previous cache has expired and the complete data needs to be mined again.

[0253] Determine the operation based on the memory cache status:

[0254] The memory cache exists and the [Auxiliary Task Completion Flag] is 2:

[0255] This means that full data mining has been completed and the data has been organized and processed and stored in the database cache.

[0256] The backend directly filters out complete data from the database cache and organizes and processes it, such as converting data formats and adding necessary headers.

[0257] Finally, the sorted data is exported to the file format requested by the user (such as Excel or CSV) and provided to the user for download through the front end or delivered to the user through other means.

[0258] The memory cache exists but the [Auxiliary Task Completion Flag] is 0 or 1:

[0259] This indicates that full data mining has not yet been completed or is in progress.

[0260] The complete data needs to be mined again.

[0261] If the auxiliary task completion flag is 0, it means that the auxiliary task has not been started yet, and the auxiliary task (task2) needs to be started to organize data asynchronously.

[0262] Memory cache does not exist:

[0263] This indicates that the relevant data has not been queried before or the cache has expired, and the complete data needs to be mined again.

[0264] Execute the complete query process, including data mining, organization and processing, and recording to memory cache.

[0265] Re-mine the complete data:

[0266] If the data needs to be mined again, the system will start the main task (task1) to mine the complete data.

[0267] Start the auxiliary task (task2) to asynchronously process data mining, organize and process the mined data and store it in the database real table.

[0268] Once the data is ready, that is, after the auxiliary task is completed, the [auxiliary task completion flag] in the memory cache is updated to 2 (indicating completion).

[0269] Get data from the database and export it:

[0270] Once the data is ready, use the secondary task ID (task2) as a filter condition to filter out the complete data from the database cache.

[0271] After the filtered data is organized and processed, it is exported to the file format selected by the user (such as Excel or CSV).

[0272] After the file is generated, the system provides a download link or sends it directly to the user via email.

[0273] According to an embodiment of the present application, the method further includes: refreshing data, specifically:

[0274] The front end generates a new main task ID, the back end generates a new auxiliary task ID, and starts to re-execute the query task.

[0275] Specifically, the front end generates a new main task identifier:

[0276] When the user decides to refresh the data, the front end will generate a new main task identifier (newTask1), which is used to identify the new query task to ensure that the new and old tasks are not confused.

[0277] Generating a new master task ID is necessary because each query task needs to have a unique identifier so that the backend can distinguish between different query tasks.

[0278] The backend generates a new auxiliary task ID:

[0279] After the backend receives the new main task identifier passed in by the frontend, it will also generate a new auxiliary task identifier (newTask2).

[0280] The new auxiliary task identifier is used to identify the new auxiliary task to ensure that the new data mining task will not be confused with the old task.

[0281] Generating new auxiliary task identifiers helps ensure the independence and accuracy of data mining tasks.

[0282] Start re-executing the query task:

[0283] After the front-end and back-end cooperate to generate new main task and auxiliary task identifiers, the system will start to re-execute the query task.

[0284] At this point, the backend will execute the complete data mining process based on the new task identifier, including but not limited to:

[0285] Calculate the total number of rows that meet the criteria.

[0286] Determine the paging parameters based on the total number of rows and the preset number of rows to be displayed on a single page.

[0287] Get and organize the first page of data.

[0288] Record new task information into the memory cache, including the new main task ID, auxiliary task ID, and auxiliary task completion ID.

[0289] At the same time, the backend will also start concurrent threads to execute new auxiliary tasks, mine complete data and store it in the database.

[0290] After the auxiliary task is executed, the auxiliary task completion flag in the memory cache is updated to completed (2).

[0291] According to an embodiment of the present application, the method further includes: releasing resources, specifically:

[0292] When the memory cache is created, a destruction time is set. The memory is automatically released when the time is reached. Expired task data cache is checked and cleared through asynchronous threads, and a scheduled task is set to completely clear the database cache at 2 a.m. every day.

[0293] Specifically, the life cycle management of memory cache:

[0294] When a memory cache is created, a scheduled destruction time is set for it. This means that the cache does not occupy memory space indefinitely.

[0295] Once the cache life cycle ends, that is, the scheduled destruction time is reached, the memory cache will be automatically released, thereby freeing up memory space.

[0296] Asynchronously check for expired cache:

[0297] To prevent cached data from occupying too much memory resources, the system will insert a record into the database task table while saving the memory cache. The record contains the main task ID (task1), the auxiliary task ID (task2), and the start time (startTime).

[0298] Each time the backend performs a query operation, an asynchronous thread is started to check the data records in the task table whose start time exceeds a certain time.

[0299] This asynchronous checking process ensures that even when processing a large number of requests, the performance of the current query is not affected by the data cleansing operation.

[0300] Clear expired cache:

[0301] When the asynchronous thread detects a data record whose start time exceeds a certain time, the system will clear the corresponding cache data in the database according to the main task identifier (task1) and the auxiliary task identifier (task2).

[0302] This method can avoid the current query performance being affected by excessive data volume.

[0303] Scheduled task to clean up the database cache:

[0304] In addition to asynchronously checking and clearing expired cache, the system also sets a scheduled task, usually at 2 a.m. every day, to perform a comprehensive clearing of the database cache.

[0305] Such scheduled tasks can prevent cache data from accumulating for a long time without being cleaned up due to power outages or other unexpected situations.

[0306] By periodically cleaning up, you can ensure that the database cache always remains at a reasonable size and avoid unnecessary waste of resources.

[0307] Example 2

[0308] An intelligent method for asynchronous pre-processing of query data, the main content is: under large-volume query, first meet the user's demand for fast display, according to the sorting configuration, only mine the data of the first page, organize and process the first page data, and transmit it to the front end to achieve the purpose of fast display; and while transmitting a single page of data, asynchronously generate multi-threaded operations in the background, intelligently mine the complete data required by the user in advance, organize and process the data, cache it to the database, and wait for the user to call. The time for mining complete data is the most time-consuming link. Therefore, before the data is pre-processed, the user may start the next task, such as clicking to display page x or exporting to a document. At this time, the main thread cannot wait for the asynchronous thread and should execute a new query task. The process logic is as follows Figure 1 and Figure 2 shown.

[0309] The processing steps are as follows:

[0310] 1. Start the query task

[0311] The front end starts the task, transfers the main task (task1) to the back end, and then starts a new thread to concurrently execute the auxiliary task (task2). The identifiers of these two tasks are both 36-bit globally unique strings generated by GUID.

[0312] The main task belongs to the first thread, which is responsible for realizing the user's demand for fast display of the first page. First, the total number of rows (totalCount) of data that meet the conditions is mined under the big data. According to the total number of rows and the configured number of rows displayed on a single page (perPageCount), the starting row number (startIndex) of the first page is 1, and the ending row number (endIndex) is the number of rows displayed on a single page. If the total number of rows (totalCount) can divide the number of rows displayed on a single page (perPageCount), then the total number of pages (pageCount) = the total number of rows (totalCount) / the number of rows displayed on a single page (perPageCount). If it cannot be divided evenly, the total number of pages (pageCount) = round up (totalCount / perPageCount) + 1.

[0313] When obtaining data, it is sorted according to the user's configuration method, and the data rows within the start and end row numbers (inclusive), that is, the first page of data, are intercepted. The intercepted data is inserted into the global temporary table. Because the inserted data is limited, it takes less time.

[0314] Then organize and process the data in the global temporary table, for example, obtain the number and name of the foreign key field of the master data according to the Guid code dictionary ID.

[0315] Finally, record this task in the server memory cache. If it is a multi-server that supports load balancing, the cache should support distributed services. The cache should record the main task identifier (task1), auxiliary task identifier (task2), auxiliary task completion identifier (or asynchronous acquisition of complete data completion status, 0 means not executed, 1 means asynchronous acquisition, 2 means completed), query data filter items, total data volume, etc. It should be noted that the mined query data cannot be stored in the memory cache, even if only a single page is stored, because the system cannot control how many users are operating concurrently at the same time, and the data stored in the memory cache is easily accumulated and causes memory overflow (OOM). The memory cache needs to set a destruction time when it is created. If this time limit is exceeded, the memory is released to prevent unlimited memory consumption.

[0316] 2. Start asynchronous tasks

[0317] After the main task is completed, the concurrent thread is started. The original thread transmits the main task identifier (task1), the total data volume, and the first page of data to the front end, and displays it to the user after rendering and layout. The second thread performs the auxiliary task.

[0318] Before starting the auxiliary task, set the [auxiliary task completion flag] in the memory cache to 1 (indicating that it is being obtained asynchronously) to prevent multiple repeated tasks, and then perform complete mining. At this time, the mined data cannot be stored in the global temporary table. Global temporary tables are divided into session-level global temporary tables and transaction-level global temporary tables. Big data queries do not involve the addition, deletion, and modification of business data, and transaction locking will affect performance, so the session level is more suitable. The characteristic of session-level global temporary tables is that the data is automatically cleared when the session is disconnected. Obviously, data cannot be stored for a long time, so a database real table should be used as the carrier of cache data.

[0319] The cached data should be marked with task2 to distinguish it from other users' tasks and other tasks of this user. Unlike main tasks, auxiliary tasks do not need to be marked. The session-level global temporary table only has data operated in this session. It can be said that the entire session has its own task mark.

[0320] The cached data should be marked with continuous and uninterrupted row numbers, which can be used for sorting and for obtaining a row or page of data separately. The total row should be the largest number.

[0321] After the cached data is organized and processed, it waits for user calls.

[0322] If the main task is to display complete data, then the main task should be executed as a secondary task, the data should be cached in the database real table, and the [Secondary Task Completion Flag] should be set to 2 (indicating completion) and stored in the memory cache.

[0323] 3. Switch query data page number

[0324] That is, execute the query display of the xth page that is not the current page, for example, switch from page 1 to page 2, switch from the last page to page 1, etc. Assuming that pageIndex is the page number of page x, then pageIndex must be an integer greater than or equal to 1 and less than or equal to the total number of pages (pageCount). The starting row number (startIndex) and ending row number (endIndex) of page x can also be calculated:

[0325] startIndex=(pageIndex-1)*pageSize+1

[0326] endIndex=pageIndex*pageSize

[0327] Assume that the end row number (endIndex) exceeds the total number of rows (totalCount), that is, the number of rows on the page is less than the number of rows displayed on a single page (perPageCount), then this page must be the last page, and the end row number (endIndex) should be equal to the total number of rows (totalCount).

[0328] The page switching task is also initiated by the front end. After the back end receives it, it first checks whether the server memory cache exists based on the main task identifier (task1) transmitted.

[0329] If the memory cache does not exist, it means that it has not been queried before or the life cycle of the memory cache has exceeded. In either case, you should query again, that is, start the operation in step 1.

[0330] The existence of memory cache means that this task has been queried before. If the [Auxiliary task completion flag] in the cache is 0 (not executed) or 1 (in progress), it means that the auxiliary task of mining complete data has not been started or completed in the background, and a new main task task1 needs to be started to mine the data on page x. If the [Auxiliary task completion flag] is 2 (completed), it means that the background has cached the complete query data in the database. You can directly use the auxiliary task task2 flag and the start and end row numbers as filtering conditions to filter the database cache and get the data on page x.

[0331] Whether it is re-mining or filtering directly from the database cache, the [auxiliary task completion flag] is finally identified. If the flag is 0 (indicating that it has not been executed), the auxiliary task should be intelligently enabled to asynchronously organize data, which is the second step.

[0332] 4. Export all data

[0333] The front end passes the main task identifier (task1) to the back end. As with the page switching step, it first checks the server's memory cache.

[0334] If the memory cache does not exist, it means that it has not been queried before or the life cycle of the memory cache has exceeded. You should query again, that is, start the operation in step 1. Unlike switching pages, the main task is to display complete data.

[0335] The existence of memory cache means that this task has been queried before. If the [Auxiliary Task Completion Flag] in the cache is No, it means that the background has not completed the auxiliary task of mining complete data, and you need to restart step 1. If the [Auxiliary Task Completion Flag] is 2 (indicates completion), it means that the background has cached the complete query data in the database. You can directly use the auxiliary task task2 flag as a filtering condition to filter the database cache and obtain complete data. Export the processed full data to a physical file, such as Excel, CSV, etc.

[0336] 5. Refresh data

[0337] Refreshing means discarding the cache and re-obtaining the latest data. The front end regenerates a new main task identifier (newTask1) and passes it to the back end, and the back end also regenerates a new auxiliary task identifier (newTask2) to start step 1.

[0338] It should be noted that the main task ID remains unchanged and the auxiliary task ID cannot be changed; if the main task ID changes, the auxiliary task ID must also change.

[0339] 6. Release resources

[0340] As mentioned earlier, the destruction time is set when the memory cache is created, and the memory space is automatically released when the time comes. The same can be done for database cache. While saving the memory cache, a task data is inserted into the database task table. The data includes the main task (task1), auxiliary task (task2), and start time (startTime). Each time the backend performs a query, a new thread is started to asynchronously check the data in the task table that exceeds a certain time during the start period, and clear the database cache according to the main task (task1) and auxiliary task (task2). Using asynchrony can avoid excessive data volume affecting the performance of this query.

[0341] Finally, configure a scheduled task, for example, at 2 a.m. every day, to completely clear the database cache to prevent the cache from accumulating for a long time due to unexpected conditions such as power outages.

[0342] Multithreaded asynchronous processing:

[0343] Multithreading technology allows the system to execute multiple tasks at the same time, and each task (thread) can run independently without interfering with each other. When processing a large number of requests concurrently or executing complex tasks, multithreading can significantly improve the utilization of system resources and overall performance.

[0344] Asynchronous processing further improves the system's response speed and throughput. By putting time-consuming operations (such as I / O operations, database queries, etc.) into asynchronous tasks, the main thread can continue to process other tasks without waiting for these time-consuming operations to complete. When the asynchronous task is completed, the result is returned to the main thread through callbacks or event notifications, thereby achieving non-blocking concurrent processing.

[0345] Global temporary tables:

[0346] A global temporary table is a special data structure used to share temporary data between multiple sessions or requests. Unlike session-level temporary tables, the data of global temporary tables is visible to all users, but exists only within the session in which they are created or within a specific time frame.

[0347] In the present invention, the global temporary table may be used to store intermediate results, cache high-frequency query results or implement complex data processing logic. By reducing repeated calculations and data access times, the global temporary table helps to improve the data processing efficiency and query performance of the system.

[0348] Distributed Memory Cache:

[0349] Distributed memory cache is a cache solution that stores data in the memory of multiple servers. Compared with traditional local cache, distributed memory cache has higher scalability and fault tolerance.

[0350] In the present invention, the distributed memory cache is used to store hot data, session information or any data that needs to be accessed quickly. By reducing the number of database accesses and reducing network latency, the distributed memory cache can significantly improve the response speed and throughput of the system.

[0351] Database Cache:

[0352] In the present invention, a database real table is used to cache query results of the full amount of data, so that the results can be directly obtained from the cache in subsequent requests.

[0353] Example 3

[0354] A computer program product comprising instructions, when running on a device, enables the device to execute the steps in a method for implementing intelligent asynchronous pre-processing query data.

[0355] Example 4

[0356] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps in a method for intelligent asynchronous preprocessing query data.

[0357] Example 5

[0358] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the intelligent asynchronous pre-processing query data method are implemented.

[0359] Anything not described in this application can be achieved by adopting or drawing on existing technologies.

[0360] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0361] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An intelligent asynchronous pre-processing query data method, characterized in that: include: When a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows to be displayed on a single page, and the first page of data is obtained and organized; Record task information to the memory cache and manage the memory cache, specifically: record the task queried this time to the memory cache of the server, and the recorded information includes the main task identifier, auxiliary task identifier, auxiliary task completion identifier and other related information, Ensure that the memory cache does not store actual query data to prevent memory overflow, and set the destruction time of the memory cache; When the main task is completed, an asynchronous task is started, specifically: after the main task is completed, the backend returns the main task identifier, the total data volume and the first page of data for display, starts a concurrent thread to execute the auxiliary task, sets the auxiliary task completion identifier in the memory cache to being obtained asynchronously, and the auxiliary task executes complete data mining, and stores the mined data in the database real table after organization and processing, and when the auxiliary task is completed, updates the auxiliary task completion identifier in the memory cache to completed; When a request to switch the page number is initiated, the memory cache is checked, and if the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database; Otherwise, re-mine the data and start asynchronous task processing according to the mining status; When an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again based on the cache identifier. Once the data is ready, it can be exported from the database.

2. The method according to claim 1, characterized in that When a query request is initiated, the main task is started, the total number of rows of task data is mined, the paging parameters are determined according to the total number of rows and the preset number of rows displayed on a single page, and the first page of data is obtained and organized, specifically: The user initiates a query request, and the backend generates a globally unique identifier as the main task identifier according to the request. The backend calculates the total number of rows of data that meet the query conditions, and calculates the starting row number, the ending row number, and the total number of pages of the first page of data based on the total number of rows and the preset number of rows to be displayed on a single page. The data rows between the start row sequence number and the end row sequence number are intercepted according to the sorting rule specified by the user, and inserted into the global temporary table.

3. The method according to claim 1, characterized in that When a request to switch the page number is initiated, the memory cache is checked. If the memory cache exists and the status is completed, the corresponding page data is directly obtained from the database cache; otherwise, the data is mined again and asynchronous task processing is started according to the mining status, specifically: The front end initiates a request to switch the page number, and the back end checks whether the memory cache of the server exists according to the main task identifier. If the memory cache does not exist or the auxiliary task completion identifier is 0 or 1, the data of the specified page is re-mined. If the auxiliary task completion identifier is 2, the data of the specified page is filtered out from the database cache. If the auxiliary task completion identifier is 0, the auxiliary task is started to asynchronously organize data.

4. The method according to claim 1, characterized in that: When an export request is initiated, the memory cache status is checked, and it is determined whether the complete data needs to be mined again based on the cache identifier. Once the data is ready, it can be exported from the database, specifically: The front end sends an export request to the back end, carrying the main task identifier, checks the memory cache, and decides whether to mine the complete data again according to the state of the memory cache, or directly obtains the complete data from the database cache and exports it.

5. The method according to claim 1, characterized in that Also includes: Refresh data, specifically: The front end generates a new main task ID, the back end generates a new auxiliary task ID, and starts to re-execute the query task.

6. The method according to claim 1, characterized in that Also includes: Release resources, specifically: The memory cache is created with a destruction time set, and the memory is automatically released when the time is reached. Expired task data caches are checked and cleared through asynchronous threads, and a scheduled task is set to completely clear the database cache at 2 a.m. every day.

7. A computer program product comprising instructions, which, when executed on a device, is characterized in that: The device is enabled to execute the steps in the intelligent asynchronous pre-processing query data method as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the intelligent asynchronous pre-processing query data method as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the intelligent asynchronous pre-processing query data method as described in any one of claims 1-6 are implemented.

Images