Data processing optimization method
By adopting a paging or streaming query in the database query, the data query results cached the data query results to the middleware, and the data is processed in memory, which solves the memory overflow problem caused by large data query, and realizes more efficient memory usage and databases and databases. Connect management.
Patent Information
- Application Number
- CN202311373176.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2025-01-28
AI Technical Summary
When the data volume reaches one million or even tens of millions, obtaining data from the database for analysis may lead to memory soaring or even memory overflow.
By setting the query conditions, turn on the database data query, use pagination inquiries or streaming queries, cache the data query results to the middleware, such as Redis, and obtain the required data from the middleware in the memory.
Avoid loading all data to memory at one time, thereby avoiding the survival of the memory or memory overflow, and reducing the memory usage rate through a current query, avoiding the database chief.
Smart Images
Figure CN119377267A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of data processing, and in particular to a data processing optimization method. Background Art
[0002] As the system operates and business data flows in, the information in the system is increasing. In actual development scenarios, you may need to query data for a month, a quarter, or even a year. When the amount of data reaches millions or even tens of millions, it is a troublesome problem to obtain this data from the database for data analysis. If no optimization measures are taken, loading all the data into the memory at once will cause a surge in memory usage or even memory overflow. Summary of the invention
[0003] In order to avoid memory overflow, the present application provides a data processing optimization method.
[0004] The technical solution adopted by the present invention to solve the above problems is:
[0005] Data processing optimization methods, including:
[0006] Step 1. Set the query conditions and start database data query;
[0007] Step 2: Cache the data query results to the middleware;
[0008] Step 3: The memory obtains the query results from the middleware and processes them.
[0009] Furthermore, in step 1, the data query method is paging query or streaming query.
[0010] Furthermore, after caching the data query results in the middleware, step 2 further includes: disconnecting the connection between the database and the middleware.
[0011] Furthermore, the step 3 also includes: saving the processing results and corresponding query conditions to the middleware.
[0012] Furthermore, after setting the query conditions, step 1 further includes: determining whether the query conditions are the first query, and if so, starting the database data query; if not, the memory directly obtains the processing result from the middleware.
[0013] Furthermore, the middleware is Redis.
[0014] Compared with the prior art, the present invention has the following beneficial effects: by storing the data query results in the middleware, the memory obtains the required data from the middleware for processing, thereby avoiding loading all data into the memory at once, which may cause a memory surge or memory overflow. When querying the database, the use of streaming query can further reduce the memory usage. After the database query is completed, the connection with the database can be disconnected to avoid a long database connection that affects the use of the database. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 Flowchart of the method for data processing optimization;
[0016] Figure 2 It is a flow chart of the data processing optimization method corresponding to the embodiment. DETAILED DESCRIPTION
[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0018] like Figure 1 As shown, the data processing optimization method comprises:
[0019] Step 1. Set the query conditions and start database data query. The data query method is paging query or streaming query.
[0020] Step 2: Cache the data query results to the middleware;
[0021] Step 3: The memory obtains the query results from the middleware and processes them.
[0022] Example
[0023] Take streaming query as an example, Figure 2 As shown, the data processing optimization method comprises:
[0024] Step S1: In the backend framework configuration file, enable database streaming query;
[0025] Step S2: The user initiates a large amount of data query operation based on monthly, quarterly, or annual basis;
[0026] Step S3: The program configures the data stream query object, sets the SQL statement and query parameters;
[0027] Step S4: Create a data stream query method;
[0028] Step S5: The program requests the database server and keeps connected to the remote database;
[0029] Step S6: The database server applies for a data buffer;
[0030] Step S7, execute the streaming query statement and put the result set into the buffer;
[0031] Step S8: The database server returns an iterator;
[0032] Step S9: The program obtains data from the iterator one by one and caches it in Redis;
[0033] Step S10: All result set data is cached in the middleware, and the program releases the occupied database connection to avoid long database connection;
[0034] Step S11, smoothly load data from Redis to program memory and perform analysis and processing operations;
[0035] Step S12: Generate final response data and store the response data in Redis cache;
[0036] Step S13: Clear the result set in Redis to release memory. The next time the same condition query is made, the cache will be used first to improve query efficiency.
[0037] Step S14: The interface returns response data.
Claims
1. A data processing optimization method, characterized in that: include: Step 1. Set the query conditions and start database data query; Step 2: Cache the data query results to the middleware; Step 3: The memory obtains the query results from the middleware and processes them.
2. The data processing optimization method according to claim 1, characterized in that: In step 1, the data query method is paging query or streaming query.
3. The data processing optimization method according to claim 1, characterized in that: After the step 2 caches the data query results to the middleware, it also includes: disconnecting the connection between the database and the middleware.
4. The data processing optimization method according to any one of claims 1 to 3, characterized in that: The step 3 also includes: saving the processing results and corresponding query conditions to the middleware.
5. The data processing optimization method according to claim 4, characterized in that: After setting the query condition in step 1, the step also includes: determining whether the query condition is the first query, if so, starting the database data query; if not, the memory directly obtains the processing result from the middleware.
6. The data processing optimization method according to claim 4, characterized in that: The middleware is Redis.