Data processing method and device, storage medium and computer equipment

Abstract

The invention discloses a data processing method and device, a storage medium and computer equipment, and relates to the technical field of data processing, the technical field of medical health and the technical field of finance, and the method comprises the following steps: packaging an original structured query language (SQL) query statement input by a user, and generating a packaged SQL query statement; performing data query on a preset database based on the packaging SQL query statement to obtain the line number and the data volume of target data; performing calculation processing based on the line number, the data volume, a preset paging guide line number and a resource parameter to obtain a concurrent number and a paging query parameter; and based on the paging query parameters, starting a thread number corresponding to the concurrent number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set. According to the method, time consumption and resource consumption can be balanced, and the user experience is improved while the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of data processing, medical and health technology, and financial technology, and particularly relates to a data processing method, device, storage medium, and computer device. Background Art

[0002] Traditional data processing processes such as querying and exporting data in back-end projects are very common applications. In the field of medical and health technology, the data queried and exported can be medical data, such as personal health records, prescriptions, inspection reports, etc.; in the field of financial technology, the data queried and exported can be payment data, transaction data, etc. After users query and summarize data on the page, they mostly export the detailed data for further analysis. At this time, if the amount of detailed data is very large and there are many users operating the export simultaneously, the back-end code saves a large amount of data queried from the database in a List container (memory) and writes to the disk at the same time, which is very CPU and memory resource-consuming and may cause interface timeouts or server crashes. Summary of the Invention

[0003] In view of this, the present invention provides a data processing method, device, storage medium, and computer device, mainly aiming to solve the problem that when querying and obtaining large-scale data currently, one-time querying and extraction will cause slow database response, and the transmission and processing of a large amount of data will occupy a large amount of memory and network resources, resulting in low query efficiency.

[0004] To solve the above problems, the present application provides a data processing method, including:

[0005] Wrapping the structured original SQL query statement input by the user to generate a wrapped SQL query statement;

[0006] Based on the wrapped SQL query statement, querying data from a preset database to obtain the number of rows and the amount of data of the target data;

[0007] Based on the number of rows, the amount of data, the preset paging export number of rows, and resource parameters, performing calculation processing to obtain the concurrency number and paging query parameters;

[0008] Based on the paging query parameters, starting the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set.

[0009] Optionally, the wrapping the structured original SQL query statement input by the user to generate a wrapped SQL query statement specifically includes:

[0010] Perform data preprocessing on the original SQL query statement to obtain a first SQL query statement;

[0011] Perform nested wrapping on the first SQL query statement based on a preset first aggregation function and a preset second aggregation function to obtain the wrapped SQL query statement;

[0012] Among them, the preset first aggregation function is used to query the number of rows of the target data, and the preset second aggregation function is used to query the data volume of the target data.

[0013] Optionally, querying the preset database based on the wrapped SQL query statement to obtain the number of rows and the data volume of the target data specifically includes:

[0014] Parse the wrapped SQL query statement and execute a subquery to obtain the target data corresponding to the original SQL query statement;

[0015] Query the target data based on the preset first aggregation function to obtain the number of rows;

[0016] Query the target data based on the preset second aggregation function to obtain the data volume.

[0017] Optionally, calculating and processing based on the number of rows, the data volume, the preset paging export number of rows, and the resource parameters to obtain the concurrency number and the paging query parameters specifically includes:

[0018] Perform calculation and processing based on the number of rows and the preset paging export number of rows to obtain the paging query times;

[0019] Perform calculation and processing based on the paging query times and the number of cores in the resource parameters to obtain the concurrency number;

[0020] Perform calculation and processing based on the remaining memory and the maximum memory usage rate in the resource parameters to obtain the available memory parameter;

[0021] When the data volume is less than or equal to the available memory parameter, perform calculation and processing based on the concurrency number and the data volume to obtain the paging query parameters.

[0022] Optionally, based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set, specifically including:

[0023] Allocate query tasks to each thread based on each paging query parameter;

[0024] Execute the original SQL query statement to generate sub - SQL query statements respectively corresponding to each of the query tasks;

[0025] Extract pages of the target data based on each of the sub - SQL query statements to obtain page - segmented data sets corresponding to each of the threads.

[0026] Optionally, the method further includes:

[0027] In the case where the page - segmented data sets corresponding to each of the threads are not queried within a preset time period, adjust the timeout parameter of the reverse proxy server.

[0028] Optionally, after starting the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform page - segmented data extraction processing on the target data in the preset database based on the page - query parameters and obtaining a page - segmented data set, the method further includes:

[0029] Analyze the export file structure corresponding to the user to obtain custom attribute parameters of the exported document;

[0030] Perform splicing processing on the page - segmented data set based on the attribute parameters to obtain a target document;

[0031] Send the target document to the user.

[0032] To solve the above problems, the present application provides a data processing device, including:

[0033] A packaging module, configured to package the structured original SQL query statement input by the user to generate a packaged SQL query statement;

[0034] A data query module, configured to perform data query on a preset database based on the packaged SQL query statement to obtain the number of rows and the data volume of the target data;

[0035] A calculation module, configured to perform calculation processing based on the number of rows, the data volume, the preset number of rows for page - segmented export, and resource parameters to obtain the concurrency number and page - query parameters;

[0036] A data processing module, configured to start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform page - segmented data extraction processing on the target data in the preset database based on the page - query parameters to obtain a page - segmented data set.

[0037] To solve the above problems, the present application provides a storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the above - mentioned data processing method are implemented.

[0038] To solve the above problems, the present application provides a computer device, which at least includes a memory and a processor. A computer program is stored on the memory, and when the processor executes the computer program on the memory, the steps of the above-mentioned data processing method are implemented.

[0039] Beneficial effects in the present application: Based on the intercepted structured original SQL query statement input by the user, a double aggregation function for querying the number of rows of target data and the data volume of target data is wrapped for enhancement processing. The wrapped query statement is used to query the number of rows of target data and the data volume of target data, enabling the query and acquisition of a large amount of data. Combining the memory parameter and core parameter in the resource parameters, the appropriate concurrency number and paging query parameters are dynamically matched according to the number of rows and data volume of the target data, preventing the server crash phenomenon caused by the query and acquisition of a large number of data, and improving the efficiency of data query and data acquisition.

[0040] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are hereinafter specifically exemplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0042] Figure 1 It shows a schematic diagram of the application environment of a data processing method provided by an embodiment of the present application;

[0043] Figure 2 It shows a schematic flowchart of a data processing method provided by another embodiment of the present application;

[0044] Figure 3 It shows a schematic flowchart of another data processing method provided by another embodiment of the present application;

[0045] Figure 4 It shows a structural block diagram of a data processing device provided by another embodiment of the present application;

[0046] Figure 5 It shows a schematic structural diagram of a computer device in an embodiment of the present application;

[0047] Figure 6 It shows another schematic structural diagram of a computer device in an embodiment of the present application. Detailed Implementation Modes

[0048] Reference is made herein to the various aspects and features of the present application with reference to the accompanying drawings.

[0049] It should be understood that various modifications can be made to the embodiments applied herein. Therefore, the above description should not be regarded as limiting, but merely as an example of the embodiments. Those skilled in the art will think of other modifications within the scope and spirit of the present application.

[0050] The accompanying drawings, which are included in and form a part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0051] These and other features of the present application will become apparent from the following description of the preferred forms of the embodiments given by way of non-limiting examples with reference to the accompanying drawings.

[0052] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art can surely implement many other equivalent forms of the present application.

[0053] When taken in conjunction with the accompanying drawings, the above and other aspects, features, and advantages of the present application will become more apparent in view of the following detailed description.

[0054] Specific embodiments of the present application are hereinafter described with reference to the accompanying drawings; however, it should be understood that the embodiments claimed are merely examples of the present application and can be implemented in various ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely as a basis for the claims and a representative basis for teaching those skilled in the art to use the present application in substantially any suitable detailed structure in a variety of ways.

[0055] This specification may use the phrases "in one embodiment", "in another embodiment", "in yet another embodiment", or "in other embodiments", each of which may refer to one or more of the same or different embodiments according to the present application.

[0056] The sales process optimization method provided by the embodiments of the present application can be applied, for example, in Figure 1In the application environment, the client communicates with the server through the network. The server can receive the structured original SQL query statement input by the user through the client, and wrap the structured original SQL query statement input by the user to generate a wrapped SQL query statement; perform a data query on the preset database based on the wrapped SQL query statement to obtain the number of rows and the data volume of the target data; perform calculation processing based on the number of rows, the data volume, the preset number of rows for paging export, and the resource parameters to obtain the concurrency number and the paging query parameters; based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set, and feedback the paging data set to the client. The present application can dynamically adjust the paging query parameters and the number of threads according to the number of rows, the data volume of the target data, and the current server resource usage situation, which can improve the query efficiency while optimizing the resource utilization rate and enhancing the user experience.

[0057] An embodiment of the present application provides a data processing method, as Figure 2 shown, including:

[0058] Step S101: Wrap the structured original SQL query statement input by the user to generate a wrapped SQL query statement;

[0059] In the specific implementation process, first perform data preprocessing on the structured original SQL query statement input by the user, for example: intercept the semicolon at the end of the original SQL query statement, etc., and then perform nested wrapping on the preprocessed SQL query statement to generate the wrapped SQL query statement. Nested wrapping includes a preset first aggregation function and a preset second aggregation function, which are used to obtain the number of rows and the data volume of the target data respectively.

[0060] Step S102: Perform a data query on the preset database based on the wrapped SQL query statement to obtain the number of rows and the data volume of the target data;

[0061] In the specific implementation process, parse the wrapped SQL query statement, execute a subquery to obtain the target data corresponding to the original SQL query statement; query the target data based on a preset first aggregation function to obtain the number of rows; query the target data based on a preset second aggregation function to obtain the data volume. For example, in the application in the field of medical and health, the target data can come from the hospital's database, and in the field of financial technology, the target data can be business data, transaction data, payment data, etc. stored in the database.

[0062] Step S103: Perform calculation processing based on the number of rows, the data volume, the preset number of rows for paging export, and resource parameters to obtain the concurrency number and paging query parameters;

[0063] In the specific implementation process, perform calculation processing based on the number of rows and the preset number of rows for paging export to obtain the number of paging query times; perform calculation processing based on the number of paging query times and the number of cores in the resource parameters to obtain the concurrency number; perform calculation processing based on the remaining memory and the maximum memory usage rate in the resource parameters to obtain the available memory parameter; when the data volume is less than or equal to the available memory parameter, perform calculation processing based on the concurrency number and the data volume to obtain the paging query parameters.

[0064] Step S104: Based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set.

[0065] In the specific implementation process, based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to allocate query tasks to each thread to obtain the start and end row numbers of the query corresponding to each thread; execute each query task, and perform paging processing on the target data based on each start and end row number of the query to obtain a paging data set corresponding to each thread.

[0066] This application enhances the intercepted structured original SQL query statement input by the user by wrapping a double aggregation function for querying the number of rows of the target data and the data volume of the target data, and uses the wrapped query statement to query and obtain the number of rows of the target data and the data volume of the target data, which can realize the query and acquisition of a large amount of data. Combining the memory parameter and the core parameter in the resource parameters, it dynamically matches appropriate concurrency numbers and paging query parameters for the number of rows and data volume of the target data, prevents the server crash phenomenon caused by the query and acquisition of a large number of data, and improves the efficiency of data query and data acquisition.

[0067] Another embodiment of this application provides another data processing method, as Figure 3 shown, including:

[0068] Step S201: Perform data preprocessing on the original SQL query statement to obtain a first SQL query statement;

[0069] In the specific implementation process of this step, the preprocessing process includes preprocessing processes such as removing redundant symbols and formatting operations. Specifically, the semicolon at the end of the original SQL query statement is intercepted. The semicolon at the end of the SQL statement is optional in some databases, but may cause syntax errors in nested queries or dynamic SQL generation. Therefore, it is usually necessary to remove the semicolon at the end of the original SQL. Clean up the redundant spaces, line breaks, and tab characters in the SQL statement to make it more compact and readable. Convert SQL keywords (such as SELECT, FROM, WHERE, etc.) to uppercase or lowercase uniformly to avoid parsing errors caused by inconsistent case, and obtain the first SQL query statement. The preprocessing process can also include preprocessing processes such as syntax and structure checking, parameterization and security processing, performance optimization, etc. Through the data preprocessing process, the correctness, security, and efficiency of the SQL query during execution can be ensured.

[0070] Step S202: Nest and wrap the first SQL query statement based on a preset first aggregation function and a preset second aggregation function to obtain the wrapped SQL query statement;

[0071] In the specific implementation process of this step, the preset first aggregation function can be select count(*); the preset second aggregation function can be sum(length(column)); the preset first aggregation function is used by the user to query the number of rows of the target data, and the preset second aggregation function is used to query the data volume of the target data.

[0072] Step S203: Parse the wrapped SQL query statement, execute the subquery to obtain the target data corresponding to the original SQL query statement;

[0073] In the specific implementation process of this step, the target data refers to the result returned by the subquery, and these data are the results expected by the original SQL query statement. By executing the subquery, the target data can be obtained. It lays a foundation for subsequently obtaining the number of rows and data volume of the target data. In the application in the field of medical and health, the target data can come from personal health records, prescriptions, inspection reports, etc. in the hospital's database; while in the field of financial technology, the target data can be business data, transaction data, payment data, etc. stored in the database.

[0074] Step S204: Query the target data based on the preset first aggregation function to obtain the number of rows;

[0075] In the specific implementation process of this step, after executing the wrapped SQL, the returned result contains two columns, including the total number of rows of the target data corresponding to total_rows obtained by querying the target data based on the preset first aggregation function.

[0076] Step S205: Query the target data based on a preset second aggregation function to obtain the data volume;

[0077] In the specific implementation process of this step, after executing the packaged SQL, the returned result contains two columns, including the total length (in bytes) of the target column data corresponding to total_data_size obtained by querying the target data based on a preset second aggregation function. By parsing the returned result, the number of rows and the data volume of the target data can be obtained.

[0078] Step S206: Calculate based on the number of rows and the preset paging export number of rows to obtain the paging query times;

[0079] In the specific implementation process of this step, perform a division operation based on the number of rows and the preset paging export number of rows to calculate the paging query times.

[0080] Step S207: Calculate based on the paging query times and the number of cores in the resource parameters to obtain the concurrency number;

[0081] In the specific implementation process of this step, compare the paging query times with the number of cores. When the paging query times are greater than or equal to the number of cores, determine the number of cores as the concurrency number; when the paging query times are less than the number of cores, determine the paging query times as the concurrency number. Taking the smaller value of the paging times and the number of cores to dynamically adjust the number of threads can make full use of hardware resources, avoid the number of threads exceeding the CPU core number, reduce the context switching overhead, reduce the pressure on the database connection pool, and ensure the stable operation of the database. In the field of medical and health, the hospital's user profile system generates a large amount of medical profile data every day. Taking the smaller value of the paging times and the number of cores to dynamically adjust the number of threads to query the user profile data can reduce the system pressure of the medical user profile system and ensure the stability of the database.

[0082] Step S208: Calculate based on the remaining memory and the maximum memory usage rate in the resource parameters to obtain the available memory parameter;

[0083] In the specific implementation process of this step, the maximum memory usage rate can be set to 80%, and the maximum memory usage rate can be set according to actual needs. By performing a multiplication operation based on the remaining memory and the maximum memory usage rate in the resource parameters, the available memory parameter can be obtained, which can make full use of the hardware resources. By reasonably managing memory usage, the system can reduce unnecessary memory occupancy without affecting performance, thereby reducing energy consumption. This method is particularly obvious in the field of medical and health. Using the maximum memory usage rate to determine the available memory parameter can avoid unnecessary memory occupancy and prevent the database system from crashing, ensuring the data security and stability of the system while maximizing the use of resources.

[0084] Step S209: When the data volume is less than or equal to the available memory parameter, perform a calculation process based on the concurrency number and the data volume to obtain a paging query parameter.

[0085] In the specific implementation process of this step, the paging query parameter includes the target query page number corresponding to each thread, the starting row number and the ending row number corresponding to each thread; specifically, perform a calculation process based on the data volume and the preset number of rows for paging export to obtain the total number of pages; perform a division operation based on the total number of pages and the concurrency number to obtain the target query page number processed by each thread; a thread can export only one page of data or multiple pages of data. Perform a multiplication operation based on the target query page number and the preset number of rows for paging export to obtain the target data volume processed by each thread. Perform a calculation process based on the target query page number, the target data volume corresponding to each thread, and the total data volume to obtain the starting row number and the ending row number corresponding to each thread respectively.

[0086] Step S210: Allocate query tasks for each thread based on each paging query parameter.

[0087] In the specific implementation process of this step, allocate the starting row number and the ending row number corresponding to each thread one by one to obtain the query tasks corresponding to each thread respectively. For example: Thread 1: starting row number startRow = 1, ending row number endRow = 300; Thread 2: starting row number startRow = 301, ending row number endRow = 600; Thread 3: starting row number startRow = 601, ending row number endRow = 900; Thread 4: starting row number startRow = 901, ending row number endRow = 1000.

[0088] Step S211: Execute the original SQL query statement to generate sub - SQL query statements corresponding to each query task respectively.

[0089] In the specific implementation process of this step, the original SQL query statement is executed to generate sub - SQL query statements respectively corresponding to the query tasks of each thread. For example, the sub - SQL query statement of thread 1 is SELECT * FROM your_table LIMIT 300 OFFSET 0; the sub - SQL query statement of thread 2 is SELECT * FROM your_table LIMIT 300 OFFSET 300; the sub - SQL query statement of thread 3 is SELECT * FROM your_table LIMIT 300 OFFSET 600; the sub - SQL query statement of thread 4 is SELECT * FROM your_table LIMIT 100 OFFSET 900.

[0090] Step S212: Based on each of the sub - SQL query statements, perform paged extraction on the target data to obtain paged data sets corresponding to each of the threads;

[0091] In the specific implementation process of this step, each of the sub - SQL query statements is executed to perform paged extraction on the target data, obtaining paged data sets corresponding to each of the threads, and each of the paged data sets is saved in a preset storage area. The preset storage area can be a cache area in memory. The query results are stored in memory first instead of being directly written to disk. The read - write speed of memory is much faster than that of disk, so the waiting time of the program can be reduced.

[0092] Step S213: Parse the export file structure corresponding to the user to obtain the custom attribute parameters of the export document;

[0093] In the specific implementation process of this step, the custom attribute parameters include header parameters and styles. The header parameters include the display name of the header, which is identified by the header tag. The styles include field formats, and the field formats include bold, italic, font size, font color, etc., which are identified by the style tag. Specifically, a library for exporting excel files can be referenced, such as introducing github.com, fast - excel or the export library. The export file structure can be configured in Go language. For example, a struct named Example is predefined, including the following fields: BaseExpStrunct is an embedded field used to inherit the basic export function and export fields; FieldA int is an integer - type field used to store numerical values; FieldB string is a string - type field used to store text. By parsing the export file structure corresponding to the user, custom attribute parameters such as the header name and field format of the export document are obtained.

[0094] Step S214: Based on the attribute parameters, splice the paged data sets to obtain a target document;

[0095] In the specific implementation of this step, based on the attribute parameters of the target document to be downloaded defined by the user, splice each of the paged data sets to obtain the target document, realizing the custom export document format and improving the user experience and personalization. Inside the library, the structure fields and their tags are parsed through reflection, and the header and style of the target document are dynamically generated. The target document can be an excel document; the user can easily adjust the exported content and style by modifying the structure definition without modifying the export logic code. In the case where the paged data sets corresponding to the threads are not queried within a preset duration, adjust the timeout parameters of the reverse proxy server. The timeout parameters include: the timeout time for the reverse proxy server to establish a connection with the backend server, the timeout time for the reverse proxy server to send a request to the backend server, the timeout time for the reverse proxy server to wait for a response from the backend server, etc. In the case of having a reverse proxy, when downloading the target document, a connection timeout between the reverse proxy server and the backend server will cause error prompts such as response timeouts. Therefore, it is still necessary to cooperate with the overall architecture design to adjust the configuration parameters of middleware such as the reverse proxy server to optimize the export file function.

[0096] Step S215: Send the target document to the user.

[0097] In the specific implementation of this step, the target document can be sent to the user according to the user's target acquisition method; alternatively, the exported target document can be stored in a predetermined storage area, and the download link is recorded and sent to the user to prompt the user to download the target document according to the download link.

[0098] This application preprocesses and packages the original SQL query statement to enhance the original SQL query statement for obtaining the number of rows and data volume of the target data to be queried. It is applicable to the query acquisition and export processing of a large amount of data, can customize the number of paged export rows, and dynamically adjust the number of threads for paged data extraction processing based on the queried number of rows, data volume, and the preset number of paged export rows in combination with the server core number and memory parameters. Dynamically adjust the resource consumption according to the server cpu and memory usage conditions, and balance the time consumption and resource consumption as much as possible to make the export process smooth, which can greatly improve the user experience.

[0099] Another embodiment of this application provides a data processing device, as Figure 4 shown, including:

[0100] Packaging module 1, which is used to package the structured original SQL query statement input by the user to generate a packaged SQL query statement;

[0101] Data query module 2, which is used to perform data query on a preset database based on the packaged SQL query statement to obtain the number of rows and the data volume of the target data;

[0102] Calculation module 3, which is used to perform calculation processing based on the number of rows, the data volume, the preset paging export number of rows, and resource parameters to obtain the concurrency number and paging query parameters;

[0103] Data processing module 4, which is used to start the number of threads corresponding to the concurrency number based on the paging query parameters to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set.

[0104] In the specific implementation process, the packaging module 1 is specifically used for: performing data preprocessing on the original SQL query statement to obtain a first SQL query statement; performing nested packaging on the first SQL query statement based on a preset first aggregation function and a preset second aggregation function to obtain the packaged SQL query statement; wherein, the preset first aggregation function is used to query the number of rows of the target data, and the preset second aggregation function is used to query the data volume of the target data.

[0105] In the specific implementation process, the data query module 2 is specifically used for: parsing the packaged SQL query statement, executing a subquery to obtain the target data corresponding to the original SQL query statement; querying the target data based on a preset first aggregation function to obtain the number of rows; querying the target data based on a preset second aggregation function to obtain the data volume.

[0106] In the specific implementation process, the calculation module 3 is specifically used for: performing calculation processing based on the number of rows and the preset paging export number of rows to obtain the paging query times; performing calculation processing based on the paging query times and the number of cores in the resource parameters to obtain the concurrency number; performing calculation processing based on the remaining memory and the maximum memory usage rate in the resource parameters to obtain an available memory parameter; when the data volume is less than or equal to the available memory parameter, performing calculation processing based on the concurrency number and the data volume to obtain the paging query parameters.

[0107] In the specific implementation process, the data processing module 4 is specifically configured to: allocate query tasks to each thread based on each of the paging query parameters; execute the original SQL query statement to generate sub-SQL query statements respectively corresponding to each of the query tasks; perform paging extraction on the target data based on each of the sub-SQL query statements to obtain paged data sets corresponding to each of the threads.

[0108] In the specific implementation process, the device further includes a parameter adjustment module, and the parameter adjustment module is specifically configured to: adjust the timeout parameter of the reverse proxy server in the case where no paged data sets corresponding to each of the threads are queried within a preset duration.

[0109] In the specific implementation process, the device further includes an export module, and the export module is specifically configured to: parse the export file structure corresponding to the user to obtain custom attribute parameters of the export document; perform splicing processing on the paged data set based on the attribute parameters to obtain a target document; send the target document to the user.

[0110] This application preprocesses and packages the original SQL query statement to enhance the original SQL query statement for obtaining the number of rows and data volume of the target data to be queried. It is applicable to querying and exporting a large amount of data, can customize the number of rows for paging export, and dynamically adjusts the number of threads for paging data extraction processing based on the number of rows, data volume, and preset number of paging export rows in combination with the server core number and memory parameters. It dynamically adjusts resource consumption according to the server CPU and memory usage conditions, balances the time consumption and resource consumption as much as possible, makes the export process smooth, and can greatly improve the user experience.

[0111] Another embodiment of this application provides a storage medium, and the storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0112] Step 1: Package the structured original SQL query statement input by the user to generate a packaged SQL query statement;

[0113] Step 2: Perform data query on a preset database based on the packaged SQL query statement to obtain the number of rows and data volume of the target data;

[0114] Step 3: Perform calculation processing based on the number of rows, the data volume, the preset number of paging export rows, and resource parameters to obtain the concurrency number and paging query parameters;

[0115] Step 4: Based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database, and obtain a paging data set.

[0116] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0117] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0118] For the specific implementation process of the above method steps, reference can be made to the embodiments of any of the above data processing methods, and this embodiment will not be repeated here.

[0119] This application performs data preprocessing and packaging on the original SQL query statement to enhance the original SQL query statement for obtaining the number of rows and data volume of the target data to be queried. It is applicable to querying and exporting large volumes of data, can customize the number of rows for paging export, and dynamically adjusts the number of threads for paging data extraction processing based on the number of rows, data volume, and the preset number of paging export rows in combination with the server core count and memory parameters. It dynamically adjusts resource consumption according to the server CPU and memory usage conditions, balances the time consumption and resource consumption as much as possible, makes the export process smooth, and can greatly improve the user experience.

[0120] Another embodiment of this application provides a computer device, which can be a server, and its internal structure diagram can be as Figure 5 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer device program is executed by the processor, it realizes the functions or steps on the server side of a data processing method.

[0121] In one embodiment, a computer device is provided, which can be a client. Its internal structure diagram can be as Figure 6 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer device program is executed by the processor, it realizes the functions or steps on the client side of a data processing method.

[0122] Another embodiment of this application provides a computer device, at least including a memory and a processor. A computer program is stored on the memory. When the processor executes the computer program on the memory, the following method steps are realized:

[0123] Step 1: Package the structured original SQL query statement input by the user to generate a packaged SQL query statement;

[0124] Step 2: Perform data query on the preset database based on the packaged SQL query statement to obtain the number of rows and the data volume of the target data;

[0125] Step 3: Perform calculation processing based on the number of rows, the data volume, the preset number of rows for paging export, and the resource parameters to obtain the concurrency number and the paging query parameters;

[0126] Step 4: Based on the paging query parameters, start the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database, and obtain a paging data set.

[0127] For the specific implementation process of the above method steps, reference can be made to the embodiments of any of the above data processing methods, and this embodiment will not be repeated here.

[0128] In this application, data preprocessing and packaging are performed on the original SQL query statement to enhance the original SQL query statement for obtaining the number of rows and the data volume of the target data to be queried. It is applicable to querying and exporting a large amount of data, and the number of rows for paging export can be customized. Based on the queried number of rows, data volume, and the preset number of rows for paging export, combined with the server core number and memory parameters, the number of threads for paging data extraction processing is dynamically adjusted. According to the server CPU and memory usage conditions, the resource consumption is dynamically adjusted to balance the time consumption and resource consumption as much as possible, making the export process smooth and greatly improving the user experience.

[0129] The above embodiments are only exemplary embodiments of this application and are not used to limit this application. The protection scope of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of this application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of this application.

Claims

1. A data processing method, characterized in that: include: Package the structured original SQL query statement input by the user to generate a packaged SQL query statement; Performing data query on a preset database based on the packaged SQL query statement to obtain the number of rows and data volume of target data; Calculation is performed based on the number of rows, the amount of data, the preset number of rows exported by paging, and resource parameters to obtain the number of concurrent queries and paging query parameters; Based on the paging query parameters, the number of threads corresponding to the concurrency number is started to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set.

2. The method according to claim 1, characterized in that The step of packaging the structured original SQL query statement input by the user to generate a packaged SQL query statement specifically includes: Performing data preprocessing on the original SQL query statement to obtain a first SQL query statement; Nesting and packaging the first SQL query statement based on a preset first aggregation function and a preset second aggregation function to obtain the packaged SQL query statement; Among them, the preset first aggregation function is used to query the number of rows of the target data, and the preset second aggregation function is used to query the data volume of the target data.

3. The method according to claim 2, characterized in that The querying of the preset database based on the packaged SQL query statement to obtain the number of rows and amount of target data specifically includes: Parsing the packaged SQL query statement, executing a subquery to obtain the target data corresponding to the original SQL query statement; Query the target data based on a preset first aggregation function to obtain the number of rows; The target data is queried based on a preset second aggregation function to obtain the data volume.

4. The method according to claim 1, characterized in that The calculation and processing based on the number of rows, the amount of data, the preset number of rows exported by paging, and resource parameters to obtain the number of concurrent queries and paging query parameters specifically includes: Calculate and process based on the number of rows and the preset number of paging export rows to obtain the number of paging queries; Calculate and process based on the number of paging queries and the number of cores in the resource parameter to obtain the number of concurrent queries; Calculate and process the remaining memory and the maximum memory usage in the resource parameters to obtain an available memory parameter; When the data volume is less than or equal to the available memory parameter, a paging query parameter is obtained by performing calculation based on the concurrency number and the data volume.

5. The method according to claim 1, characterized in that Based on the paging query parameter, starting the number of threads corresponding to the concurrency number to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set specifically includes: Allocate query tasks to each thread based on each of the paging query parameters; Execute the original SQL query statement to generate sub-SQL query statements corresponding to each of the query tasks; The target data is extracted by pages based on each of the sub-SQL query statements to obtain a paginated data set corresponding to each of the threads.

6. The method according to claim 5, characterized in that The method further comprises: When the paging data set corresponding to each thread is not found within a preset time period, the timeout parameter of the reverse proxy server is adjusted.

7. The method according to claim 1, characterized in that After starting the number of threads corresponding to the number of concurrent queries based on the paging query parameters to execute the original SQL query statement to perform paging data extraction processing on the target data in the preset database to obtain a paging data set, the method further includes: Parsing the export file structure corresponding to the user to obtain custom attribute parameters of the export document; Based on the attribute parameters, the paginated data sets are spliced ​​to obtain a target document; The target document is sent to the user.

8. A data processing device, characterized in that: include: The packaging module is used to package the structured original SQL query statement input by the user and generate a packaged SQL query statement; A data query module is used to query the preset database based on the packaged SQL query statement to obtain the number of rows and data volume of the target data; A calculation module, used to perform calculation processing based on the number of rows, the amount of data, the preset number of paging export rows and resource parameters to obtain the number of concurrent queries and paging query parameters; The data processing module is used to start the number of threads corresponding to the concurrency number to execute the original SQL query statement based on the paging query parameters to perform paging data extraction processing on the target data in the preset database to obtain a paging data set.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer device, characterized in that: The device at least comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program on the memory.

Images