Method, system and equipment for optimizing single-machine governance of large-volume data and medium
Through index paging query and multi-threaded processing, the database query and update process for large amounts of data is optimized, solving the problems of slow database query speed and long waiting time, and achieving more efficient data management and resource utilization.
Patent Information
- Application Number
- CN202510649034.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-20
- Publication Date
- 2025-09-23
AI Technical Summary
How to optimize large-scale data governance processes, improve database query speed, and save service waiting time.
Use indexes for efficient paging queries in the database, use the maximum code of the previous page as the starting parameter for the next page, store the processed data in batches, parallelize the data processing process through multi-threaded processing, and update the database in batches after all the data on each page is processed.
It improves database query speed, saves service waiting time, alleviates database computing pressure, optimizes server resource allocation, and improves data management efficiency.
Smart Images
Figure CN120687429A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer software application technology, and in particular to an optimization method, system, device and medium for single-machine management of large-volume data. Background Art
[0002] With the rapid development of internet technology, people's work is increasingly inseparable from information technology. In daily work, people are increasingly using internet platforms to maintain and manage data. Compared to traditional paper materials, electronic data maintenance is more convenient and easier to preserve, greatly improving the work efficiency of data managers. However, internet services often face speed and performance bottlenecks as data volume increases.
[0003] Therefore, how to optimize large-scale data governance processes, improve database query speed, and save service waiting time is a technical problem that needs to be solved urgently. Summary of the Invention
[0004] The technical task of the present invention is to provide an optimization method, system, equipment and medium for single-machine management of large-volume data to solve the problem of how to optimize large-scale data management processes, improve database query speed, and save service waiting time.
[0005] The technical task of the present invention is achieved in the following manner: an optimization method for single-machine management of large-volume data. The method uses indexes to perform efficient database paging queries. During paging queries, the maximum code of the previous page is used as the starting parameter for the next page. The managed data is stored in batches to improve data update efficiency. The details are as follows:
[0006] Data query: Create a numeric index for the table, with the index field as index and the index start parameter as startIndex. Process data concurrently, obtain the index of the last entry, set startIndex = index, insert result data in batches, and obtain data query results.
[0007] Data management: After obtaining a set of data query results, multi-threaded processing is performed. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop.
[0008] Data update: When updating the managed data to the database, wait until all the data on each page is managed and then update it to the database in batches.
[0009] Preferably, data query is performed by paging query (starting from index 0), obtaining the data list page by page, processing each page of data, and dynamically updating the query starting index (taking the index value of the last record on the previous page) until there is no more data to query; the details are as follows:
[0010] Initialize variables: define a long integer variable startIndex with an initial value of 0L to indicate the starting index of the query; also define an integer variable page with an initial value of 1 to record the current page number;
[0011] First query: call the getList(startIndex) method, pass in startIndex (initial 0), and get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record;
[0012] Loop data processing: call the dealList(list) method through a do-while loop to clean and convert the list data on the current page. The page is incremented to indicate the next page.
[0013] Update the starting index: Get the "index" field value of the last record from the list on the current page, that is, through list.get(list.size()-1).get("index"), convert the corresponding value to a string and then to a long integer, and assign it to startIndex as the starting index for the next page query;
[0014] Query the next page of data: call getList(startIndex), pass in the new startIndex, and get the list of the next page;
[0015] Loop condition: When the size of the newly queried list is 0 (that is, there is no more data), exit the do-while loop;
[0016] Termination condition: When the list returned by getList(startIndex) is empty (or its size is zero), it means that all data has been processed and the loop ends.
[0017] As a preferred method, data processing is to start 10 asynchronous tasks in parallel through the thread pool executor, each task processes a different part of the finalList (100 data in each part), so as to achieve the purpose of using multi-threading to accelerate the data management process; the details are as follows:
[0018] Define the paging size: define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread each time (i.e. 100 data items per page);
[0019] Create asynchronous tasks: Use the thread pool executor to submit multiple asynchronous tasks (a total of 10), each task calls the dealPeopleList method to process data; each task returns the processing result through the Future object (such as listFuture1, listFuture2, etc.), and the result is List <Map<String,Object> >Type, indicating the processed data list;
[0020] Task parameter description: The input parameters of the 10 asynchronous tasks listFuture10 are finalList (total data list), thread number and pageSize;
[0021] Asynchronous processing logic: Each thread independently processes a portion of the data in finalList divided by thread number and pageSize. Specifically, the dealPeopleList method is used to page, convert, or clean the data, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object.
[0022] As a preference, each thread loop manages data based on the page number pageIndex and page size pageSize passed in, intercepts the data of the corresponding page from the total data list list, and processes each data through the processOnePeople method; if the list is empty or there is no data on the current page, returns null; otherwise, returns the processing result of the current page; the details are as follows:
[0023] Method definition: define a public method dealPeopleList, which receives three parameters: list, pageIndex and pageSize, and returns a List <Map<String,Object> >, indicating the data list after processing the current page; the list parameter refers to the type of List <Map<String,Object> >, indicating the total data list to be processed; the pageIndex parameter is of type int, indicating the page number currently being processed (starting from 1); the pageSize parameter is of type int, indicating the number of data items to be processed per page;
[0024] Initialize the current page data list: create an empty ArrayList object currentList to store the data of the current page;
[0025] Empty list check: call StringUtil.isEmptyList(list) to check whether the input list list is empty: if it is empty, it returns null directly, indicating that there is no data to process;
[0026] Page range check: Check whether the size of the input list is less than (pageIndex-1)*pageSize+1: If so, it means there is no data on the current page, and null is returned;
[0027] Intercept the current page data, the situation is as follows:
[0028] The data is less than one page: If the size of the list is less than pageIndex*pageSize, it means that the current page data is less than pageSize. Use list.subList((pageIndex-1)*pageSize, list.size()) to intercept the data from (pageIndex-1)*pageSize to the end of the list and store it in currentList;
[0029] The data fills one page: If the size of the list is not less than pageIndex*pageSize, use list.subList((pageIndex-1)*pageSize, pageIndex*pageSize) to intercept the data from (pageIndex-1)*pageSize to pageIndex*pageSize and store it in currentList;
[0030] Processing a single piece of data: traverse each piece of data in currentList (Map<String,Object> Type), and call the processOnePeople(people) method to perform specific data cleaning and conversion operations on each data;
[0031] Return result: Returns the processed currentList (which may be null or the data list of the current page).
[0032] More optimally, data updates are achieved by obtaining the processing results of multiple asynchronous tasks through getFuture, merging the results of multiple asynchronous tasks into a list, and finally updating the merged data in batches to the database, thereby aggregating the data processed by multiple threads and persisting them uniformly. The details are as follows:
[0033] Get asynchronous task results: Get the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type data lists are stored in list1 to list10 variables respectively; the second parameter 0 of the getFuture method may be a timeout or other configuration (the specific meaning depends on the context);
[0034] Merge data lists: Use the list1.addAll() method to append the data from list2 to list10 to list1 in sequence. Finally, list1 contains the data processed by all asynchronous tasks, that is, the collection of list1 to list10.
[0035] Batch update to the database: Call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and store the processed data persistently or perform other database operations.
[0036] An optimization system for single-machine management of large-volume data, the system comprising:
[0037] The data query module is used to create a numeric index for the table. The index field is index and the index start parameter is startIndex. It processes data concurrently, obtains the index of the last record, sets startIndex = index, inserts result data in batches, and obtains data query results.
[0038] The data management module is used to obtain a set of data query results and then perform multi-threaded processing. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop.
[0039] The data update module is used to update the managed data to the database. After all the data on each page is managed, it is updated to the database in batches.
[0040] Preferably, the data query module obtains the data list page by page through a paging query (starting from index 0), processes each page of data, and dynamically updates the starting index of the query (taking the index value of the last record on the previous page) until there is no more data to query; the data query module includes:
[0041] The variable initialization submodule is used to define a long integer variable startIndex with an initial value of 0L, which is used to indicate the starting index of the query; and also defines an integer variable page with an initial value of 1, which is used to record the current page number;
[0042] The first query submodule is used to call the getList(startIndex) method, pass in the startIndex (initial 0), and get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record;
[0043] The data processing submodule is used to call the dealList(list) method through a do-while loop to clean and convert the list data on the current page. The page is automatically incremented to indicate the next page.
[0044] The starting index update submodule is used to obtain the "index" field value of the last record from the list of the current page, that is, through list.get(list.size()-1).get("index"), and convert the corresponding value into a string and then into a long integer, and assign it to startIndex as the starting index for the next page query;
[0045] The next page data query submodule is used to call getList(startIndex), pass in the new startIndex, and get the list of the next page;
[0046] The loop condition submodule is used to exit the do-while loop when the size of the newly queried list is 0 (that is, there is no more data);
[0047] The conditional termination submodule is used to indicate that all data has been processed and the loop ends when the list returned by getList(startIndex) is empty (or its size is zero).
[0048] More optimally, the data processing module uses a thread pool executor to launch 10 asynchronous tasks in parallel. Each task processes a different part of the finalList (100 data items per part), thereby accelerating the data management process through multithreading. The data processing module includes:
[0049] The paging size definition submodule is used to define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread each time (i.e., 100 data items per page);
[0050] The asynchronous task creation submodule is used to submit multiple asynchronous tasks (10 in total) using the thread pool executor. Each task calls the dealPeopleList method to process data. Each task returns the processing result through a Future object (such as listFuture1, listFuture2, etc.). The result is a List<T> <Map<String,Object> >Type, indicating the processed data list;
[0051] The task parameter description submodule, which is used for the 10 asynchronous tasks, has the following input parameters: finalList (total data list), thread number, and pageSize;
[0052] The asynchronous processing logic submodule is used by each thread to independently process a portion of the data in finalList divided by thread number and pageSize. Specifically, the dealPeopleList method is used to paginate, convert, or clean the data, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object.
[0053] The data update module obtains the processing results of multiple asynchronous tasks through getFuture, merges the results of multiple asynchronous tasks into a list, and finally updates the merged data in batches to the database, achieving the purpose of aggregating and persisting the data processed by multiple threads. The data update module includes:
[0054] The asynchronous task result acquisition submodule is used to obtain the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type data lists are stored in list1 to list10 variables respectively; the second parameter 0 of the getFuture method may be a timeout or other configuration (the specific meaning depends on the context);
[0055] The data list merging submodule is used to append the data from list2 to list10 to list1 in sequence using the list1.addAll() method. Finally, list1 contains the data after all asynchronous tasks are processed, that is, the collection of list1 to list10.
[0056] The batch update submodule is used to call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and persist the processed data or perform other database operations.
[0057] An electronic device comprising: a memory and at least one processor;
[0058] Wherein, the memory stores a computer program;
[0059] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the above-mentioned optimization method for single-machine management of large-volume data.
[0060] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the optimization method for single-machine management of large-volume data as described above.
[0061] The optimization method, system, device, and medium for single-machine management of large-volume data of the present invention have the following advantages:
[0062] (1) When the service is deployed on a single node, the present invention queries a large amount of data in a data table, performs data management, and finally updates the data to the database. At each node in this data processing flow, the speed and performance are optimized to achieve the optimal time solution for the flow.
[0063] (2) The present invention uses an index paging method to efficiently increase database query speed, save service waiting time, and at the same time relieve database computing pressure, optimize server resource allocation, and reduce server pressure;
[0064] (3) The present invention can flexibly configure the number of threads for multi-threaded data processing according to actual needs, find the fastest time for parallel processing of list data when machine resources allow, and improve the processing time and efficiency of single-page data;
[0065] (4) The present invention saves more time and resources by updating data in batches than storing data in a single place, thus improving the efficiency of data updating;
[0066] (5) The present invention optimizes the three nodes of data query, management, and update, thereby improving data management efficiency. At the same time, it optimizes the configuration of machine and database resources and improves database efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The present invention will be further described below with reference to the accompanying drawings.
[0068] Attachment Figure 1 Schematic diagram of the process of data query for MySQL database. DETAILED DESCRIPTION
[0069] The optimization method, system, device and medium for single-machine management of large-volume data of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.
[0070] Example 1:
[0071] This embodiment provides an optimized method for managing large amounts of data on a single machine. This method uses indexes to perform efficient database paging queries. During paging queries, the maximum code on the previous page is used as the starting parameter for the next page. The managed data is stored in batches, improving data update efficiency. The details are as follows:
[0072] S1. Data query: Create a numeric index for the table, with the index field as index and the index start parameter as startIndex. Process the data concurrently, obtain the index of the last entry, set startIndex = index, insert the result data in batches, and obtain the data query results.
[0073] S2. Data management: After obtaining a set of data query results, multi-threaded processing is performed. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop.
[0074] S3. Data update: When updating the managed data to the database, wait until all the data on each page is managed and then update it to the database in batches.
[0075] Large volumes of data are typically queried in pages, processing each page before proceeding to the next. For example, in MySQL, assuming 1,000 records per page, a typical query would be to append "limit 0,1000" to the query statement for the first page, "limit 9000,10000" for the 100th page, and "limit 1000*(n-1),n*1000" for the nth page. This conventional approach has a performance impact: queries become slower as they progress. The first page is the fastest query, but when the data volume in the table reaches a certain level, querying larger pages becomes increasingly difficult, wasting significant waiting time.
[0076] In the above case, to optimize the paging query, first create an index for the table, and a numeric index will have the best optimization effect. Assume that the index field is index and the index start parameter is startIndex, as shown in the attached Figure 1 As shown in the example below, using MySQL as an example, the query SQL for the first page ends with "order by index asc limit 1000". After the program finishes processing the first page, the index of the last record is assigned to startIndex, i.e., startIndex = index. The query SQL for the second page ends with "WHERE index>#{startIndex}order by index asc limit 1000". Assuming the index field value of the last record on page N-1 is x, the query SQL for page N ends with "WHERE index>x order by index asc limit 1000", and so on. Using index sorting and comparison for paging queries significantly improves query speed the larger the data volume.
[0077] The data query in step S1 of this embodiment is performed by paging (starting from index 0), obtaining a data list page by page, processing each page of data, and dynamically updating the starting index of the query (taking the index value of the last record on the previous page) until there is no more data to query; the details are as follows:
[0078] S101. Initialize variables: define a long integer variable startIndex with an initial value of 0L to indicate the starting index of the query; also define an integer variable page with an initial value of 1 to record the current page number;
[0079] S102, first query: call getList(startIndex) method, pass in startIndex (initial 0), get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record;
[0080] S103, loop processing data: call the dealList(list) method through a do-while loop to perform data cleaning and conversion operations on the list data of the current page, and the page is automatically incremented to indicate the next page;
[0081] S104. Update the starting index: Get the "index" field value of the last record in the list of the current page, that is, through list.get(list.size()-1).get("index"), convert the corresponding value into a string and then into a long integer, and assign it to startIndex as the starting index for the next page query;
[0082] S105. Query the next page of data: call getList(startIndex), pass in the new startIndex, and get the list of the next page;
[0083] S106, loop condition: when the size of the newly queried list is 0 (i.e., there is no more data), exit the do-while loop;
[0084] S107. Termination condition: When the list returned by getList(startIndex) is empty (or its size is zero), it means that all data has been processed and the loop ends.
[0085] The key codes are as follows:
[0086] Long startIndex=0L;
[0087]
[0088] The data processing in step S2 of this embodiment is to start 10 asynchronous tasks in parallel through the thread pool executor. Each task processes a different part of the finalList (each part has 100 data items), so as to achieve the purpose of using multithreading to accelerate the data management process. The details are as follows:
[0089] S201. Define the paging size: define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread each time (i.e., 100 data items per page);
[0090] S202. Create asynchronous tasks: Use the thread pool executor to submit multiple asynchronous tasks (a total of 10), each task calls the dealPeopleList method to process data; each task returns the processing result through a Future object (such as listFuture1, listFuture2, etc.), and the result is List <Map<String,Object> >Type, indicating the processed data list;
[0091] S203, Task Parameter Description: The input parameters of the 10 tasks are finalList (total data list), thread number and pageSize;
[0092] S204, asynchronous processing logic: Each thread independently processes a portion of the data in finalList divided by thread number and pageSize, specifically: paging, conversion or cleaning operations are performed on the data through the dealPeopleList method, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object.
[0093] The key codes are as follows:
[0094] int pageSize = 100; / / Each thread processes a part, that is, each thread processes 100 items. / / Create multiple threads for asynchronous data management. FinalList is the total data list for this query.
[0095] Future <List<Map<String,Object> >>listFuture1=executor.submit(()->dealPeopleList(finalList,1,pageSize));
[0096] Future<List<Map<String,Object>>>listFuture2=executor.submit(()->dealPeopleList(finalList,2,pageSize));
[0097] Future<List<Map<String,Object>>>listFuture3=executor.submit(()->dealPeopleList(finalList,3,pageSize));
[0098] Future<List<Map<String,Object>>>listFuture4=executor.submit(()->dealPeopleList(finalList,4,pageSize));
[0099] Future<List<Map<String,Object>>>listFuture5=executor.submit(()->dealPeopleList(finalList,5,pageSize));
[0100] Future<List<Map<String,Object>>>listFuture6=executor.submit(()->dealPeopleList(finalList,6,pageSize));
[0101] Future<List<Map<String,Object>>>listFuture7=executor.submit(()->dealPeopleList(finalList,7,pageSize));
[0102] Future<List<Map<String,Object>>>listFuture8=executor.submit(()->dealPeopleList(finalList,8,pageSize));
[0103] Future<List<Map<String,Object>>>listFuture9=executor.submit(()->dealPeopleList(finalList,9,pageSize));
[0104] Future <List<Map<String,Object> >>listFuture10=executor.submit(()->dealPeopleList(finalList,fieldList,10,pageSize));
[0105] In the above pseudo code, after a set of data is queried, multi-threaded processing is performed, and each thread only processes a fixed number of the same data. This can ensure that the end time of all threads is as close as possible, saving waiting time for subsequent processes.
[0106] In step S2 of this embodiment, each thread loops through the data management process by extracting the data of the corresponding page from the total data list list according to the page number pageIndex and the page size pageSize passed in, and processes each data item through the processOnePeople method. If the list is empty or there is no data on the current page, null is returned; otherwise, the processing result of the current page is returned. The details are as follows:
[0107] (1) Method definition: Define a public method dealPeopleList, which receives three parameters: list, pageIndex, and pageSize, and returns a List <Map<String,Object> >, indicating the data list after processing the current page; the list parameter refers to the type of List <Map<String,Object> >, indicating the total data list to be processed; the pageIndex parameter is of type int, indicating the page number currently being processed (starting from 1); the pageSize parameter is of type int, indicating the number of data items to be processed per page;
[0108] (2) Initialize the current page data list: create an empty ArrayList object currentList to store the data of the current page;
[0109] (3) Empty list check: call StringUtil.isEmptyList(list) to check whether the input list list is empty: if it is empty, it returns null directly, indicating that there is no data to process;
[0110] (4) Page range check: Check whether the size of the input list is less than (pageIndex-1)*pageSize+1: If so, it means that there is no data on the current page and returns null;
[0111] (5) Intercept the current page data, the situation is as follows:
[0112] ① The data is less than one page: If the size of the list is less than pageIndex*pageSize, it means that the current page data is less than pageSize. Use list.subList((pageIndex-1)*pageSize, list.size()) to intercept the data from (pageIndex-1)*pageSize to the end of the list and store it in currentList;
[0113] ② The data fills one page: If the size of the list is not less than pageIndex*pageSize, use list.subList((pageIndex-1)*pageSize,pageIndex*pageSize) to intercept the data from (pageIndex-1)*pageSize to pageIndex*pageSize and store it in currentList;
[0114] (6) Processing a single piece of data: Traverse each piece of data in currentList (Map<String,Object> Type), and call the processOnePeople(people) method to perform specific data cleaning and conversion operations on each data;
[0115] (7) Return result: Returns the processed currentList (which may be null or the data list of the current page).
[0116] The key codes are as follows:
[0117] / Loop through the data in the list based on the page number
[0118]
[0119] The data update in step S3 of this embodiment is to obtain the processing results of multiple asynchronous tasks through getFuture, merge the results of multiple asynchronous tasks into a list, and finally update the merged data in batches to the database, so as to achieve the purpose of aggregating the data processed by multiple threads and making it uniformly persistent; the details are as follows:
[0120] S301, get asynchronous task results: get the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type data lists are stored in list1 to list10 variables respectively; the second parameter 0 of the getFuture method may be a timeout or other configuration (the specific meaning depends on the context);
[0121] S302. Merge data lists: Use the list1.addAll() method to append the data from list2 to list10 to list1 in sequence. Finally, list1 contains the data processed by all asynchronous tasks, that is, the collection of list1 to list10.
[0122] S302. Batch update to database: call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and store the processed data persistently or perform other database operations.
[0123] In this embodiment, when updating the managed data to the database, the speed of single update or single insertion is much lower than batch processing. Therefore, batch update to the database after each page of data is fully managed can save more database operation time. The key code is as follows:
[0124] List <Map<String,Object> >list1=getFuture(listFuture1,0);
[0125] List <Map<String,Object> >list2=getFuture(listFuture2,0);
[0126] List <Map<String,Object> >list3=getFuture(listFuture3,0);
[0127] List <Map<String,Object> >list4=getFuture(listFuture4,0);
[0128] List <Map<String,Object> >list5=getFuture(listFuture5,0);
[0129] List <Map<String,Object> >list6=getFuture(listFuture6,0);
[0130] List <Map<String,Object> >list7=getFuture(listFuture7,0);
[0131] List <Map<String,Object> >list8=getFuture(listFuture8,0);
[0132] List <Map<String,Object> >list9=getFuture(listFuture9,0);
[0133] List <Map<String,Object> >list10=getFuture(listFuture10,0);
[0134] list1.addAll(list2);
[0135] list1.addAll(list3);
[0136] list1.addAll(list4);
[0137] list1.addAll(list5);
[0138] list1.addAll(list6);
[0139] list1.addAll(list7);
[0140] list1.addAll(list8);
[0141] list1.addAll(list9);
[0142] list1.addAll(list10);
[0143] batchUpdateToDB(list1); / / Batch update to the database.
[0144] Example 2:
[0145] This embodiment provides an optimization system for single-machine management of large-volume data, which includes:
[0146] The data query module is used to create a numeric index for the table. The index field is index and the index start parameter is startIndex. It processes data concurrently, obtains the index of the last record, sets startIndex = index, inserts result data in batches, and obtains data query results.
[0147] The data management module is used to obtain a set of data query results and then perform multi-threaded processing. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop.
[0148] The data update module is used to update the managed data to the database. After all the data on each page is managed, it is updated to the database in batches.
[0149] The data query module in this embodiment obtains a data list page by page through a paging query (starting from index 0), processes each page of data, and dynamically updates the query starting index (using the index value of the last record on the previous page) until there is no more data to query. The data query module includes:
[0150] The variable initialization submodule is used to define a long integer variable startIndex with an initial value of 0L, which is used to indicate the starting index of the query; and also defines an integer variable page with an initial value of 1, which is used to record the current page number;
[0151] The first query submodule is used to call the getList(startIndex) method, pass in the startIndex (initial 0), and get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record;
[0152] The data processing submodule is used to call the dealList(list) method through a do-while loop to clean and convert the list data on the current page. The page is automatically incremented to indicate the next page.
[0153] The starting index update submodule is used to obtain the "index" field value of the last record from the list of the current page, that is, through list.get(list.size()-1).get("index"), and convert the corresponding value into a string and then into a long integer, and assign it to startIndex as the starting index for the next page query;
[0154] The next page data query submodule is used to call getList(startIndex), pass in the new startIndex, and get the list of the next page;
[0155] The loop condition submodule is used to exit the do-while loop when the size of the newly queried list is 0 (that is, there is no more data);
[0156] The conditional termination submodule is used to indicate that all data has been processed and the loop ends when the list returned by getList(startIndex) is empty (or its size is zero).
[0157] The data processing module in this embodiment uses the thread pool executor to launch 10 asynchronous tasks in parallel. Each task processes a different part of the finalList (each part has 100 data items), thereby accelerating the data management process by using multithreading. The data processing module includes:
[0158] The paging size definition submodule is used to define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread each time (i.e., 100 data items per page);
[0159] The asynchronous task creation submodule is used to submit multiple asynchronous tasks (10 in total) using the thread pool executor. Each task calls the dealPeopleList method to process data. Each task returns the processing result through a Future object (such as listFuture1, listFuture2, etc.). The result is a List<T> <Map<String,Object> >Type, indicating the processed data list;
[0160] The task parameter description submodule uses 10 asynchronous tasks listFuture10, whose incoming parameters are finalList (total data list), thread number and pageSize;
[0161] The asynchronous processing logic sub-module is used for each thread to independently process a portion of the data in the finalList divided by thread number and pageSize. Specifically, the dealPeopleList method is used to page, convert or clean the data, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object.
[0162] The data update module in this embodiment obtains the processing results of multiple asynchronous tasks through getFuture, merges the results of multiple asynchronous tasks into a list, and finally updates the merged data in batches to the database, so as to achieve the purpose of aggregating the data processed by multiple threads and making it uniform and persistent. The data update module includes:
[0163] The asynchronous task result acquisition submodule is used to obtain the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type data lists are stored in list1 to list10 variables respectively; the second parameter 0 of the getFuture method may be a timeout or other configuration (the specific meaning depends on the context);
[0164] The data list merging submodule is used to append the data from list2 to list10 to list1 in sequence using the list1.addAll() method. Finally, list1 contains the data after all asynchronous tasks are processed, that is, the collection of list1 to list10.
[0165] The batch update submodule is used to call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and persist the processed data or perform other database operations.
[0166] Example 3:
[0167] This embodiment also provides an electronic device, including: a memory and at least one processor;
[0168] wherein the memory stores computer-executable instructions;
[0169] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the optimization method for single-machine management of large-volume data in any embodiment of the present invention.
[0170] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.
[0171] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can also include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.
[0172] Example 4:
[0173] This embodiment further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor and cause the processor to execute the optimization method for single-machine management of large amounts of data according to any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, wherein the storage medium stores software program code that implements the functions of any of the above-described embodiments, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.
[0174] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.
[0175] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.
[0176] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.
[0177] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.
[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. An optimization method for single-machine management of large-volume data, characterized in that: This method uses indexes to perform efficient database paging queries. When querying by page, the maximum code of the previous page is used as the starting parameter for the next page. The managed data is stored in batches to improve data update efficiency. The details are as follows: Data query: Create a numeric index for the table, with the index field as index and the index start parameter as startIndex. Process data concurrently, obtain the index of the last entry, set startIndex = index, insert result data in batches, and obtain data query results. Data management: After obtaining a set of data query results, multi-threaded processing is performed. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop. Data update: When updating the managed data to the database, wait until all the data on each page is managed and then update it to the database in batches.
2. The optimization method for single-machine management of large-volume data according to claim 1 is characterized in that: Data query is done through paging query, obtaining the data list page by page, processing each page of data, and dynamically updating the starting index of the query until there is no more data to query; the details are as follows: Initialize variables: define a long integer variable startIndex with an initial value of 0L to indicate the starting index of the query; also define an integer variable page with an initial value of 1 to record the current page number; First query: call the getList(startIndex) method, pass in startIndex, and get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record; Loop data processing: call the dealList(list) method through a do-while loop to clean and convert the list data on the current page. The page is incremented to indicate the next page. Update the starting index: Get the "index" field value of the last record from the list on the current page, that is, through list.get(list.size()-1).get("index"), convert the corresponding value to a string and then to a long integer, and assign it to startIndex as the starting index for the next page query; Query the next page of data: call getList(startIndex), pass in the new startIndex, and get the list of the next page; Loop condition: When the size of the newly queried list is 0, exit the do-while loop; Termination condition: When the list returned by getList(startIndex) is empty, it means that all data has been processed and the loop ends.
3. The optimization method for single-machine management of large-volume data according to claim 1 is characterized in that: Data processing is done by launching 10 asynchronous tasks in parallel through the thread pool executor. Each task processes a different part of the finalList, achieving the purpose of using multithreading to accelerate the data management process. The details are as follows: Define the paging size: define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread at a time; Create asynchronous tasks: Use the thread pool executor to submit multiple asynchronous tasks. Each task calls the dealPeopleList method to process data. Each task returns the processing result through the Future object, and the result is a List. <Map<String,Object> >Type, indicating the processed data list; Task parameter description: The input parameters of the 10 asynchronous tasks are fieldList, thread number and pageSize; Asynchronous processing logic: Each thread independently processes a portion of the data in finalList divided by thread number and pageSize. Specifically, the dealPeopleList method is used to page, convert, or clean the data, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object.
4. The optimization method for single-machine management of large-volume data according to claim 1 is characterized in that: Each thread loop manages data by intercepting the corresponding page data from the total data list list based on the page number pageIndex and page size pageSize passed in, and processing each data through the processOnePeople method; if the list is empty or there is no data on the current page, null is returned; otherwise, the processing result of the current page is returned; the details are as follows: Method definition: define a public method dealPeopleList, which receives three parameters: list, pageIndex and pageSize, and returns a List <Map<String,Object> >, indicating the data list after processing the current page; the list parameter refers to the type of List <Map<String,Object> >, indicating the total data list to be processed; the pageIndex parameter is of type int, indicating the page number currently being processed; the pageSize parameter is of type int, indicating the number of data items to be processed per page; Initialize the current page data list: create an empty ArrayList object currentList to store the data of the current page; Empty list check: call StringUtil.isEmptyList(list) to check whether the input list list is empty: if it is empty, it returns null directly, indicating that there is no data to process; Page range check: Check whether the size of the input list is less than (pageIndex-1)*pageSize+1: If so, it means there is no data on the current page, and null is returned; Intercept the current page data, the situation is as follows: The data is less than one page: If the size of the list is less than pageIndex*pageSize, it means that the current page data is less than pageSize. Use list.subList((pageIndex-1)*pageSize, list.size()) to intercept the data from (pageIndex-1)*pageSize to the end of the list and store it in currentList; The data fills one page: If the size of the list is not less than pageIndex*pageSize, use list.subList((pageIndex-1)*pageSize, pageIndex*pageSize) to intercept the data from (pageIndex-1)*pageSize to pageIndex*pageSize and store it in currentList; Processing a single piece of data: traverse each piece of data in currentList (Map<String,Object> Type), and call the processOnePeople(people) method to perform specific data cleaning and conversion operations on each data; Return result: Returns the processed currentList.
5. The optimization method for single-machine management of large-volume data according to any one of claims 1 to 4, characterized in that: Data update is to obtain the processing results of multiple asynchronous tasks through getFuture, merge the results of multiple asynchronous tasks into a list, and finally update the merged data in batches to the database, so as to achieve the purpose of aggregating the data processed by multiple threads and making it persisted uniformly; the details are as follows: Get asynchronous task results: Get the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type of data list, stored in list1 to list10 variables respectively; Among them, the second parameter 0 of the getFuture method is the timeout period or other configuration; Merge data lists: Use the list1.addAll() method to append the data from list2 to list10 to list1 in sequence. Finally, list1 contains the data processed by all asynchronous tasks, that is, the collection of list1 to list10. Batch update to the database: Call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and store the processed data persistently or perform other database operations.
6. An optimization system for single-machine management of large-volume data, characterized by: The system includes: The data query module is used to create a numeric index for the table. The index field is index and the index start parameter is startIndex. It processes data concurrently, obtains the index of the last record, sets startIndex = index, inserts result data in batches, and obtains data query results. The data management module is used to obtain a set of data query results and then perform multi-threaded processing. Each thread only processes a fixed number of identical data items, and each thread manages the data in a loop. The data update module is used to update the managed data to the database. After all the data on each page is managed, it is updated to the database in batches.
7. The optimization system for single-machine management of large-volume data according to claim 6 is characterized in that: The data query module obtains the data list page by page through paging query, processes each page of data, and dynamically updates the starting index of the query until there is no more data to query; The data query module includes: The variable initialization submodule is used to define a long integer variable startIndex with an initial value of 0L, which is used to indicate the starting index of the query; and also defines an integer variable page with an initial value of 1, which is used to record the current page number; The first query submodule is used to call the getList(startIndex) method, pass in the startIndex, and get the data list list of the first page; where list is a list containing multiple Map<String,Object> A collection of, each Map represents a data record; The data processing submodule is used to call the dealList(list) method through a do-while loop to clean and convert the list data on the current page. The page is automatically incremented to indicate the next page. The starting index update submodule is used to obtain the "index" field value of the last record from the list of the current page, that is, through list.get(list.size()-1).get("index"), and convert the corresponding value into a string and then into a long integer, and assign it to startIndex as the starting index for the next page query; The next page data query submodule is used to call getList(startIndex), pass in the new startIndex, and get the list of the next page; The loop condition submodule is used to exit the do-while loop when the size of the newly queried list is 0; The conditional termination submodule is used to indicate that all data has been processed and the loop ends when the list returned by getList(startIndex) is empty.
8. The optimization system for single-machine management of large-volume data according to claim 6 or 7, characterized in that: The data processing module uses the thread pool executor to start 10 asynchronous tasks in parallel. Each task processes a different part of the finalList, thereby accelerating the data management process by using multithreading. The data processing module includes: The paging size definition submodule is used to define an integer variable pageSize with a value of 100, which represents the number of data items processed by each thread each time; The asynchronous task creation submodule is used to submit multiple asynchronous tasks using the thread pool executor. Each task calls the dealPeopleList method to process data. Each task returns the processing result through the Future object, and the result is a List <Map<String,Object> >Type, indicating the processed data list; The task parameter description submodule is used for the 10 asynchronous tasks listFuture10. The input parameters are finalList, thread number and pageSize; The asynchronous processing logic submodule is used by each thread to independently process a portion of the data in finalList divided by thread number and pageSize. Specifically, the dealPeopleList method is used to paginate, convert, or clean the data, and the execution result of the asynchronous task, that is, the processed data list, is obtained through the Future object. The data update module obtains the processing results of multiple asynchronous tasks through getFuture, merges the results of multiple asynchronous tasks into a list, and finally updates the merged data in batches to the database, achieving the purpose of aggregating and persisting the data processed by multiple threads. The data update module includes: The asynchronous task result acquisition submodule is used to obtain the execution results of 10 asynchronous tasks listFuture1 to listFuture10 through the getFuture method. The result of each task is a List <Map<String,Object> >Type of data list, stored in list1 to list10 variables respectively; Among them, the second parameter 0 of the getFuture method is the timeout period or other configuration; The data list merging submodule is used to append the data from list2 to list10 to list1 in sequence using the list1.addAll() method. Finally, list1 contains the data after all asynchronous tasks are processed, that is, the collection of list1 to list10. The batch update submodule is used to call the batchUpdateToDB(list1) method to batch update the merged data list list1 to the database, and persist the processed data or perform other database operations.
9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the optimization method for single-machine management of large-volume data as described in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the optimization method for single-machine management of large-volume data as described in any one of claims 1 to 5.