Sorting method based on mass data

Abstract

The invention discloses a sequencing method based on mass data, which belongs to the technical field of data sequencing, and comprises the following steps: data splitting: splitting a data source M to be sequenced into N files; block sorting amp is carried out; extracting: sorting each file according to the sorting fields, and extracting the first X pens of each file; merging and sorting: integrating the sorted and extracted data together, sorting, and storing the first X rows into a result set; and displaying: displaying the result set data in sequence. According to the method, the success of sorting processing can be ensured, and the conditions of memory overflow errors and system crash or extremely slow change are avoided; in the aspect of performance, the processing and sorting quantity is greatly reduced, computing resources can be saved, the CPU utilization rate can be reduced, the response time can be shortened, and remarkable delay generated in the sorting process is avoided; in the aspects of maintenance and management, the system architecture can be simplified, and the level of a fault-tolerant mechanism can be reduced, so that the complexity of system design and maintenance is reduced.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of data sorting, and specifically to a sorting method based on massive data. Background Art

[0002] In factories, it is often accompanied by the sorting, viewing, processing and analysis of massive data. The traditional technology is to put massive data into a unified container (file / computer memory, etc.), sort all data in the computer memory, generate a complete and ordered result set, and display it. There are the following defects and shortcomings:

[0003] Memory limitations: Massive amounts of data may far exceed the physical memory capacity of the system. Trying to load all the data into memory will result in an out-of-memory error, and the system may crash or become extremely slow.

[0004] Performance issues: Even with sufficient physical memory, processing and sorting large amounts of data can consume a lot of computing resources, resulting in high CPU usage, long response times, and even significant delays during the sorting process.

[0005] Expensive: Memory large enough to store massive amounts of data requires expensive hardware, which increases the total cost of ownership (TCO).

[0006] Maintenance and management complexity: Large-scale in-memory data management requires a more complex system architecture and a higher level of fault tolerance, which increases the complexity of system design and maintenance. Summary of the invention

[0007] The technical solution of the present invention aims at the technical problem that the existing technical solutions are too single, and provides a solution that is significantly different from the existing technologies. It mainly provides a sorting method based on massive data, which is used to solve the technical problems of the traditional data sorting methods proposed in the above background technology, such as memory limitations, performance issues, high costs, maintenance and management complexity.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A sorting method based on massive data includes the following steps:

[0010] Data splitting: Split the M data sources to be sorted into N files;

[0011] Block sorting & extraction: Sort each file by sorting field and extract the first X records of each file;

[0012] Merge sort: Integrate the extracted data after sorting, sort them, and take the first X rows and store them in the result set;

[0013] Display: The result set data is displayed in sequence.

[0014] Furthermore, in data splitting, M and N are both positive integers, the amount of data in each file is M / N±100, and M / N±100 is less than or equal to the threshold of the amount of data that the system can process at one time.

[0015] Furthermore, in the block sorting & extraction, it is first determined whether the number of data A that the user wants to view is less than or equal to the set value B. The set value is determined according to the threshold value / N of the number of data that the system processes at one time; when A is less than or equal to B, X takes the value of A; when A is greater than B, X takes the value of A / n, where n is a positive integer greater than or equal to 2, and X is less than or equal to B.

[0016] Furthermore, the formation of the result set includes the following steps:

[0017] (1) Perform the first merge sort and store X rows of data into the result set;

[0018] (2) Determine whether the number of data C in the result set reaches the number of data A that the user wants to view. If not, proceed to step (3); if reached, proceed to step (6);

[0019] (3) Update the data extracted from each document: In the data extracted from each document, identify and remove the data that has been stored in the result set, and extract a corresponding number of data that are ranked first from the remaining data of the corresponding document to form new extracted data;

[0020] (4) Integrate and sort the data again, take the first X rows and store them in the result set, and update the number of data in the result set C;

[0021] (5) Repeat steps (2) to (4) until the number of data C in the result set reaches the number of data A that the user wants to view;

[0022] (6) Form a result set.

[0023] As an implementation mode, the display includes the following steps:

[0024] a1. Determine whether the number of data in the result set C is less than or equal to the number of rows displayed on the page at one time D. If C is less than or equal to D, proceed to step b; if C is greater than D, proceed to step c;

[0025] b1. Display row C of result set data and end dynamic loading;

[0026] c1. Display the first D rows of data in the result set;

[0027] d1. Remove row D from the result set and update row C to row CD.

[0028] e1. Repeat steps a1-d1.

[0029] As another embodiment, the display includes the following steps:

[0030] a2. Determine whether the number of data in the result set C is less than or equal to the number of rows displayed on the page at one time D. If C is less than or equal to D, proceed to step b; if C is greater than D, proceed to step c;

[0031] b2. Display row C of result set data and end dynamic loading;

[0032] c2. Display the first D rows of data in the result set;

[0033] d2. Remove the first D-1 rows of data from the result set, and update C to C-D+1;

[0034] e2. Repeat steps a2-d2.

[0035] Furthermore, it is displayed after each merge sort.

[0036] Preferably, the file type is a Parquet file.

[0037] Preferably, multiple threads or multiple processes are used to read and write files in parallel, and asynchronous reading and writing methods are used to avoid blocking; N files are sorted in parallel.

[0038] Preferably, parallel processing is performed during the merging process.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention splits the data source into multiple files, sorts each file in parallel, extracts the pre-order data, integrates the extracted data and sorts them uniformly, and can obtain the correct data in the data source that is sorted first. Compared with the traditional one-time sorting method, the present invention cuts the massive data into multiple blocks for sorting, which can ensure the success of the sorting process and avoid memory overflow errors and system crashes or extremely slow conditions; in terms of performance, it greatly reduces the number of processes and sorts, which can save computing resources, reduce CPU usage, shorten response time, and avoid significant delays in the sorting process; in terms of cost, it does not require expensive hardware to provide a large enough memory, which is conducive to reducing costs; in terms of maintenance and management, it can simplify the system architecture and reduce the level of fault tolerance mechanisms, thereby reducing the complexity of system design and maintenance.

[0041] (2) The present invention takes into account the amount of data that the user wants to view in the sorting method, determines the amount of data extracted after each sorting according to the user's needs, and uses the user's needs as the end condition for the formation of the result set, so as to save unnecessary data processing work. In addition, when extracting data after each sorting, the number of extracted data is also limited to a set value based on the threshold value of the number of data to be processed at one time by the system / the number of files, and the amount of extracted data is controlled below the set value, ensuring that the amount of data processed after integration and unified sorting is within the threshold range of the number of data to be processed at one time by the system, which is conducive to preventing excessive data processing from having an adverse effect on the system sorting process.

[0042] (3) When the amount of data that a customer wants to view is large, the present invention divides the amount of data that the customer wants to view into at least two equal parts, and each part is subjected to the steps of extracting data from each file, integrating and sorting the data, and storing the extracted data into a result set. This is performed in batches, which can ensure the uniformity of data processing in different batches.

[0043] (4) The present invention takes into account the number of rows displayed at one time on the page, implements dynamic loading of scroll bars, is easy to use, and can display the result set data as quickly as possible. In the optimization scheme, each scroll loading displays the last row of the previous display and the newly added D-1 row, which is convenient for determining the connection order before and after and avoiding errors and omissions in the middle.

[0044] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A step diagram of a mass data sorting method provided by the present invention;

[0046] Figure 2 A flow chart for forming a result set in the present invention;

[0047] Figure 3 A flow chart showing a solution of the present invention;

[0048] Figure 4 This is a diagram of the processing results of each step in Example 6 of the present invention. DETAILED DESCRIPTION

[0049] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.

[0050] Please refer to the attached Figure 1, a sorting method based on massive data, comprising the following steps:

[0051] Data splitting: Split the M data sources to be sorted into N files; M and N are both positive integers, the amount of data in each file is M / N±100, and M / N±100 is less than or equal to the threshold of the amount of data that the system can process at one time.

[0052] Block sorting & extraction: Sort each file by sorting field and extract the first X records of each file.

[0053] First, determine whether the amount of data A that the user wants to view is less than or equal to the set value B. The set value is determined according to the threshold value / N of the amount of data that the system processes at one time; when A is less than or equal to B, X takes the value of A; when A is greater than B, X takes the value of A / n, where n is a positive integer greater than or equal to 2, and X is less than or equal to B.

[0054] Merge sort: Integrate the extracted data after sorting, sort them, and take the first X rows and store them in the result set;

[0055] Please refer to the attached Figure 2 , the formation of the result set includes the following steps:

[0056] (1) Perform the first merge sort and store X rows of data into the result set;

[0057] (2) Determine whether the number of data C in the result set reaches the number of data A that the user wants to view. If not, proceed to step (3); if reached, proceed to step (6);

[0058] (3) Update the data extracted from each document: In the data extracted from each document, identify and remove the data that has been stored in the result set, and extract a corresponding number of data that are ranked first from the remaining data of the corresponding document to form new extracted data;

[0059] (4) Integrate and sort the data again, and take the first X rows and store them in the result set;

[0060] (5) Repeat steps (2) to (4) until the number of data C in the result set reaches the number of data A that the user wants to view;

[0061] (6) Form a result set.

[0062] Display: The result set data is displayed in sequence.

[0063] Please refer to the attached Figure 3 , when displayed, the following steps are included:

[0064] a. Determine whether the number of data in the result set C is less than or equal to the number of rows displayed on the page at one time D. If C is less than or equal to D, proceed to step b; if C is greater than D, proceed to step c;

[0065] b. Display row C of result set data and end dynamic loading;

[0066] c. Display the first D rows of data in the result set;

[0067] d. Remove row D from the result set and update row C to row CD;

[0068] Alternatively, the first D-1 rows of data displayed in the result set are removed, and C is updated to C-D+1;

[0069] e. Repeat steps ad.

[0070] Embodiment 1: 1000W (W means ten thousand) records of data need to be sorted, and a computer page displays 100 records at a time, and the user wants to view the first 100 records.

[0071] The sorting method of the above data is:

[0072] S1. Data Splitting

[0073] Split the massive data into:

[0074] a. Data source: 10 million records;

[0075] b. Split ratio: write one document for every 50,000 records;

[0076] c. File type: Parquet file;

[0077] d. Generate 200 parquet files.

[0078] There is a critical value for the time consumed by computer systems for reading and writing. After testing, we found that 5w / copy is a good critical value, which ensures low overall time consumption.

[0079] Use multiple threads or multiple processes to read and write files in parallel, and use asynchronous reading and writing methods to avoid blocking.

[0080] S2, block sorting & extraction

[0081] 1) Sort each file by sort field

[0082] a. Sort field: lot;

[0083] b. Sorting method: ascending;

[0084] Sort the split files in parallel.

[0085] 2) Data extraction

[0086] a. Number of visible lines on the page: 100 lines;

[0087] b. After sorting, take the first 100 records of each file (determined by the amount of data the user wants to view), ensuring that the first 100 records of the expected sorting of the total data are included.

[0088] Advantages of block sorting: When the amount of data to be sorted at one time is too large, it will be limited by the system hardware and fail to sort successfully; although block sorting will be slower, it can ensure the success of the sorting.

[0089] S3, Merge Sort

[0090] 1) Integrate the extracted data after the above sorting into the control (dataTable), with a total of 20,000 records (200*100);

[0091] Parallel processing is also possible during the merge (combination) process.

[0092] 2) Sort the integrated 2w records in ascending order in dataTable (using system memory);

[0093] 3) After sorting, take the first 100 rows (determined by the amount of data the user wants to view) to form a result set.

[0094] S4. Display

[0095] Display the result set data directly.

[0096] On the whole, the present invention uses a solid-state drive (SSD) to significantly improve the reading and writing speed, and uses a larger buffer to reduce the number of disk I / O times, ensuring that files are stored continuously to reduce seek time; uses memory-mapped files to effectively map files to memory, thereby avoiding frequent file reading and writing operations; uses a library specifically used to process large-scale data, such as Sort-merge or other optimized external sorting tools; uses a GPU for acceleration (hardware acceleration); compresses data without losing information, which can reduce the amount of data processed.

[0097] Embodiment 2: This embodiment differs from Embodiment 1 in that:

[0098] To sort 1000W (W means ten thousand) records, the computer page displays 100 records at a time, and the user wants to view the first 200 records.

[0099] In step S2, when extracting data, the first 200 records of each file are taken after sorting (when the number of data the user wants to view is less than or equal to 250, this data is determined based on the number of data the user wants to view, and the set value is determined based on the threshold of the number of data processed by the system at one time / the number of files, that is, 5W / 200).

[0100] In step S3, there are a total of 40,000 records (200*200) of integrated data; after sorting, the first 200 rows are taken to form a result set.

[0101] Step S4, dynamic loading and display: first take the first 100 rows for display, then scroll the scroll bar to load, and then take the 101st to 200th rows for display.

[0102] The rest is the same as the first embodiment.

[0103] Embodiment 3: This embodiment differs from Embodiment 1 in that:

[0104] To sort 1000W (W means ten thousand) records, the computer page displays 100 records at a time, and the user wants to view the first 400 records.

[0105] In step S2, when extracting data, the first 200 records of each file are taken after sorting (this data is obtained by dividing the data that the user wants to view as equally as possible, and is less than or equal to the set value of 250, which is determined by the threshold of the number of data processed by the system at one time / the number of files, that is, 5W / 200=250);

[0106] In step S3, there are a total of 40,000 records (200*200) of integrated data; after sorting, the first 200 rows are first taken and stored in the result set;

[0107] Step S4, updating the data extracted from each document: in the data extracted from each document, identifying and removing the data that has been stored in the result set, and supplementing the corresponding number of first few rows of data from the remaining data of the corresponding document (which has been sorted) to form new extracted data;

[0108] Step S5, integrate and sort the data again: Integrate the newly extracted data in step S4, a total of 40,000 records (200*200); after sorting, take the first 200 rows and add them to the result set to form the final result set;

[0109] Step S6, dynamic loading and display: the data in the result set is loaded in order by scrolling the scroll bar, and 100 rows are displayed each time until the data in the result set is displayed. The last display may be less than 100 rows.

[0110] The rest is the same as the first embodiment.

[0111] Embodiment 4: This embodiment differs from Embodiment 3 in that:

[0112] In this embodiment, in step S6, when dynamically loading and displaying, 100 rows are displayed for the first time, and then the last row displayed in the previous display and the newly added 99 rows are displayed each time after scrolling and loading, and the last row displayed in the previous display and the remaining data are displayed for the last time. Specifically: 1-100 rows; 100-199 rows; 199-298 rows; 298-397 rows; 397-400 rows. This setting makes it easy to determine the connection order before and after, and avoid errors and omissions in the middle.

[0113] The rest is the same as in the third embodiment.

[0114] Embodiment 5: This embodiment differs from Embodiment 4 in that:

[0115] In this embodiment, while performing steps S4 and S5, the result set data in step S3 is first displayed to cope with the situation where the user wants to view a large amount of data, thereby reducing the user's waiting time.

[0116] The rest is the same as that of the fourth embodiment.

[0117] Embodiment 6: This embodiment differs from Embodiment 1 in that:

[0118] In this embodiment, if Figure 4 As shown, take 15 data items as an example, and the result shows 4 data items. The sorting method is as follows Figure 2 shown.

[0119] S1. Data splitting: divide the 15 data into 3 files with 5 data in each file;

[0120] S2, block sorting & extraction: sort the three files separately and extract the first four records;

[0121] S3, merge sort: Integrate the data extracted from the three files to obtain 12 records of data, sort them, and take the first 4 rows and store them in the result set;

[0122] S4, display: result set data display.

[0123] The rest is the same as the first embodiment.

[0124] In summary, the present invention can effectively utilize limited memory resources to sort massive data, thereby improving system stability and data processing capacity.

[0125] The above is an exemplary description of the present invention in combination with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A sorting method based on massive data, characterized by: The steps include: Data splitting: Split the M data sources to be sorted into N files; Block sorting & extraction: Sort each file by sorting field and extract the first X records of each file; Merge sort: Integrate the extracted data after sorting, sort them, and take the first X rows and store them in the result set; Display: The result set data is displayed in sequence.

2. A method for sorting based on massive data according to claim 1, characterized in that: In data splitting, M and N are both positive integers, the amount of data in each file is M / N±100, and M / N±100 is less than or equal to the threshold of the amount of data that the system can process at one time.

3. The method for sorting massive data according to claim 1, characterized in that: In block sorting & extraction, first determine whether the number of data A that the user wants to view is less than or equal to the set value B. The set value is determined based on the threshold value / N of the number of data that the system processes at one time; when A is less than or equal to B, X is A; when A is greater than B, X is A / n, where n is a positive integer greater than or equal to 2, and X is less than or equal to B.

4. The method for sorting massive data according to claim 3, characterized in that: The formation of the result set includes the following steps: (1) Perform the first merge sort and store X rows of data into the result set; (2) Determine whether the number of data C in the result set reaches the number of data A that the user wants to view. If not, proceed to step (3); if reached, proceed to step (6); (3) Update the data extracted from each document: In the data extracted from each document, identify and remove the data that has been stored in the result set, and extract a corresponding number of data that are ranked first from the remaining data of the corresponding document to form new extracted data; (4) Integrate and sort the data again, and take the first X rows and store them in the result set; (5) Repeat steps (2) to (4) until the number of data C in the result set reaches the number of data A that the user wants to view; (6) Form a result set.

5. The method for sorting massive data according to claim 1, characterized in that: The demonstration includes the following steps: a1. Determine whether the number of data in the result set C is less than or equal to the number of rows displayed on the page at one time D. If C is less than or equal to D, proceed to step b; if C is greater than D, proceed to step c; b1. Display row C of result set data and end dynamic loading; c1. Display the first D rows of data in the result set; d1. Remove row D from the result set and update row C to row CD. e1. Repeat steps a1-d1.

6. The method for sorting massive data according to claim 1, characterized in that: The demonstration includes the following steps: a2. Determine whether the number of data in the result set C is less than or equal to the number of rows displayed on the page at one time D. If C is less than or equal to D, proceed to step b; if C is greater than D, proceed to step c; b2. Display row C of result set data and end dynamic loading; c2. Display the first D rows of data in the result set; d2. Remove the first D-1 rows of data from the result set, and update C to C-D+1; e2. Repeat steps a2-d2.

7. The method for sorting massive data according to claim 1, characterized in that: Displayed after each merge sort.

8. The method for sorting massive data according to claim 1, characterized in that: The file type is Parquet file.

9. The method for sorting massive data according to claim 1, characterized in that: Use multiple threads or multiple processes to read and write files in parallel, and use asynchronous reading and writing methods to avoid blocking; Sort N files in parallel.

10. A method for sorting based on massive data according to claim 1, characterized in that: Parallel processing is performed during the merge process.

Images