Data sorting method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of data processing, and provides a data sorting method and device, electronic equipment and a storage medium, and the data sorting method comprises the steps: converting a to-be-sorted one-dimensional array into a matrix; dividing the matrix into a plurality of sub-matrixes, and determining reference elements in each sub-matrix; performing partition operation on each sub-matrix based on a single instruction multiple data stream (SIMD) instruction set to obtain a first partition and a second partition of each sub-matrix; sorting the data elements in the first partition and the second partition of each sub-matrix to obtain a target sub-matrix after sorting of each sub-matrix is completed; and recombining the target sub-matrixes of the plurality of sub-matrixes, and converting the recombined target matrix into an ordered target one-dimensional array. The large-scale data sets can be efficiently sorted, and the processing efficiency of the large-scale data sets is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data sorting method, device, electronic device and storage medium. Background Art

[0002] With the continuous advancement of information technology, large-scale data set processing has become a core demand in many fields. In the fields of government affairs, industry, finance, transportation, telecommunications, and spatial geography, big data has been widely used as the underlying support for industry solutions such as smart cities, smart government affairs, smart parks, and smart public security. For example, in the field of urban management, the use of big data can realize intelligent transportation, environmental monitoring, urban planning, and intelligent security. Through big data analysis, the time of vehicle congestion on a road section can be predicted, and plans to alleviate traffic congestion can be formulated. The processing capacity of large-scale data sets directly determines the innovation speed and business efficiency in these fields.

[0003] When processing large-scale data sets, the traditional quick sort algorithm has high time efficiency. However, due to the large size of the data set, the required memory space may far exceed the computer's memory capacity, resulting in frequent disk IO operations during the sorting process, further reducing the sorting efficiency.

[0004] In view of this, how to efficiently sort large-scale data sets and improve the processing efficiency of large-scale data sets is a technical problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present application provide a data sorting method, device, electronic device and storage medium, which can efficiently sort large-scale data sets and improve the processing efficiency of large-scale data sets.

[0006] In a first aspect, an embodiment of the present application provides a data sorting method, comprising:

[0007] Convert the one-dimensional array to be sorted into a matrix;

[0008] Dividing the matrix into a plurality of sub-matrices, and determining a reference element within each of the sub-matrices;

[0009] Performing a partition operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices, wherein the data elements in the first partition are not greater than a reference element within the sub-matrix, and the data elements in the second partition are greater than the reference element within the sub-matrix;

[0010] Sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices has been sorted;

[0011] The target sub-matrices of the plurality of sub-matrices are reorganized, and the reorganized target matrix is ​​converted into an ordered target one-dimensional array.

[0012] In a possible implementation manner of the first aspect, the step of performing a partition operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices includes:

[0013] Loading the data elements in the submatrix into a SIMD register by a SIMD load instruction;

[0014] Comparing the data element in the SIMD register with the reference element by a SIMD comparison instruction;

[0015] Based on the comparison result, the data elements in the submatrix are logically partitioned by SIMD conditional selection instructions, data elements not greater than the reference element are selected to a first partition, and data elements greater than the reference element are selected to a second partition;

[0016] The partitioning result is stored back into the memory through the SIMD storage instruction, thereby completing the partitioning of the sub-matrix.

[0017] In a possible implementation manner of the first aspect, the step of sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices is sorted includes:

[0018] respectively determining whether the number of data elements in the first partition and the second partition of the submatrix exceeds a preset number threshold;

[0019] If the number of data elements in a partition exceeds the preset number threshold, recursively sorting the data elements in the partition;

[0020] If the number of data elements in the partition does not exceed the preset number threshold, the data elements in the partition are simply sorted;

[0021] The sorted first partitions and the second partitions are merged to obtain a target sub-matrix of the sub-matrix.

[0022] In a possible implementation manner of the first aspect, the step of recursively sorting the data elements in the partition includes:

[0023] Dividing the partition into a plurality of matrix units, and determining a reference element within each of the matrix units;

[0024] Performing a partition operation on each of the matrix units based on the SIMD instruction set to obtain a first sub-partition and a second sub-partition of each of the matrix units, wherein a data element in the first sub-partition is not greater than a reference element inside the matrix unit, and a data element in the second sub-partition is greater than a reference element inside the matrix unit;

[0025] Sorting the data elements in the first sub-partition and the second sub-partition of each of the matrix units to obtain a target matrix unit after the matrix units are sorted;

[0026] The target matrix units of the plurality of matrix units are reorganized to obtain sorted partitions.

[0027] In a possible implementation manner of the first aspect, the step of dividing the matrix into a plurality of sub-matrices and determining a reference element within each of the sub-matrices includes:

[0028] Dividing the matrix based on rows or columns of the matrix, and determining a row or a column in the matrix as a submatrix;

[0029] A reference element for each row or column in the matrix is ​​determined.

[0030] In a possible implementation manner of the first aspect, the step of dividing the matrix into a plurality of sub-matrices and determining a reference element within each of the sub-matrices includes:

[0031] Dividing the matrix into a plurality of matrix blocks, each matrix block being a sub-matrix;

[0032] A reference element in each of the sub-matrices is determined.

[0033] In a possible implementation manner of the first aspect, the step of determining a reference element within each of the sub-matrices includes:

[0034] Determine the data element at the center position of the sub-matrix as the reference element inside the sub-matrix;

[0035] Alternatively, a data element is randomly selected from the sub-matrix and determined as a reference element within the sub-matrix.

[0036] In a second aspect, an embodiment of the present application provides a data sorting device, including:

[0037] A matrix processing unit, used for converting the one-dimensional array to be sorted into a matrix;

[0038] A reference determination unit, used to divide the matrix into a plurality of sub-matrices and determine a reference element within each of the sub-matrices;

[0039] A matrix partitioning unit, configured to perform a partitioning operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices, wherein the data elements in the first partition are not greater than a reference element within the sub-matrix, and the data elements in the second partition are greater than the reference element within the sub-matrix;

[0040] A sorting unit, used for sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices has been sorted;

[0041] The reorganization unit is used to reorganize the target sub-matrices of the plurality of sub-matrices and convert the reorganized target matrix into an ordered target one-dimensional array.

[0042] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data sorting method as described in the first aspect above is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data sorting method as described in the first aspect above is implemented.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the data sorting method described in the first aspect above.

[0045] In the embodiment of the present application, by converting the one-dimensional array to be sorted into a matrix, dividing the matrix into several sub-matrices, the amount of data processed at a single time can be effectively reduced, and then the reference elements inside each sub-matrix are determined, and each sub-matrix is ​​partitioned in parallel based on the single instruction multiple data stream SIMD instruction set to obtain the first partition and the second partition of each sub-matrix, which can accelerate the sorting process, sort the data elements in the first partition and the second partition of each sub-matrix, and obtain the target sub-matrix after each sub-matrix is ​​sorted, reorganize the target sub-matrix of the several sub-matrices, and convert the reorganized target matrix into an ordered target one-dimensional array. The present application scheme converts the one-dimensional array corresponding to the large-scale data set into a matrix, and divides the sub-matrix into sub-matrices for sorting respectively, which can reduce memory usage and memory access delay, help avoid frequent disk IO operations due to insufficient memory when processing large-scale data sets, and make the sorting process more localized, and then combine the SIMD instruction set to partition the matrix, and use the SIMD characteristics of modern processors to improve the efficiency of parallel processing of multiple data elements, accelerate the key operations in the sorting process, and achieve efficient sorting of large-scale data sets, thereby improving the processing efficiency of large-scale data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 is a flowchart of the implementation of the data sorting method provided in the embodiment of the present application;

[0048] Figure 2 This is a specific implementation flow chart of step S103 in the data sorting method provided in the embodiment of the present application;

[0049] Figure 3 This is a specific implementation flow chart of step S104 in the data sorting method provided in the embodiment of the present application;

[0050] Figure 4 It is a specific implementation flow chart of recursively sorting data elements in a partition in the data sorting method provided in the embodiment of the present application;

[0051] Figure 5 is a structural block diagram of a data sorting device provided in an embodiment of the present application;

[0052] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0054] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0055] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0056] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0057] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0058] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0059] As an example but not a limitation, the data sorting method provided in the embodiment of the present application can be applied to smart devices and servers for various types of data that need to perform data sorting. The smart devices can specifically include mobile phones, tablet computers, laptops, ultra-mobile personal computers (UMPCs), desktop computers, and other smart devices that can process data. The embodiment of the present application does not impose any restrictions on the specific types of smart devices.

[0060] Figure 1 The implementation process of the data sorting method provided in the embodiment of the present application is shown. In the embodiment of the present application, the execution end is an electronic device, and the method flow includes steps S101 to S105. The specific implementation principle of each step is as follows:

[0061] Step S101: convert the one-dimensional array to be sorted into a matrix.

[0062] In the data processing process, large-scale data sets are stored in the form of a one-dimensional data group data structure. The one-dimensional array includes a large number of data elements to be sorted. Directly sorting the one-dimensional array may become inefficient due to the large size of the data. In order to improve the performance of data sorting, the embodiment of the present application converts the one-dimensional array to be sorted into a matrix. Through such a conversion, the locality and parallelism of matrix operations can be used to accelerate the sorting process.

[0063] In the embodiment of the present application, the length of the one-dimensional array must match the matrix dimension, and the conversion process involves determining the number of rows and columns of the two-dimensional matrix. In addition, converting the one-dimensional array to a two-dimensional matrix requires ensuring that the data type of the converted two-dimensional matrix is ​​consistent with the data type of the original one-dimensional array. For example, if the one-dimensional array is an integer array, then the two-dimensional matrix should also be an integer matrix.

[0064] In this embodiment, the one-dimensional array to be sorted is converted into a two-dimensional matrix, and the number of columns of the two-dimensional matrix needs to be determined. According to the number of columns and the length of the one-dimensional array, the number of rows of the two-dimensional matrix is ​​determined. If the length of the one-dimensional array cannot divide the number of columns, the last row can be supplemented by a fill value.

[0065] For example, for a one-dimensional array of length N, set the number of columns of the two-dimensional matrix to n, calculate the number of rows of the two-dimensional matrix m = N / n, and use the upward integer division method. Map the index of the data element in the one-dimensional array to the two-dimensional matrix, specifically, map it to the two-dimensional matrix row by row or column by column according to the initial order of the data elements in the one-dimensional array. When N is not an integer multiple of n, it means that the last row of the two-dimensional matrix cannot be filled, and the last row can be filled with a fill value such as 0.

[0066] In a possible implementation, the one-dimensional array to be sorted is converted into a three-dimensional matrix.

[0067] In some specific cases, in order to meet specific algorithm requirements, optimize computing performance, or represent more complex data structures, a one-dimensional array can be converted to a three-dimensional matrix. For example, when processing data in three-dimensional space (such as three-dimensional images, volume data, etc.), it may be more appropriate to use a three-dimensional matrix.

[0068] Converting a one-dimensional array to a three-dimensional matrix involves more dimension selection. You need to determine the depth (or the size of the third dimension), the number of rows, and the number of columns of the three-dimensional matrix.

[0069] For example, for a one-dimensional array of length N, determine the depth d and the number of columns n, calculate the number of rows m = n / / (d*n), and use upward division. Map the data element index in the one-dimensional array to the three-dimensional matrix. Similarly, when N is not an integer multiple of (d*n), fill values ​​such as 0 can be used to fill the three-dimensional matrix.

[0070] Step S102: Divide the matrix into a plurality of sub-matrices, and determine a reference element within each of the sub-matrices.

[0071] In a possible implementation, the matrix is ​​divided based on the rows or columns of the matrix, and a row or a column in the matrix is ​​determined as a submatrix; and the reference element of each row or column in the matrix is ​​determined. That is, a row in the matrix is ​​determined as a submatrix, the number of submatrices is equal to the number of rows, and the submatrices corresponding to all rows constitute the matrix corresponding to the one-dimensional array, and the reference elements in each row are determined respectively. Alternatively, a column in the matrix is ​​determined as a submatrix, the number of submatrices is equal to the number of columns, and the submatrices corresponding to all columns constitute the matrix corresponding to the one-dimensional array, and the reference elements in each column are determined respectively.

[0072] In a possible implementation, the matrix is ​​divided into a plurality of matrix blocks, each of which is a submatrix; and the reference element in each of the matrix blocks is determined. That is, each matrix block is regarded as a submatrix, and the matrix corresponding to the one-dimensional array can be obtained by splicing the plurality of matrix blocks. The sizes of the plurality of matrix blocks may be the same or different. The size refers to the number of rows and columns included in the submatrix. The number of rows and columns of the submatrices corresponding to different matrix blocks may be the same or different.

[0073] By dividing the matrix into several sub-matrices, different sub-matrices can be assigned to different processors or computing nodes for separate processing. The parallelism of matrix operations can be used to accelerate the sorting process, thereby improving the sorting efficiency and the processing efficiency of large-scale data sets.

[0074] In an embodiment of the present application, a data element at the center position of the sub-matrix is ​​determined as a reference element within the sub-matrix; or, a data element is randomly selected from the sub-matrix and determined as a reference element within the sub-matrix.

[0075] For a submatrix divided by rows or columns, a data element at the center of a row or a column is determined as a reference element of the submatrix; or, a data element is randomly selected from a row or a column to be determined as a reference element of the submatrix.

[0076] For the submatrix obtained by dividing the matrix blocks, in a possible implementation, the reference elements can also be determined by row or column, but one submatrix corresponds to more than one reference element. For example, the data element at the center of a row or column in the submatrix is ​​determined as the reference element of the row or column, or a data element is randomly selected from a row or column to be determined as the reference element of the row or column, that is, the submatrix includes multiple reference elements, and the number of reference elements corresponding to the submatrix is ​​the same as the number of rows or columns in the submatrix. In another possible implementation, the data elements in the submatrix that meet specific characteristics can be determined as reference elements, and each submatrix corresponds to one reference element.

[0077] Step S103: performing a partition operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices.

[0078] The data elements in the first partition are not greater than the reference elements in the submatrix, and the data elements in the second partition are greater than the reference elements in the submatrix.

[0079] The first partition and the second partition are determined according to the position of the reference element. For example, the first partition may be on the left side of the reference element, and the second partition may be on the right side of the reference element. Alternatively, the first partition may be on the upper side of the reference element, and the second partition may be on the lower side of the reference element.

[0080] Converting a one-dimensional array into a matrix is ​​to process the data elements in the one-dimensional array more conveniently and quickly. After dividing the matrix into several sub-matrices, the embodiment of the present application uses a single instruction, multiple data (SIMD) instruction set to independently perform partition operations on the data elements within the sub-matrices, which can effectively speed up the sorting efficiency.

[0081] As a possible implementation of the present application, the SIMD instruction set includes SIMD load instructions, SIMD comparison instructions, SIMD conditional selection instructions and SIMD storage instructions. Figure 2A specific implementation process of step S103 in the data sorting method provided in the embodiment of the present application is shown, and is described in detail as follows:

[0082] A1: Load the data elements in the sub-matrix into the SIMD register through a SIMD load instruction.

[0083] Generally, a SIMD register can only store a fixed number of data elements at the same time. Depending on the size of a sub-matrix, multiple SIMD registers may be used to load and store the data elements of the sub-matrix.

[0084] A2: Compare the data element in the SIMD register with the reference element through a SIMD comparison instruction. Specifically determine whether the data element loaded into the SIMD register is greater than the reference element. SIMD comparison instructions can compare multiple data elements in one instruction cycle. Using Intel's AVX2 instruction set, 8 32-bit integers can be compared in one instruction cycle.

[0085] A3: Based on the comparison result, the data elements in the submatrix are logically partitioned by SIMD conditional selection instructions, and data elements that are not greater than the reference element are selected to the first partition, and data elements that are greater than the reference element are selected to the second partition. The SIMD conditional selection instruction can be specifically a parallel exchange instruction to implement an exchange operation on multiple data elements within one instruction cycle. Among them, the operation of logically partitioning the data elements does not actually move the data elements to different locations in the memory, but logically grouping the data elements into different vectors according to the comparison results, and storing them in the SIMD register in the form of vectors.

[0086] A4: The partitioning result is stored back into the memory through a SIMD storage instruction, thereby completing the partitioning of the sub-matrix.

[0087] In practical applications, whether to store the results back to the memory or SIMD registers is determined based on the needs of the algorithm and the size of the data. If the data size is large, or the original data needs to be retained for multiple iterative processing, the results can usually be stored back to the memory. If the data size is small, or only simple data processing operations are being performed, the results can be stored in another SIMD register to reduce the overhead of memory access.

[0088] In some scenarios, if the iteration does not require immediate access to memory, the partition result can be stored in another SIMD register and the final result can be stored in memory to complete the entire processing process.

[0089] In this embodiment, a mask is generated when the data element in the SIMD register is compared with the reference element by the SIMD comparison instruction, and according to the mask, the data element that is less than or equal to the reference element is written to the left side of the reference element, and the data element that is greater than the reference element is written to the right side of the reference element. Each bit in the mask indicates whether the corresponding data element is greater than the reference element.

[0090] Prepare two SIMD vectors, one with all elements zero (zero vector) and the other with all elements the maximum possible value (for padding). Using the mask, use SIMD conditional selection instructions to selectively write data elements less than or equal to the benchmark element into the zero vector (because they are 1 in the mask), and data elements greater than the benchmark element into the maximum value vector (because they are 0 in the mask). Merge the two vectors into a new vector, with elements less than the benchmark element on the left and elements greater than the benchmark element on the right. Then store the partition result back to the specified location such as memory or register.

[0091] In the embodiment of the present application, a SIMD instruction set can be used to simultaneously process multiple data elements in a single instruction cycle, thereby significantly improving the efficiency of the partitioning operation.

[0092] Step S104: sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after sorting of each of the sub-matrices.

[0093] The data elements in each submatrix are sorted respectively to obtain the target submatrix after each submatrix is ​​sorted.

[0094] As a possible implementation of this application, Figure 3 A specific implementation process of step S104 in the data sorting method provided in the embodiment of the present application is shown, and is described in detail as follows:

[0095] B1: respectively determining whether the number of data elements in the first partition and the second partition of the submatrix exceeds a preset number threshold.

[0096] B2: If the number of data elements in a partition exceeds the preset number threshold, recursively sort the data elements in the partition.

[0097] As a possible implementation of this application, Figure 4 A specific implementation process of recursively sorting data elements in a partition in the data sorting method provided in an embodiment of the present application is shown, and is described in detail as follows:

[0098] B21: Divide the partition into a plurality of matrix units, and determine a reference element inside each of the matrix units.

[0099] B22: Perform partition operations on each of the matrix units based on the SIMD instruction set to obtain a first sub-partition and a second sub-partition of each of the matrix units.

[0100] The data elements in the first sub-partition are not larger than the reference elements in the matrix unit, and the data elements in the second sub-partition are larger than the reference elements in the matrix unit.

[0101] B23: Sort the data elements in the first sub-partition and the second sub-partition of each of the matrix units to obtain a target matrix unit after sorting of each of the matrix units.

[0102] B24: reorganizing the target matrix units of the plurality of matrix units to obtain a sorted partition. Reorganizing the target matrix units of the plurality of matrix units in the first partition to obtain a sorted first partition, and reorganizing the target matrix units of the plurality of matrix units in the second partition to obtain a sorted second partition.

[0103] In the embodiment of the present application, recursive sorting is adopted for the partitions in the submatrix that exceed the preset number threshold, and the submatrix division, the determination of the reference element, the partition operation and the sorting of the data elements are recursively performed. For large-scale data sets, the sorting efficiency can be effectively improved by recursive sorting. The specific contents of the above steps B21 to B24 can refer to steps S102 to S104, which are not repeated here.

[0104] B3: If the number of data elements in the partition does not exceed the preset number threshold, the data elements in the partition are simply sorted. Common simple sorts include bubble sort, selection sort, and insertion sort.

[0105] B4: Merge the sorted first partitions and the second partitions to obtain a target submatrix of the submatrix.

[0106] In the embodiment of the present application, for partitions whose data elements exceed a preset number threshold, the sorting efficiency can be accelerated, and for partitions whose number of data elements does not exceed the preset number threshold, using simple sorting is faster than recursive sorting.

[0107] Step S105: reorganizing the target sub-matrices of the plurality of sub-matrices, and converting the reorganized target matrix into an ordered target one-dimensional array.

[0108] In this embodiment, the several sub-matrices obtained by division are reorganized in the division order to obtain a target matrix. The scale and size of the target matrix are the same as the original matrix corresponding to the one-dimensional array. The target matrix is ​​then converted into an array to obtain a sorted target one-dimensional array, thereby completing the sorting of the one-dimensional array.

[0109] As can be seen from the above, in the embodiment of the present application, by converting the one-dimensional array to be sorted into a matrix, dividing the matrix into several sub-matrices, the amount of data processed at a single time can be effectively reduced, and then the reference elements inside each sub-matrix are determined, and each sub-matrix is ​​partitioned in parallel based on the single instruction multiple data stream SIMD instruction set to obtain the first partition and the second partition of each sub-matrix, which can accelerate the sorting process, sort the data elements in the first partition and the second partition of each sub-matrix, and obtain the target sub-matrix after each sub-matrix is ​​sorted, reorganize the target sub-matrices of the several sub-matrices, and convert the reorganized target matrix into an ordered target one-dimensional array. The present application scheme converts the one-dimensional array corresponding to the large-scale data set into a matrix, and divides the sub-matrix into sub-matrices for sorting respectively, which can reduce memory usage and memory access delay, help avoid frequent disk IO operations due to insufficient memory when processing large-scale data sets, and make the sorting process more localized, and then combine the SIMD instruction set to partition the matrix, and use the SIMD characteristics of modern processors to improve the efficiency of parallel processing of multiple data elements, accelerate the key operations in the sorting process, and achieve efficient sorting of large-scale data sets, thereby improving the processing efficiency of large-scale data sets.

[0110] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0111] Corresponding to the data sorting method described in the above embodiment, Figure 5 A structural block diagram of a data sorting device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0112] Reference Figure 5 The data sorting device comprises: a matrix processing unit 51, a reference determination unit 52, a matrix partitioning unit 53, a sorting unit 54, and a reorganization unit 55, wherein:

[0113] A matrix processing unit 51, used for converting the one-dimensional array to be sorted into a matrix;

[0114] A reference determination unit 52, used to divide the matrix into a plurality of sub-matrices and determine a reference element within each of the sub-matrices;

[0115] A matrix partitioning unit 53 is used to perform a partitioning operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices, wherein the data elements in the first partition are not greater than the reference elements within the sub-matrix, and the data elements in the second partition are greater than the reference elements within the sub-matrix;

[0116] A sorting unit 54 is used to sort the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices has been sorted;

[0117] The reorganization unit 55 is used to reorganize the target sub-matrices of the plurality of sub-matrices, and convert the reorganized target matrix into an ordered target one-dimensional array.

[0118] As a possible implementation of the present application, the matrix partitioning unit 53 is used to:

[0119] Loading the data elements in the submatrix into a SIMD register by a SIMD load instruction;

[0120] Comparing the data element in the SIMD register with the reference element by a SIMD comparison instruction;

[0121] Based on the comparison result, the data elements in the submatrix are logically partitioned by SIMD conditional selection instructions, data elements not greater than the reference element are selected to a first partition, and data elements greater than the reference element are selected to a second partition;

[0122] The partitioning result is stored back into the memory through the SIMD storage instruction, thereby completing the partitioning of the sub-matrix.

[0123] As a possible implementation of the present application, the sorting unit 54 specifically includes:

[0124] An element number determination module, used to determine whether the number of data elements in the first partition and the second partition of the submatrix exceeds a preset number threshold;

[0125] A first sorting module, configured to recursively sort the data elements in a partition if the number of data elements in the partition exceeds the preset number threshold;

[0126] A second sorting module, configured to perform a simple sort on the data elements in the partition if the number of data elements in the partition does not exceed the preset number threshold;

[0127] The partition merging module is used to merge the sorted first partitions with the second partitions to obtain a target sub-matrix of the sub-matrix.

[0128] As a possible implementation of the present application, the first sorting module is specifically used for:

[0129] Dividing the partition into a plurality of matrix units, and determining a reference element within each of the matrix units;

[0130] Performing a partition operation on each of the matrix units based on the SIMD instruction set to obtain a first sub-partition and a second sub-partition of each of the matrix units, wherein a data element in the first sub-partition is not greater than a reference element inside the matrix unit, and a data element in the second sub-partition is greater than a reference element inside the matrix unit;

[0131] Sorting the data elements in the first sub-partition and the second sub-partition of each of the matrix units to obtain a target matrix unit after the matrix units are sorted;

[0132] The target matrix units of the plurality of matrix units are reorganized to obtain sorted partitions.

[0133] As a possible implementation of the present application, the benchmark determination unit 52 is specifically used for:

[0134] Dividing the matrix based on rows or columns of the matrix, and determining a row or a column in the matrix as a submatrix;

[0135] A reference element for each row or column in the matrix is ​​determined.

[0136] As a possible implementation manner of the present application, the benchmark determination unit 52 is further configured to:

[0137] Dividing the matrix into a plurality of matrix blocks, each matrix block being a sub-matrix;

[0138] A reference element in each of the sub-matrices is determined.

[0139] As a possible implementation of the present application, the benchmark determination unit 52 includes:

[0140] A first reference determination module, used to determine the data element at the center position of the sub-matrix as the reference element inside the sub-matrix;

[0141] Alternatively, the first reference determination module is used to randomly select a data element from the sub-matrix to be determined as a reference element within the sub-matrix.

[0142] As can be seen from the above, in the embodiment of the present application, by converting the one-dimensional array to be sorted into a matrix, dividing the matrix into several sub-matrices, the amount of data processed at a single time can be effectively reduced, and then the reference elements inside each sub-matrix are determined, and each sub-matrix is ​​partitioned in parallel based on the single instruction multiple data stream SIMD instruction set to obtain the first partition and the second partition of each sub-matrix, which can accelerate the sorting process, sort the data elements in the first partition and the second partition of each sub-matrix, and obtain the target sub-matrix after each sub-matrix is ​​sorted, reorganize the target sub-matrices of the several sub-matrices, and convert the reorganized target matrix into an ordered target one-dimensional array. The present application scheme converts the one-dimensional array corresponding to the large-scale data set into a matrix, and divides the sub-matrix into sub-matrices for sorting respectively, which can reduce memory usage and memory access delay, help avoid frequent disk IO operations due to insufficient memory when processing large-scale data sets, and make the sorting process more localized, and then combine the SIMD instruction set to partition the matrix, and use the SIMD characteristics of modern processors to improve the efficiency of parallel processing of multiple data elements, accelerate the key operations in the sorting process, and achieve efficient sorting of large-scale data sets, thereby improving the processing efficiency of large-scale data sets.

[0143] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0144] The present application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, Figures 1 to 4 Represents the steps of any data sorting method.

[0145] The embodiment of the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, Figures 1 to 4 Represents the steps of any data sorting method.

[0146] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the following Figures 1 to 4 Represents the steps of any data sorting method.

[0147] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 6As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned data sorting method embodiments are implemented, for example Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of units 51 to 55 are shown.

[0148] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the electronic device 6.

[0149] The electronic device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 6 may also include input and output devices, network access devices, buses, etc.

[0150] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0151] The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0152] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0153] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data sorting method, characterized in that: include: Convert the one-dimensional array to be sorted into a matrix; Dividing the matrix into a plurality of sub-matrices, and determining a reference element within each of the sub-matrices; Performing a partition operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices, wherein the data elements in the first partition are not greater than a reference element within the sub-matrix, and the data elements in the second partition are greater than the reference element within the sub-matrix; Sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices has been sorted; The target sub-matrices of the plurality of sub-matrices are reorganized, and the reorganized target matrix is ​​converted into an ordered target one-dimensional array.

2. The method according to claim 1, characterized in that The step of performing a partition operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices comprises: Loading the data elements in the submatrix into a SIMD register by a SIMD load instruction; Comparing the data element in the SIMD register with the reference element by a SIMD comparison instruction; Based on the comparison result, the data elements in the submatrix are logically partitioned by SIMD conditional selection instructions, data elements not greater than the reference element are selected to a first partition, and data elements greater than the reference element are selected to a second partition; The partitioning result is stored back into the memory through the SIMD storage instruction, thereby completing the partitioning of the sub-matrix.

3. The method according to claim 1, characterized in that The step of sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after sorting each of the sub-matrices comprises: respectively determining whether the number of data elements in the first partition and the second partition of the submatrix exceeds a preset number threshold; If the number of data elements in a partition exceeds the preset number threshold, recursively sorting the data elements in the partition; If the number of data elements in the partition does not exceed the preset number threshold, the data elements in the partition are simply sorted; The sorted first partitions and the second partitions are merged to obtain a target sub-matrix of the sub-matrix.

4. The method according to claim 3, characterized in that: The step of recursively sorting the data elements in the partition comprises: Dividing the partition into a plurality of matrix units, and determining a reference element within each of the matrix units; Performing a partition operation on each of the matrix units based on the SIMD instruction set to obtain a first sub-partition and a second sub-partition of each of the matrix units, wherein a data element in the first sub-partition is not greater than a reference element inside the matrix unit, and a data element in the second sub-partition is greater than a reference element inside the matrix unit; Sorting the data elements in the first sub-partition and the second sub-partition of each of the matrix units to obtain a target matrix unit after the matrix units are sorted; The target matrix units of the plurality of matrix units are reorganized to obtain sorted partitions.

5. The method according to claim 1, characterized in that The step of dividing the matrix into a plurality of sub-matrices and determining a reference element within each of the sub-matrices comprises: Dividing the matrix based on rows or columns of the matrix, and determining a row or a column in the matrix as a submatrix; A reference element for each row or column in the matrix is ​​determined.

6. The method according to claim 1, characterized in that The step of dividing the matrix into a plurality of sub-matrices and determining a reference element within each of the sub-matrices comprises: Dividing the matrix into a plurality of matrix blocks, each matrix block being a sub-matrix; A reference element in each of the sub-matrices is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The step of determining the reference elements within each of the sub-matrices comprises: Determine the data element at the center position of the sub-matrix as the reference element inside the sub-matrix; Alternatively, a data element is randomly selected from the sub-matrix and determined as a reference element within the sub-matrix.

8. A data sorting device, characterized in that: include: A matrix processing unit, used for converting the one-dimensional array to be sorted into a matrix; A reference determination unit, used to divide the matrix into a plurality of sub-matrices and determine a reference element within each of the sub-matrices; A matrix partitioning unit, configured to perform a partitioning operation on each of the sub-matrices based on a single instruction multiple data (SIMD) instruction set to obtain a first partition and a second partition of each of the sub-matrices, wherein the data elements in the first partition are not greater than a reference element within the sub-matrix, and the data elements in the second partition are greater than the reference element within the sub-matrix; A sorting unit, used for sorting the data elements in the first partition and the second partition of each of the sub-matrices to obtain a target sub-matrix after each of the sub-matrices has been sorted; The reorganization unit is used to reorganize the target sub-matrices of the plurality of sub-matrices and convert the reorganized target matrix into an ordered target one-dimensional array.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data sorting method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data sorting method according to any one of claims 1 to 7 is implemented.

Images