Mathematical algorithm calculation system and optimization method based on big data

Abstract

The invention discloses a mathematical algorithm calculation system and optimization method based on big data, and the method comprises the steps: calculating the local entropy density of a data set, constructing an entropy gradient field, and extracting a representative data subset with global sorting information through employing a multi-level recursive strategy; calculating initial partition mapping based on the subset, and introducing an entropy difference driven partition adjustment mechanism to dynamically optimize a data partition boundary; constructing a relative sequence relation of data points in each partition, calculating an optimal exchange sequence by using a minimum cost and maximum flow method, and reducing global adjustment times through a recursive hierarchical structure to achieve local sorting lower than linear time; after sorting, local optimal exchange or global correction is carried out through a dynamic entropy difference adjustment strategy; in a real-time data stream scene, a dynamic sorting topological graph is constructed, only a local area is updated, and global resorting is avoided. Compared with a traditional sorting method, the method has remarkable advantages in the aspects of real-time performance, high efficiency and calculation cost.

Description

Technical Field

[0001] The present invention relates to the technical field of big data sorting, and particularly to a mathematical algorithm computing system and optimization method based on big data. Background Art

[0002] In today's data-driven era, with the rapid development of fields such as the Internet, Internet of Things, and artificial intelligence, the amount of data globally has shown exponential growth. Whether it is real-time processing of massive transaction records in a financial trading system, quickly responding to user query requests in a search engine, or dynamically sorting and recommending content on a social media platform, it involves the efficient sorting of large-scale data sets. Sorting, as a basic operation in computer science, directly affects the efficiency of data processing and the overall performance of the system. However, in the face of data volumes of terabytes or even petabytes, traditional sorting algorithms are gradually showing bottlenecks in computational complexity and time costs, and cannot meet the requirements of high-concurrency and low-latency real-time data processing.

[0003] Currently, common sorting methods include comparison-based sorting (such as quicksort, mergesort) and non-comparison sorting (such as radix sort, counting sort). Although these methods perform well on small-scale data sets, when the data volume reaches tens of millions, hundreds of millions or even larger scales, the time complexity of O(n log n) or O(n) of traditional sorting algorithms causes their computational overhead to increase sharply, resulting in the sorting speed being difficult to meet real-time requirements. In addition, the data distribution in a distributed computing environment is often highly unbalanced, making it difficult for traditional parallel sorting algorithms to balance the load, thereby further reducing the sorting efficiency.

[0004] In response to these challenges, researchers have proposed a series of optimization schemes in recent years, such as bucket-based hash sorting, bit-parallel sorting based on bit operations, and intelligent sorting algorithms that use machine learning to optimize sorting decisions. However, these methods more or less rely on pre-set partitioning strategies, are difficult to adapt to complex actual data distributions, and most still maintain a time complexity of at least O(n). Therefore, in the face of large-scale and rapidly growing data, it is still difficult to break through the computational performance bottleneck.

[0005] In the basic operation of sorting big data, for large-scale data sets, traditional linear-time algorithms may not meet real-time requirements, so it is necessary to develop a new and creative sublinear-time algorithm, that is, the running time of the algorithm grows slower than linearly with the growth of the data scale. How to design a more efficient sublinear-time algorithm remains a challenge. Summary of the Invention

[0006] To solve the above problems, the present invention provides a mathematical algorithm computing system and optimization method based on big data.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] On the one hand, the present invention discloses a mathematical algorithm calculation optimization method based on big data, comprising:

[0009] Step 1: Calculate the local entropy density of the data set, construct the entropy gradient field, perform adaptive weighted sampling based on the entropy gradient value, and use a multi-level recursive sampling strategy to extract a data subset that retains the global sorting information;

[0010] Step 2: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce the partition adjustment mechanism driven by entropy difference, dynamically optimize the data partition boundaries, allocate more partitions to areas with high data density, build non-uniform partition mapping, and combine with variable-length hash index to improve query efficiency;

[0011] Step 3: In each non-uniform partition, construct the relative order relationship of the data points, calculate the optimal exchange degree, generate an exchange graph and use the minimum cost maximum flow method to calculate the optimal exchange sequence, use the recursive hierarchy and jump step strategy to reduce the number of data exchanges and global adjustments, and achieve local sorting with less than linear time complexity;

[0012] Step 4: After the sorting is completed, the sorting error is calculated, and the minimum adjustment point is selected to perform local optimal exchange or recursively correct the global sorting through the dynamic entropy difference adjustment strategy to further optimize the sorting result;

[0013] Step 5: For the real-time data stream update scenario, build a dynamic sorting topology map, calculate the relative order of the newly added data and determine the insertion point. Through local topology dependency updates and recursive topology corrections, only the affected local areas are adjusted to avoid global reordering.

[0014] Further: Step 1 comprises:

[0015] Estimate the probability density of data points and then calculate the global information entropy of the data set to obtain the overall information characteristics of the data set;

[0016] Determine the sorting neighborhood of the data points, calculate the local entropy contribution of each data point, and measure its impact on the global sorting information in the local area;

[0017] Calculate the entropy gradient vector for each data point to determine the importance of the data point in the sorting space;

[0018] Based on the entropy gradient field information, weighted sampling is performed according to the size of the entropy gradient value, so that the area with drastic sorting changes is selected with a higher probability;

[0019] Adopt a multi-level recursive sampling strategy. After initial sampling, perform local refinement and recursive iteration until the entropy stability condition is met, and finally extract a key data subset that retains the global sorting information.

[0020] Furthermore: The said step 2 includes:

[0021] Set the original data range, divide the high-information entropy data subset obtained by global entropy reduction sampling into initial equal-width intervals, and calculate the local information entropy of each partition to measure the data complexity of each partition;

[0022] Introduce an entropy difference driving mechanism, calculate the entropy difference between adjacent partitions, and reflect the change of information complexity between partitions;

[0023] If the multiple of the partition information entropy is higher than the average entropy, then split the partition into two new partitions, calculate the new partition boundaries, and make the entropy values of the sub-partitions close;

[0024] If the multiple of the partition information entropy is lower than the average entropy, then try to merge with the adjacent low-entropy partition to reduce the number of partitions;

[0025] After adjustment, obtain an optimized set of non-uniform partitions, calculate the non-uniform partition mapping function, and ensure that data points can be correctly classified into the adjusted partitions;

[0026] Construct a dynamically adjusted hash range for each partition to form a variable-length hash index.

[0027] Furthermore: The said step 3 includes:

[0028] Within each non-uniform partition, construct a relative relationship matrix between data points, record the relative order relationship of data points, and make inferences by combining local entropy information and partition boundaries;

[0029] Define the optimal exchange degree of a data point within the current partition to measure the exchange priority of the data point in the local sorting. The smaller the optimal exchange degree value, the closer the data point is to the optimal sorting position;

[0030] Construct a data exchange graph, where the vertices represent data points, the edges represent the optimal exchange paths, and the edge weights are the exchange costs. The smaller the cost, the higher the priority;

[0031] Select the path with the minimum exchange cost, exchange the data in the path order, and minimize the sorting error;

[0032] Recursively divide the exchange levels. At the first level, perform a preliminary exchange on the entire partition. At the second level, perform local refinement on the data points with larger exchange errors. If there are still errors, continue recursive adjustment until the convergence condition is met.

[0033] Furthermore: The said step 4 includes:

[0034] Based on the local sorting stability model, estimate the probability of the relative position information of data points in the sorted state, and then calculate the information entropy of the current sorting sequence;

[0035] According to the global entropy reduction model, calculate the theoretical optimal sorting entropy as the target value for sorting correction;

[0036] Define the dynamic entropy difference as the difference between the current sorting information entropy and the theoretical optimal sorting entropy, which is used to measure the sorting error; if the entropy difference is greater than zero, it means that the local orderliness is poor and local correction is required; if it is less than zero, it means over-sorting and global adjustment is required; if it is equal to zero, no adjustment is needed;

[0037] Calculate the local entropy gradient, and determine the data points that contribute greatly to the global sorting error as the minimum adjustment points, and adjust them preferentially;

[0038] For the set of data points with large sorting errors, adopt the minimum swap principle for local correction, and by swapping adjacent data points, make the overall sorting information entropy converge to the optimal direction;

[0039] If the entropy difference still cannot converge to zero after local adjustment, then adopt the recursive correction strategy, and perform cross-partition swapping and re-adjusting the optimal swapping path in turn until it converges to the optimal sorting state.

[0040] Furthermore, the step 5 includes:

[0041] Data increment modeling: Assume that the current existing sorted sequence is S, and the new data stream is ΔS. The goal is to maintain the correctness of the overall sorting through dynamic adjustment without re-executing the global sorting; construct a dynamic sorting topology graph G=(V, E) of the data stream, where the vertex set V corresponds to the ordered data points, and the edge set E represents the relative order constraint;

[0042] Calculate the entropy contribution degree of the newly added data points relative to the existing sorted sequence, and determine their insertion points in S;

[0043] For each newly added data point, find the closest adjacent data point, and add the corresponding edge in the topology graph to update the local topological order relationship;

[0044] Calculate the topological sorting inconsistency degree, and judge whether the newly added data destroys the consistency of the global topological sorting;

[0045] If there is an inconsistency, only perform recursive adjustment on the affected local subgraph, re-sort using the minimum swap principle, calculate the local entropy gradient, and preferentially adjust the higher gradient nodes to ensure that the global sequence satisfies the topological consistency.

[0046] On the other hand, the present invention discloses a mathematical algorithm computing system based on big data, including:

[0047] Global Entropy Reduction Sampling Module: Calculate the local entropy density of the dataset, construct an entropy gradient field, perform adaptive weighted sampling based on the entropy gradient value, and adopt a multi-level recursive sampling strategy to extract a data subset that retains the global sorting information;

[0048] Non-uniform Partition Mapping Module: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce an entropy difference-driven partition adjustment mechanism, dynamically optimize the data partition boundary, allocate more partitions to regions with high data density, construct a non-uniform partition mapping, and combine a variable-length hash index to improve the query efficiency;

[0049] Hierarchical Recursive Optimal Swap Sorting Module: Within each non-uniform partition, construct the relative order relationship of data points, calculate the optimal swap degree, generate a swap graph and use the minimum cost maximum flow method to calculate the optimal swap sequence, and use the recursive hierarchical structure and jump step strategy to reduce the number of data exchanges and global adjustments, achieving local sorting with a sub-linear time complexity;

[0050] Dynamic Entropy Difference Correction Module: After the sorting is completed, calculate the sorting error, and through the dynamic entropy difference adjustment strategy, select the minimum adjustment point for local optimal swapping or recursively correct the global sorting to further optimize the sorting result;

[0051] Streaming Topological Order Incremental Sorting Module: For the scenario of real-time data stream updates, construct a dynamic sorting topological graph, calculate the relative order of the newly added data and determine the insertion point, and only adjust the affected local area through local topological dependency updates and recursive topological corrections to avoid global re-sorting.

[0052] Compared with the prior art, the technical progress achieved by the present invention lies in:

[0053] In the global entropy reduction sampling stage, only the data subset needs to be processed, reducing the subsequent calculation amount and reducing the sorting scale to a small proportion of the original dataset. The adaptive partition mapping adopts an entropy difference-driven dynamic partition method, making the data distribution uniform and reducing unnecessary data movement. The hierarchical recursive optimal swap sorting adopts a swap path based on the relative relationship instead of the traditional comparison operation, greatly reducing the calculation amount. The streaming topological order incremental sorting only adjusts the local data in the data stream environment to avoid global re-sorting.

[0054] Generally speaking, the overall computational complexity of the present invention is close to O(n α )(0 < α < 1) in the optimal case, achieving sub-linear time sorting and greatly improving the sorting efficiency in the big data environment.

[0055] The present invention is not only applicable to static large data sets, but also has the ability to dynamically adjust streaming data. Streaming topological order incremental sorting adopts topological order incremental sorting to avoid global reordering when the data stream is updated. Compared with the recalculation of traditional methods, this method only performs topological adjustments locally, making the maintenance cost of sorting much lower than traditional methods. Combined with global entropy reduction sampling, new data can be quickly matched to a suitable sorting area, reducing the amount of calculation for global scanning. The present invention can be executed in parallel, making it more scalable in a distributed computing environment.

[0056] Compared with the traditional method based on comparison operation, the present invention proposes a sorting optimization strategy based on data entropy gradient, which can adaptively process different data distributions and improve sorting stability: global entropy reduction sampling finds the area with the most drastic sorting changes for sampling through local entropy density estimation, avoiding the problem of insufficient data representativeness caused by uniform sampling; adaptive partition mapping adopts a dynamic partition adjustment strategy driven by entropy difference to make data mapping more balanced, solving the problem of unbalanced load in traditional hash bucketing; dynamic entropy difference correction reduces sorting errors through entropy difference correction, ensures the global consistency of sorting results, and reduces data order deviation caused by local sorting.

[0057] In a big data environment, storage overhead is one of the important performance indicators of sorting algorithms. The present invention adopts the following strategies to reduce storage overhead: Streaming topological order incremental sorting adopts topological order sorting, which does not require complete maintenance of the sorting index. It only needs to store the local relative order to avoid storing the complete sorting index. Hierarchical recursive optimal exchange adopts optimal exchange path calculation, which avoids the additional storage requirements of temporary arrays in the middle of traditional sorting algorithms and reduces intermediate data storage. The adaptive partition mapping of adaptive partition mapping can be directly mapped to the distributed storage environment, reducing the data transmission cost between nodes and improving computing efficiency.

[0058] The present invention has good scalability and is suitable for various scenarios such as financial transactions, search engines, social media recommendations, and Internet of Things data stream processing: in financial transaction systems, it can be used for high-speed sorting of market transaction data to ensure real-time ranking under low latency; in search engines, it can be used for efficient index sorting to improve query response speed; in social media recommendation systems, it can be used for personalized content sorting to achieve an efficient dynamic recommendation mechanism; in Internet of Things data stream processing, it can be used for hierarchical data sorting on low-power devices to reduce computing resource consumption.

[0059] The present invention has sublinear time complexity, streaming data processing capability, entropy-based optimization sorting stability, low storage overhead and good scalability. Compared with traditional sorting methods, it has significant advantages in real-time, high efficiency and computing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention.

[0061] In the accompanying drawings:

[0062] Figure 1 is a flowchart of the present invention. Detailed implementation manners

[0063] The following specific embodiments can be combined with each other. For the same or similar concepts or processes, they may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0064] Embodiment 1

[0065] As Figure 1 shown, the present invention discloses a method for optimizing the calculation of a mathematical algorithm based on big data, including:

[0066] Step 1: Calculate the local entropy density of the data set, construct an entropy gradient field, perform adaptive weighted sampling based on the entropy gradient value, and adopt a multi-level recursive sampling strategy to extract a data subset that retains the global sorting information;

[0067] Step 2: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce an entropy difference-driven partition adjustment mechanism, dynamically optimize the data partition boundary, allocate more partitions to the regions with high data density, construct a non-uniform partition mapping, and combine a variable-length hash index to improve the query efficiency;

[0068] Step 3: Within each non-uniform partition, construct the relative order relationship of data points, calculate the optimal exchange degree, generate an exchange graph and use the minimum cost maximum flow method to calculate the optimal exchange sequence, and use a recursive hierarchical structure and a jump step strategy to reduce the number of data exchanges and global adjustments, and achieve local sorting with a sub-linear time complexity;

[0069] Step 4: After the sorting is completed, calculate the sorting error, and through a dynamic entropy difference adjustment strategy, select the minimum adjustment point for local optimal exchange or recursively correct the global sorting to further optimize the sorting result;

[0070] Step 5: For the scenario of real-time data stream update, construct a dynamic sorting topology graph, calculate the relative order of the newly added data and determine the insertion point, and through local topology dependency update and recursive topology correction, only adjust the affected local area to avoid global re-sorting.

[0071] Specifically, Step 1 includes:

[0072] This step aims to construct an entropy gradient field of the data distribution by calculating the local entropy density of the dataset, and perform adaptive weighted sampling based on this to ensure that the selected data points retain the global sorting information to the greatest extent. The specific implementation process is as follows:

[0073] 1. Calculate the global information entropy of the data

[0074] To obtain the overall information characteristics of the dataset, first calculate the global information entropy H(D) of the dataset D = {d1, d2,..., d n}, which is defined as follows:

[0075]

[0076] where p(d i ) is the probability density estimate of the data point d i in the dataset, which can be calculated by kernel density estimation or histogram method. The estimation method is as follows:

[0077]

[0078] where:

[0079] K(x) is the kernel function (such as Gaussian kernel ).

[0080] h is the smoothing parameter (bandwidth), which controls the smoothing degree of the data density estimation.

[0081] 2. Calculate the local entropy contribution

[0082] To evaluate which regions in the dataset have a greater impact on the global sorting information, define the local entropy contribution H(d i ) of each data point d i :

[0083]

[0084] where:

[0085] N(i) is the set of adjacent data points of the data point d i in the sorting neighborhood, which can be determined by the k-nearest neighbor (k-NN) method.

[0086] This formula measures the degree of change in the information entropy of the data point d i in the local area and can identify regions with large sorting changes.

[0087] 3. Construct the entropy gradient field

[0088] To determine the importance of data points in the sorting space, construct the entropy gradient field of the dataset, that is, calculate the entropy gradient vector of each data point:

[0089]

[0090] Among them:

[0091] If the data is one-dimensional numerical data, the entropy gradient is only

[0092] If the data has multiple features (multi-dimensional dataset), the entropy gradient is a vector representing the entropy change rate of the data points in different feature dimensions.

[0093] 4. Adaptive weighted sampling

[0094] Based on the entropy gradient field information, weighted sampling is performed according to the magnitude of the entropy gradient value, so that the regions with drastic sorting changes are selected with a higher probability. Let the data point d i The sampling probability p(d i ) is:[[]]

[0095]

[0096] Among them:

[0097] α is a smoothing factor used to control the influence of the entropy contribution on the sampling probability;

[0098] The exponential form ensures that the data points in the high-entropy region have a greater sampling probability, while avoiding the complete loss of data points in the low-entropy region.

[0099] 5. Multi-level recursive sampling optimization

[0100] In order to balance the global sorting structure and local sorting details, a multi-level recursive sampling strategy is adopted:

[0101] 1. Initial sampling: According to the above adaptive weighted sampling method, a batch of data points S0 are randomly selected.

[0102] 2. Local refinement: Perform local entropy analysis on the sampled data S0, identify the regions with large entropy changes, and perform secondary sampling to form S1.

[0103] 3. Recursive iteration: If the sorting structure still fails to converge, continue to perform sampling iteration until the entropy stability condition is met:

[0104]

[0105] where ∈ is the set stability threshold to ensure that the sampled data set fully represents the original data set in terms of information entropy.

[0106] Through the above steps, the global entropy reduction sampling algorithm can effectively extract the data subset with the richest sorting information, reducing the complexity of subsequent sorting calculations while ensuring that the sorting quality is close to the optimal solution of sorting the entire dataset. This step models the data distribution using the entropy gradient field and adopts a multi-level recursive sampling strategy, improving the representativeness and computational efficiency of the sampled data.

[0107] Specifically, step 2 includes:

[0108] Based on the high-information-entropy representative data subset S generated by global entropy reduction sampling, this step constructs a non-uniform partition mapping method through an entropy difference-driven adaptive partition adjustment mechanism to ensure the load balance of data when mapping to partitions and optimize the efficiency of subsequent sorting calculations. The core goal of this method is to make the regions with higher data density map to more partitions by dynamically adjusting the partition boundaries, reducing the local sorting burden while maintaining the overall order of the data.

[0109] 1. Calculate the initial mapping of partitions

[0110] In large-scale data sorting tasks, the traditional fixed-length hash bucketing method can lead to excessive load in regions with uneven data density, affecting the sorting efficiency. Therefore, in this embodiment, the initial partition mapping is first calculated based on the sampled data S = {s1, s2,..., s m}:

[0111] 1.1 Set the data range

[0112] Let the numerical range of the original dataset D be [X min , X max . In this embodiment, the sampled data S is first sorted and divided into B initial equal-width intervals:

[0113] B k = [X min + k·Δx, X min + (k + 1)·Δx], k ∈ {0, 1,..., B - 1}

[0114] where is the initial interval width and B is the number of initial partitions.

[0115] 1.2 Calculate the local entropy of each partition

[0116] For each initial partition B_k, calculate the local information entropy H(B k ) within this interval to measure its data complexity:

[0117]

[0118] where p(s i)Determined by the kernel density estimation (KDE) method calculated from global entropy reduction sampling.

[0119] 2. Calculate the entropy difference distribution of partitions

[0120] Since the data distribution in different regions may be extremely uneven, this embodiment introduces an entropy difference-driven partition adjustment mechanism to calculate the entropy difference between each adjacent partition:

[0121] ΔH k = H(B k ) - H(B k-1 )

[0122] The entropy difference ΔH k reflects the change in information complexity between adjacent partitions. If |ΔH k | is too large, it indicates that the sorting complexity in this area has increased suddenly, and the partition boundary needs to be further adjusted to balance the load.

[0123] 3. Perform adaptive partition adjustment

[0124] To allocate more partitions to areas with higher data density and reduce the number of partitions in low-density areas at the same time, this embodiment adopts the following strategy:

[0125] 3.1 Adjust the partitions in high-entropy areas

[0126] If the information entropy of a certain partition B k is much higher than the average entropy H avg , that is, it satisfies the condition:

[0127] H(B k ) > H avg + λ·σ H

[0128] where σ H is the standard deviation of the entropy values of all partitions, and λ is an adjustment coefficient (usually taken as 1.5 - 2 to ensure effective identification of high-entropy areas), then split this partition into two new partitions and The new boundary is calculated by the following formula:

[0129]

[0130] That is, select the splitting point X split that makes the entropy values of the two sub-partitions as close as possible.

[0131] 3.2 Merge the partitions in low-entropy areas

[0132] If the information entropy of a certain partition B j is lower than a certain proportion of the average entropy H avg , that is, it satisfies:

[0133] H(B j ) > H avg + λ·σ H

[0134] Then it attempts to merge with the adjacent low - entropy partition B j+1 to reduce the unnecessary number of partitions, making the sorting load of each partition more balanced. The entropy value of the new partition after merging is calculated as follows:

[0135] H(B′ j ) = H(B j ) + H(B j+1 )

[0136] And update the new partition boundary

[0137] 4. Calculate the final non - uniform partition mapping

[0138] After the aforementioned adaptive adjustment, the optimized non - uniform partition set {B′1, B′2,..., B′ B′} is obtained, where B′ may be different from the initial number of partitions B. Finally, in this embodiment, a non - uniform partition mapping function is calculated for the data set D:

[0139] f(d i ) = k, if d i ∈ B′ k

[0140] This mapping function f ensures that the data point d i is correctly classified into the adjusted partition B′ k , and can quickly locate the partition it belongs to, so as to improve the efficiency of subsequent sorting.

[0141] 5. Accelerate query using variable - length hash index

[0142] To quickly find the partition where the data is located and reduce the computational overhead, this embodiment constructs a variable - length hash index, such that each partition B′ k corresponds to a dynamically adjusted hash range:

[0143]

[0144] where M is the total number of buckets in the hash table. This variable - length hash mapping method avoids the serious data skew problem that may be caused by a fixed - size hash, making the data query speed faster.

[0145] Based on the representative data subset S with high information entropy generated by global entropy reduction sampling in this step, a non-uniform partition adjustment strategy driven by entropy difference is used to dynamically optimize the data partition mapping, making the sorting load more balanced, and combining variable-length hash indexing to improve the data query efficiency. Finally, the adjusted non-uniform partition mapping can effectively reduce the complexity of sorting calculation and provide a better calculation basis for the subsequent hierarchical recursive optimal exchange sorting.

[0146] Specifically, step 3 includes:

[0147] After determining the data subset with intensive sorting information through global entropy reduction sampling in step 1 and completing the efficient partitioning of data through non-uniform partition mapping driven by entropy difference in step 2, this step performs sorting optimization within each partition.

[0148] To reduce the sorting complexity, this step proposes hierarchical recursive optimal exchange sorting, constructs an optimal exchange path based on the relative order relationship of data points, and performs sorting using a recursive hierarchical structure, making the sorting time complexity lower than linear growth. The main goal of hierarchical recursive optimal exchange is to reduce unnecessary data movement, achieve more efficient local sorting through the optimal exchange strategy, and reduce the number of global data adjustments through the recursive hierarchical structure.

[0149] 1. Construct the relative order relationship between data blocks

[0150] 1.1 Calculate the relative order index of data points

[0151] Within each non-uniform partition B′ k the data point set D k ={d1, d2,..., d m} needs to be sorted. Traditional sorting algorithms rely on comparison operations. In this embodiment, a method based on relative order index is adopted to first construct a relative relationship matrix between data points:

[0152]

[0153] where M i,j records the relative order relationship between data points d i and d j , without performing direct comparison operations, but inferring by combining the local entropy information of global entropy reduction sampling and the non-uniform partition boundary generated by partition mapping.

[0154] 1.2 Calculate the optimal swap degree of data points

[0155] Define the optimal swap degree (Optimal Swap Degree, OSD) of data point d i within the current partition B′ k :

[0156]

[0157] Wherein:

[0158] N(i) is d i Local Swap Pairs within the sorting neighborhood.

[0159] is the theoretical optimal sorting relationship matrix, derived from the sampling data S through the entropy reduction rule.

[0160] When OSD(d i ) = 0, the local sorting of the data point d i in this partition is already optimal and does not need to be adjusted.

[0161] 2. Calculate the optimal swap path

[0162] 2.1 Generate the swap graph

[0163] Based on the relative order index matrix M i,j and the optimal swap degree OSD(d i ), this embodiment constructs a data swap graph G(V, E), where:

[0164] The vertex V represents the data points in the partition;

[0165] The edge E represents the optimal swap path between the data points, and the weight of the edge is defined as:

[0166] w(i, j) = OSD(d i ) + OSD(d j )

[0167] When w(i, j) is small, it means that the swap between d i and d j can reduce the global sorting error, and this swap is preferentially executed.

[0168] 2.2 Calculate the optimal swap sequence

[0169] This embodiment uses the minimum cost maximum flow method to calculate the global optimal swap sequence:

[0170] 1. Select the swap path such that the swap cost ∑w(i, j) on the path is minimized.

[0171] 2. Swap the data in the path order to minimize the data sorting error.

[0172] 3. Use a recursive hierarchy to reduce the number of data swaps

[0173] 3.1 Recursively partition the swap levels

[0174] To reduce the number of global data adjustments, this embodiment adopts a recursive hierarchical structure to gradually refine the swapping range:

[0175] 1. First level: Perform a preliminary swap on the entire partition B′ k to make the data points close to their final sorted positions;

[0176] 2. Second level: Perform local refined sorting on the data points with still relatively large swapping errors;

[0177] 3. Third level: If there are still relatively large sorting errors, continue recursive adjustment until the convergence condition is met:

[0178]

[0179] where ∈ is the set sorting error tolerance.

[0180] 3.2 Adopt a jump step size to reduce the computational complexity

[0181] To avoid excessive recursive calculations, this embodiment adopts a jump step size to reduce the computational complexity:

[0182] 1. For the data points that are already basically sorted, directly skip them and no longer participate in the swap;

[0183] 2. Only perform recursive adjustment on the data points with high OSD values to reduce unnecessary calculations.

[0184] 4. Computational complexity analysis

[0185] The time complexity of traditional quicksort in the worst case is O(n log n), while hierarchical recursive optimal swapping passes through:

[0186] 1. Adopt a relative order index to reduce the number of comparisons;

[0187] 2. Adopt an optimal swap path to reduce data swapping;

[0188] 3. Adopt a recursive hierarchical structure to reduce the number of global data adjustments;

[0189] 4. Adopt a jump step size strategy to reduce unnecessary calculations.

[0190] Finally, the sorting time complexity is close to O(n), with a better time growth rate than traditional sorting algorithms.

[0191] This step is based on the relative order index. By constructing an optimal swap path, calculating the optimal swap sequence, and adopting a hierarchical recursive structure to reduce the number of sorting adjustments, a sorting method with a sub-linear time complexity is finally achieved. This method combines the sampling results of step 1 and the partitioning structure of step 2, significantly improving the computational efficiency while ensuring the sorting accuracy.

[0192] Specifically, step 4 includes:

[0193] After extracting the key sorting information through global entropy reduction sampling in step 1 and optimizing the data distribution through entropy difference-driven non-uniform partitioning mapping in step 2, step 3 further completes the sorting based on the hierarchical recursive optimal swap strategy. However, due to the possible local order instability or global error accumulation during the sorting process, the sorting result may deviate from the global optimal solution. Therefore, dynamic entropy difference correction is required after the sorting is completed.

[0194] Dynamic entropy difference correction calculates the sorting error, adopts a dynamic entropy difference adjustment strategy, and performs the minimum adjustment operation to make the final sorting result closer to the theoretical optimal sorting structure, further improving the global sorting quality.

[0195] 1. Calculate the sorting error

[0196] 1.1 Calculate the information entropy of the current sorting sequence

[0197] Let the final sorting sequence be S = {s1, s2..., s n}, and the current sorting information entropy defined in this embodiment is:

[0198]

[0199] Where:

[0200] p(s i ) represents the probability of the relative position information of the data point s i in the sorted state, which can be estimated by the local sorting stability calculation model.

[0201] Since the data has different densities after non-uniform partitioning, the deviation of the information entropy reflects the magnitude of the local sorting error. If the information entropy of some regions deviates significantly from the theoretical value, it means that the sorting of these regions needs to be adjusted.

[0202] 1.2 Calculate the theoretical optimal sorting entropy

[0203] According to the global entropy reduction model provided in step 1, this embodiment calculates the theoretical optimal sorting entropy:

[0204]

[0205] Where, p * (si ) It is determined by the global distribution of the sampled data through global entropy reduction sampling, representing the sorting state that should theoretically be achieved.

[0206] 2. Calculate the dynamic entropy difference

[0207] To measure the sorting error, in this embodiment, the dynamic entropy difference is defined as:

[0208] ΔH(S) = H(S) - H * (S)

[0209] Where:

[0210] When ΔH(S) > 0, it indicates that the current sorting information entropy is higher than the optimal value, that is, the local orderliness of the data is poor and local correction needs to be performed;

[0211] When ΔH(S) < 0, it indicates that the current sorting information entropy is lower than the optimal value, that is, the data has been over-sorted and global adjustment needs to be performed;

[0212] When ΔH(S) ≈ 0, the sorting error is within the acceptable range and no adjustment is required.

[0213] 3. Select the minimum adjustment point

[0214] 3.1 Calculate the local entropy gradient

[0215] To avoid global re-sorting, in this embodiment, the local entropy gradient G(s i ) is calculated to determine the minimum adjustment point:

[0216]

[0217] Where:

[0218] If G(s i ) is larger, it indicates that this data point contributes more to the global sorting error and should be adjusted first;

[0219] If G(s i ) is smaller, it indicates that this data point contributes less to the global sorting error and can be skipped.

[0220] 4. Adopt the minimum adjustment strategy for sorting correction

[0221] 4.1 Local optimal swap

[0222] For the set of data points with larger sorting errors In this embodiment, the minimum swap principle is adopted for local correction, that is, only the data points within S′ are adjusted to make the overall sorting information entropy converge in the optimal direction.

[0223] Let s i and s jare adjacent data points and satisfy:

[0224] G(s i ) > G(s j ), and

[0225] then perform the swap:

[0226] (s i , s j ) ← (s j , s i )

[0227] until:

[0228]

[0229] where ∈ is the set local entropy error threshold.

[0230] 4.2 Recursive Correction of Global Sorting

[0231] If the local adjustment still cannot make ΔH(S) converge to 0, then adopt the recursive correction strategy:

[0232] 1. First-layer correction: Perform local swapping on the high entropy gradient region;

[0233] 2. Second-layer correction: If the error is still large after the first-layer adjustment, perform cross-partition swapping;

[0234] 3. Third-layer correction: If there is still a large global error, then combine the hierarchical recursive optimal swapping recursive structure to re-adjust the optimal swapping path and finally converge to the optimal sorting state.

[0235] 5. Computational Complexity Analysis

[0236] 1. Local entropy calculation: O(n) (only calculate the entropy difference, without traversing all data points)

[0237] 2. Local swapping correction: O(k log k) (where k is the number of data points to be corrected, much smaller than n)

[0238] 3. Recursive correction sorting: O(log n) (only executed for the high entropy gradient region)

[0239] Overall, the overall time complexity of dynamic entropy difference correction is close to O(n), and its computational cost is greatly reduced compared with the traditional global re-sorting method.

[0240] In this step, a dynamic entropy difference correction strategy is adopted to calculate the sorting error, select the minimum adjustment point for local exchange correction, and combine the recursive correction method to make the final sorting result closer to the globally optimal sorting state. On the premise of ensuring that the computational complexity is lower than linear growth, the sorting stability and accuracy of large-scale data sets are further improved.

[0241] Specifically, Step 5 includes:

[0242] After extracting the global information of data sorting through global entropy reduction sampling in Step 1, Step 2 adopts an adaptive partitioning strategy to optimize data load balancing, Step 3 reduces the local sorting calculation cost through the optimal exchange strategy, and Step 4 uses dynamic entropy difference correction to reduce the global sorting error. However, in a real-time data stream environment, the introduction of new data may cause changes in the existing sorting structure. If the traditional method is used to re-execute the global sorting, it will bring extremely high computational costs. Therefore, this step designs a streaming topological order incremental sorting, so that when the data stream is updated, the overall sorting stability and efficiency can be maintained only through local topological adjustment.

[0243] 1. Data increment modeling

[0244] Let the current existing sorted sequence be S t ={s1, s2,..., s n}, the newly added data stream is ΔS = {s n+1 ,..., s n+m}, and the overall data sequence becomes S t+1 = S t ∪ΔS.

[0245] The goal of this embodiment is to keep S t+1 sorting correctly and minimize the adjustment overhead without re-executing the global sorting.

[0246] To this end, this embodiment constructs a dynamic sorting topology graph of the data stream:

[0247] Define a directed graph G t =(V, E), where:

[0248] The vertex set v corresponds to the ordered data points in the data element S t .

[0249] The edge set E represents the relative order constraint between data points. For example, (s i , s j ) ∈ E means s i ≤ s j .

[0250] When the new data point s kWhen ∈ΔS arrives, the streaming topological order incremental sorting needs to dynamically update G t And adjust the sorting of the affected local area.

[0251] 2. Topological Insertion

[0252] 2.1 Calculate the relative order of the newly added data

[0253] For the newly added data stream ΔS, in this embodiment, their relative order in the existing sorting sequence S t is first calculated:

[0254] 1. Calculate the entropy contribution degree H(s k relative to the existing sorting sequence S t ): k )

[0255] H(s k ) = -P(s k ) log p(s k )

[0256] where p(s k ) is estimated from the previous global entropy reduction sampling result, representing the theoretical relative weight of s k in the global sorting.

[0257] 2. According to H(s k ), calculate the insertion point I(s k ) of s t in S k :

[0258]

[0259] This formula ensures that the data is inserted into the most appropriate local area without affecting the global sorting stability.

[0260] 2.2 Construct local topological dependencies

[0261] For each s k ∈ΔS:

[0262] Find the closest adjacent data points s t and s prev in G next :

[0263] s prev : The largest data point in S t that is less than s k .

[0264] s next : The smallest data point in S t that is greater than s k .

[0265] Add (s t in G prev , s k ) edge(s k , s next ) and update the topological order relationship.

[0266] 3. Recursive Topological Adjustment

[0267] 3.1 Topological Consistency Detection

[0268] Since G t is dynamically modified, it is necessary to ensure that the newly added data does not destroy the consistency of the global topological sorting.

[0269] Define the topological sorting inconsistency degree D(G t ):

[0270]

[0271] Where:

[0272] R(s i ) is the topological sorting index of the data point s i in the current sorting.

[0273] R(s j ) is the topological sorting index of the data point s j in the current sorting.

[0274] If D(G t ) = 0, the topological structure is consistent and no adjustment is required;

[0275] If D(G t ) > 0, local topological adjustment is required.

[0276] 3.2 Recursive Topological Correction

[0277] If the newly added data point causes D(G t ) > 0, this embodiment uses recursive topological adjustment to only correct the local sorting:

[0278] 1. Calculate the affected subgraph: Define the affected subset that only contains the local nodes affected by the new data;

[0279] 2. Perform topological sorting correction:

[0280] Reorder G' using the minimum swap principle;

[0281] Calculate the local entropy gradient G(s i ), and preferentially adjust the high-gradient nodes:

[0282]

[0283] Only adjust the internal topology of G′, without affecting the global sorting;

[0284] 3. Update the global sequence to ensure that the new sequence satisfies topological consistency.

[0285] 4. Computational complexity analysis

[0286] 1. Insertion point search: O(log n) (binary search for the optimal insertion point)

[0287] 2. Topology update: O(1) (only modify the local directed graph)

[0288] 3. Recursive topology adjustment: O(k log k) (sort the affected subgraph, where k << n)

[0289] Overall, the streaming topological order incremental sorting avoids global re - sorting, and its computational complexity is O(log n), which significantly reduces the computational cost compared with the traditional method O(n log n).

[0290] This step proposes a streaming topological order incremental sorting. When the data stream is updated, by constructing a dynamic sorting topological graph, combining local topological insertion and minimum adjustment strategies, it efficiently maintains the order of the data sequence without re - executing the global sorting, thus ensuring the sorting efficiency and stability in a large - scale streaming data environment.

[0291] Embodiment 2

[0292] The present invention discloses a mathematical algorithm computing system based on big data, including:

[0293] Global entropy reduction sampling module: Calculate the local entropy density of the data set, construct an entropy gradient field, perform adaptive weighted sampling based on the entropy gradient value, and adopt a multi - level recursive sampling strategy to extract a data subset that retains global sorting information;

[0294] Non - uniform partition mapping module: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce a partition adjustment mechanism driven by entropy difference, dynamically optimize the data partition boundary, allocate more partitions to areas with high data density, construct a non - uniform partition mapping, and combine a variable - length hash index to improve the query efficiency;

[0295] Hierarchical recursive optimal swap sorting module: In each non - uniform partition, construct the relative order relationship of data points, calculate the optimal swap degree, generate a swap graph and use the minimum - cost maximum - flow method to calculate the optimal swap sequence, and use the recursive hierarchical structure and jump step - size strategy to reduce the number of data exchanges and global adjustments, achieving local sorting with sub - linear time complexity;

[0296] Dynamic Entropy Difference Correction Module: After sorting is completed, calculate the sorting error, and through the dynamic entropy difference adjustment strategy, select the minimum adjustment point for local optimal exchange or recursively correct the global sorting to further optimize the sorting result;

[0297] Streaming Topological Order Incremental Sorting Module: For the real-time data stream update scenario, construct a dynamic sorting topology graph, calculate the relative order of the newly added data and determine the insertion point, and only adjust the affected local area through local topology dependency update and recursive topology correction to avoid global re-sorting.

[0298] The module in Embodiment 2 is used to implement the functions in Embodiment 1. This embodiment can be implemented by a system, which includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the optimization method for calculating based on the mathematical algorithm of big data according to Embodiment 1 of the present application is implemented. The system also includes other components well-known to those skilled in the art such as a communication bus and a communication interface, and their settings and functions are known in the art, so they will not be described in detail here.

[0299] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing the calculation of a mathematical algorithm based on big data, characterized in that, Including: Step 1: Calculate the local entropy density of the dataset, construct the entropy gradient field, perform adaptive weighted sampling based on the entropy gradient value, and adopt a multi-level recursive sampling strategy to extract a data subset that retains the global sorting information; Step 2: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce an entropy difference-driven partition adjustment mechanism, dynamically optimize the data partition boundary, allocate more partitions to regions with high data density, construct a non-uniform partition mapping, and combine a variable-length hash index to improve the query efficiency; Step 3: Within each non-uniform partition, construct the relative order relationship of data points, calculate the optimal swap degree, generate a swap graph and use the minimum cost maximum flow method to calculate the optimal swap sequence, and utilize the recursive hierarchical structure and jump step strategy to reduce the number of data exchanges and global adjustments, achieving local sorting with a sub-linear time complexity; Step 4: After the sorting is completed, calculate the sorting error, and through the dynamic entropy difference adjustment strategy, select the minimum adjustment point for local optimal swapping or recursively correct the global sorting to further optimize the sorting result; Step 5: For the real-time data stream update scenario, construct a dynamic sorting topology graph, calculate the relative order of the newly added data and determine the insertion point, and through local topology dependency update and recursive topology correction, only adjust the affected local area to avoid global re-sorting.

2. The optimization method for mathematical algorithm calculation based on big data according to claim 1, characterized in that The said Step 1 includes: Estimate the probability density of data points, and then calculate the global information entropy of the dataset to obtain the overall information characteristics of the dataset; Determine the sorting neighborhood of data points, calculate the local entropy contribution degree of each data point, and measure its influence on the global sorting information in the local area; Calculate the entropy gradient vector of each data point to determine the importance of the data point in the sorting space; Based on the entropy gradient field information, perform weighted sampling according to the magnitude of the entropy gradient value, so that regions with drastic sorting changes are selected with a higher probability; Adopt a multi-level recursive sampling strategy, perform local refinement and recursive iteration after the initial sampling until the entropy stability condition is met, and finally extract the key data subset that retains the global sorting information.

3. The optimization method for mathematical algorithm calculation based on big data according to claim 2, characterized in that The said Step 2 includes: Set the original data range, divide the high information entropy data subset obtained by global entropy reduction sampling into initial equal-width intervals, and calculate the local information entropy of each partition to measure the data complexity of each partition; Introduce an entropy difference-driven mechanism, calculate the entropy difference between adjacent partitions, and reflect the change of information complexity between partitions; If the partition information entropy is higher than a multiple of the average entropy, split the partition into two new partitions and calculate the new partition boundary to make the entropy values of the sub-partitions close; If the partition information entropy is lower than a multiple of the average entropy, try to merge with adjacent low-entropy partitions to reduce the number of partitions; After adjustment, obtain an optimized set of non-uniform partitions, calculate the non-uniform partition mapping function to ensure that data points can be correctly classified into the adjusted partitions; Construct a dynamically adjusted hash range for each partition to form a variable-length hash index.

4. The optimization method for calculating a mathematical algorithm based on big data according to claim 3, wherein The said Step 3 includes: Within each non-uniform partition, construct a relative relationship matrix between data points, record the relative order relationship of data points, and make inferences in combination with local entropy information and partition boundaries; Define the optimal exchange degree of the data point in the current partition, and measure the exchange priority of the data point in the local sorting. The smaller the optimal exchange degree value, the closer the data point is to the optimal sorting position. Construct a data exchange graph, where vertices represent data points, edges represent optimal exchange paths, and edge weights are exchange costs. The smaller the cost, the higher the priority. Select the path with the smallest exchange cost, exchange data in the order of the path, and minimize the sorting error; The exchange level is recursively divided. The first level performs preliminary exchange on the entire partition. The second level performs local refinement on the data points with large exchange errors. If there is still an error, recursive adjustment is continued until the convergence condition is met.

5. The optimization method for mathematical algorithm calculation based on big data according to claim 4, wherein The step 4 comprises: Based on the local sorting stability model, the relative position information probability of the data points in the sorted state is estimated, and then the information entropy of the current sorting sequence is calculated; According to the global entropy reduction model, the theoretical optimal sorting entropy is calculated as the target value of sorting correction; The dynamic entropy difference is defined as the difference between the current sorting information entropy and the theoretical optimal sorting entropy, which is used to measure the sorting error. If the entropy difference is greater than zero, it means that the local order is poor and needs local correction. If it is less than zero, it means over-sorting and global adjustment is required. If it is equal to zero, no adjustment is required. Calculate the local entropy gradient, determine the data point that contributes most to the global sorting error as the minimum adjustment point, and make adjustments first; For data point sets with large sorting errors, the minimum exchange principle is used for local correction. By exchanging adjacent data points, the overall sorting information entropy converges to the optimal direction. If the entropy difference still cannot converge to zero after local adjustment, a recursive correction strategy is adopted to perform cross-partition exchanges and readjust the optimal exchange path in turn until it converges to the optimal sorting state.

6. The optimization method for calculating a mathematical algorithm based on big data according to claim 5, characterized in that, The step 5 comprises: Data incremental modeling: Let the current sorting sequence be S and the newly added data stream be ΔS. The goal is to maintain the correctness of the overall sorting through dynamic adjustment without re-performing the global sorting. Construct a dynamic sorting topology graph G = (V, E) of the data stream, where the vertex set V corresponds to the ordered data points and the edge set E represents the relative order constraint. Calculate the entropy contribution of the newly added data point relative to the existing sorted sequence and determine its insertion point in S; For each newly added data point, find the closest adjacent data point, add the corresponding edge in the topological graph, and update the local topological order relationship; Calculate the inconsistency of topological sorting to determine whether the new data destroys the consistency of global topological sorting; If there is any inconsistency, only the affected local subgraph is recursively adjusted, reordered using the minimum exchange principle, the local entropy gradient is calculated, and the nodes with higher gradients are adjusted first to ensure that the global sequence meets the topological consistency.

7. A mathematical algorithm computing system based on big data, characterized in that, include: Global entropy reduction sampling module: calculates the local entropy density of the data set, constructs the entropy gradient field, performs adaptive weighted sampling based on the entropy gradient value, and adopts a multi-level recursive sampling strategy to extract a data subset that retains the global sorting information; Non-uniform Partition Mapping Module: Based on the representative data subset with high information entropy, calculate the initial partition mapping, introduce an entropy difference-driven partition adjustment mechanism, dynamically optimize the data partition boundary, allocate more partitions to regions with high data density, construct a non-uniform partition mapping, and combine a variable-length hash index to improve query efficiency; Hierarchical Recursive Optimal Swap Sorting Module: Within each non-uniform partition, construct the relative order relationship of data points, calculate the optimal swap degree, generate a swap graph and use the minimum-cost maximum-flow method to calculate the optimal swap sequence, and utilize the recursive hierarchical structure and jump step strategy to reduce the number of data exchanges and global adjustments, achieving local sorting with sub-linear time complexity; Dynamic Entropy Difference Correction Module: After sorting is completed, calculate the sorting error, and through the dynamic entropy difference adjustment strategy, select the minimum adjustment point for local optimal swapping or recursively correct the global sorting to further optimize the sorting result; Streaming Topological Order Incremental Sorting Module: For the scenario of real-time data stream updates, construct a dynamic sorting topological graph, calculate the relative order of the newly added data and determine the insertion point, and through local topological dependency updates and recursive topological corrections, only adjust the affected local area to avoid global re-sorting.

Images