A multi-dimensional fragmentation logic organization management system for engineering cost consultation business data

Abstract

The application relates to the technical field of database distributed storage, and discloses a multi-dimensional sharding logic organization management system for engineering cost consultation business data, which comprises the following units: a multi-dimensional feature analysis unit for extracting time, space and material feature components; a logic dimension reduction mapping unit for mapping the feature components to a binary addressing space and performing bit level concatenation according to dimension weights to generate an initial sharding key; a bit level staggered matrix control unit for monitoring storage node write rates, adjusting a bit extraction step length according to a load gradient to correct a bit level staggered matrix; and a sharding reconstruction routing unit for performing bit offset processing on the initial sharding key according to the corrected matrix, generating a derived sharding key and routing data. The application can reduce the bit extraction step length, make the aggregated data generate address offsets, share the write pressure of physical nodes, maintain the correlation of multi-dimensional business, and solve the storage blocking risk under the sudden write working condition.

Description

Technical Field

[0001] This invention belongs to the field of database distributed storage technology, and in particular relates to a multi-dimensional sharding logical organization and management system for engineering cost consulting business data. Background Technology

[0002] Current data sharding technology is used to distribute the storage pressure and access load of massive amounts of data. It maps business data to different physical storage nodes through a preset sharding key. When processing engineering cost consulting business data, existing solutions usually use single-dimensional hash algorithms or fixed-range partitioning methods. These methods have good access efficiency when processing datasets with uniform distribution and simple query requirements. However, engineering cost business data has strong multi-dimensional correlation attributes, covering pricing cycles in the time dimension, project topology in the spatial dimension, and equipment classification in the material dimension. Since the sharding key can only anchor a single dimension, cost data entities with high dimensional correlation are in a discrete distribution state in physical space. When performing joint query tasks such as cross-project audit or long-term price comparison, the discrete data distribution causes query commands to be sent to all nodes in the cluster, resulting in a network broadcast storm.

[0003] Besides physical hardware limitations such as the shape of the roller, there are also defects at the logic control level. For example, Chinese invention patent application CN121255412A discloses a distributed block storage method. It calculates the load index by collecting multi-dimensional performance indicators such as storage node processor utilization and disk input / output per second. When the load exceeds the standard, it triggers hot data sharding migration. The solution relies on hardware performance feedback lag indicators. The generation of engineering cost data has business-driven burst aggregation characteristics. Instantaneous increments are accompanied by high-frequency multi-dimensional related access. The migration mechanism should deal with system redundancy and access delay caused by frequent triggering of metadata rebalancing due to business peaks. The migration does not consider the business dimension relationship. The distributed state cannot solve cross-dimensional query network broadcast, which leads to a mismatch between mapping logic, post-remediation and business rules.

[0004] Therefore, how to construct a dynamic routing mapping mechanism that takes into account both the multidimensional business logic correlation and the underlying physical storage balance, and achieve load convergence of massive non-uniform cost data on discrete physical nodes while maintaining data logic aggregation, has become the technical problem to be solved by this invention. Summary of the Invention

[0005] This invention aims to solve the problems of low efficiency in multidimensional joint queries and overload of single-point physical writes when processing non-uniformly distributed multidimensional cost data based on single-dimensional mapping or static range partitioning strategies, due to the inability to maintain the logical locality of the data.

[0006] In this technical solution, a multi-dimensional fragmented logical organization and management system for engineering cost consulting business data includes:

[0007] The multidimensional feature parsing unit is used to extract time feature components representing the pricing cycle, spatial feature components representing the project level, and material feature components representing the cost category from the input engineering cost data entity.

[0008] The logical dimensionality reduction mapping unit is used to map the time feature components, spatial feature components and material feature components to a binary logical address space with a defined total bit width, and to concatenate the bits of each feature component according to the preset dimensional association weight to generate the initial fragmentation key.

[0009] The bit-level interleaving matrix control unit is used to monitor the instantaneous write rate of the storage node and, based on the load gradient of the instantaneous write rate relative to a preset write threshold, adjust the bit extraction step size to correct the bit-level interleaving matrix.

[0010] The fragmentation reconstruction routing unit is used to perform logical address shifting processing on the initial fragmentation key according to the corrected bit-level interleaving matrix, so that the initial fragmentation key generates an address offset to generate a derived fragmentation key that represents the split of the address space, and routes the engineering cost data entity to the logical fragment corresponding to the derived fragmentation key; wherein, the bit-level interleaving matrix control unit reduces the bit extraction step size of the target feature dimension, so that the initial fragmentation key generates an address offset when performing logical address shifting processing, thereby changing the distribution state of the engineering cost data entity among the storage nodes.

[0011] Preferably, when the multi-dimensional feature parsing unit performs feature extraction, it includes the following steps: Step 21: Identify the timestamp from the metadata of the engineering cost data entity and map it to a time feature component representing the pricing cycle; Step 22: Extract the project level attribute of the engineering cost data entity and map it to a spatial feature component representing the project level; Step 23: Map the material attribute of the engineering cost data entity to a material feature component representing the cost category according to the industry classification code table.

[0012] Preferably, when the logical dimensionality reduction mapping unit performs bit concatenation, the preset dimensionality association weight determines the bit distribution ratio of each feature component in the initial partitioning key.

[0013] Preferably, the bit-level interleaving matrix control unit internally stores a basic bit extraction matrix, which defines the initial bit extraction step size for each feature component.

[0014] Preferably, the logic of the bit-level interleaving matrix control unit for adjusting the bit extraction step size is as follows: Step 51: Calculate the deviation of the instantaneous write rate from the preset write threshold to determine the load gradient; Step 52: When the load gradient exceeds the preset gradient threshold, determine the target feature dimension that causes data aggregation; Step 53: For the target feature dimension, reduce the corresponding sampling frequency in the basic bit extraction matrix to achieve dynamic increase processing of the bit extraction step size.

[0015] Preferably, when the fragmentation reconstruction routing unit performs logical address offset processing, it extracts the step size of the corrected bit as the offset parameter and performs binary mask operation on the initial fragmentation key to generate a derived fragmentation key that represents the nonlinear splitting of the address space.

[0016] Preferably, the sharding reconstruction routing unit also includes a mapping cache module, which records the conversion correspondence between the initial sharding key and the derived sharding key, so that subsequent query commands can locate logical shards.

[0017] Preferably, the system also includes a feedback adjustment unit for monitoring the load balance of each storage node, and after the load balance reaches a preset balance threshold, the instruction bit-level interleaving matrix control unit resets the bit extraction step size to the initial bit extraction step size.

[0018] Preferably, logical sharding maintains the business relevance of multidimensional data through derived sharding keys, so that when cross-dimensional query commands access storage nodes, they are restricted to a subset of storage nodes with specific addressing prefixes, thereby achieving local convergence of query requests in the distributed network.

[0019] Compared with existing technologies, the multi-dimensional fragmented logical organization and management system for engineering cost consulting business data of this invention has the following advantages:

[0020] 1. In the multidimensional sharding logical organization and management, the synergistic effect of the multidimensional feature parsing unit and the logical dimensionality reduction mapping unit enables the logical aggregation of high-dimensional attribute data in a one-dimensional logical addressing space. By adopting an interleaved positional algorithm with dynamic weighting factors, the originally independent time dimension feature values, spatial topology feature values, and material feature values ​​are converted into alternating binary sequences, so that cost data entities with business relevance remain physically adjacent on the logical sharding. When processing multidimensional related query tasks across projects and cycles, the query command only needs to be retrieved within the adjacent logical address range, effectively reducing the network broadcast overhead between distributed cluster nodes and solving the global scanning problem caused by traditional single-dimensional sharding strategies when facing multidimensional joint queries.

[0021] 2. The system constructs a dynamic feedback adjustment mechanism driven by the physical write frequency. The write load detection unit counts the command frequency of each physical storage node in real time, and the dynamic topology reconstruction unit actively intervenes in the interleaving control matrix when the frequency exceeds the safe baseline frequency. This mechanism reduces the bit extraction step size of the target business dimension, so that the subsequently generated aggregated data will automatically undergo topology shift and be mapped to the derived new logical fragment during the logical address generation stage. This ensures that when a massive amount of data is suddenly written in a specific time period or for a specific project, the system can distribute the storage pressure through automatic reconstruction at the logical level, avoiding the risk of write blocking caused by the rapid expansion of the capacity of a single physical storage node.

[0022] 3. This system achieves deep coupling between data distribution locality and load balancing. It no longer relies on preset static sharding rules, but instead uses weight update instructions generated by dynamic interleaving reconstruction units to enable the logical address space to adapt to changes in actual write intensity. Under this mechanism, the data aggregation phenomenon that was originally difficult to predict at the physical level is transformed into controllable overflow and fission within the logical space. The system does not need to perform high-cost physical global data rebalancing tasks to maintain the routing convergence speed of massive non-uniform cost data in a distributed environment, ensuring that the distributed database still has a definite response time when facing extremely uneven data distribution. Attached Figure Description

[0023] Figure 1 This is a flowchart of the multi-dimensional fragmentation and dynamic routing processing of engineering cost data in this invention;

[0024] Figure 2 This is a mapping diagram of the interaction between external business entities and system sharding logic in this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0027] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to logical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0028] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0029] A multi-dimensional fragmented logical organization and management system for engineering cost consulting business data includes:

[0030] The multidimensional feature parsing unit is used to extract time feature components representing the pricing cycle, spatial feature components representing the project level, and material feature components representing the cost category from the input engineering cost data entity.

[0031] The logical dimensionality reduction mapping unit is used to map the time feature components, spatial feature components and material feature components to a binary logical address space with a defined total bit width, and to concatenate the bits of each feature component according to the preset dimensional association weight to generate the initial fragmentation key.

[0032] The bit-level interleaving matrix control unit is used to monitor the instantaneous write rate of the storage node and, based on the load gradient of the instantaneous write rate relative to a preset write threshold, adjust the bit extraction step size to correct the bit-level interleaving matrix.

[0033] The fragmentation reconstruction routing unit is used to perform logical address shifting processing on the initial fragmentation key according to the corrected bit-level interleaving matrix, so that the initial fragmentation key generates an address offset to generate a derived fragmentation key that represents the split of the address space, and routes the engineering cost data entity to the logical fragment corresponding to the derived fragmentation key; wherein, the bit-level interleaving matrix control unit reduces the bit extraction step size of the target feature dimension, so that the initial fragmentation key generates an address offset when performing logical address shifting processing, thereby changing the distribution state of the engineering cost data entity among the storage nodes.

[0034] Preferably, when the multi-dimensional feature parsing unit performs feature extraction, it includes the following steps: Step 21: Identify the timestamp from the metadata of the engineering cost data entity and map it to a time feature component representing the pricing cycle; Step 22: Extract the project level attribute of the engineering cost data entity and map it to a spatial feature component representing the project level; Step 23: Map the material attribute of the engineering cost data entity to a material feature component representing the cost category according to the industry classification code table.

[0035] Preferably, when the logical dimensionality reduction mapping unit performs bit concatenation, the preset dimensionality association weight determines the bit distribution ratio of each feature component in the initial partitioning key.

[0036] Preferably, the bit-level interleaving matrix control unit internally stores a basic bit extraction matrix, which defines the initial bit extraction step size for each feature component.

[0037] Preferably, the logic of the bit-level interleaving matrix control unit for adjusting the bit extraction step size is as follows: Step 51: Calculate the deviation of the instantaneous write rate from the preset write threshold to determine the load gradient; Step 52: When the load gradient exceeds the preset gradient threshold, determine the target feature dimension that causes data aggregation; Step 53: For the target feature dimension, reduce the corresponding sampling frequency in the basic bit extraction matrix to achieve dynamic increase processing of the bit extraction step size.

[0038] Preferably, when the fragmentation reconstruction routing unit performs logical address offset processing, it extracts the step size of the corrected bit as the offset parameter and performs binary mask operation on the initial fragmentation key to generate a derived fragmentation key that represents the nonlinear splitting of the address space.

[0039] Preferably, the sharding reconstruction routing unit also includes a mapping cache module, which records the conversion correspondence between the initial sharding key and the derived sharding key, so that subsequent query commands can locate logical shards.

[0040] Preferably, the system also includes a feedback adjustment unit for monitoring the load balance of each storage node, and after the load balance reaches a preset balance threshold, the instruction bit-level interleaving matrix control unit resets the bit extraction step size to the initial bit extraction step size.

[0041] Preferably, logical sharding maintains the business relevance of multidimensional data through derived sharding keys, so that when cross-dimensional query commands access storage nodes, they are restricted to a subset of storage nodes with specific addressing prefixes, thereby achieving local convergence of query requests in the distributed network.

[0042] Example 1: In a distributed engineering cost consulting business data management environment, the system needs to process non-uniformly distributed data streams involving multiple pricing cycles, multiple project levels, and a large number of material categories. When in a specific settlement cycle, the material pricing data under a single project exhibits an explosive write state. The traditional single-dimensional sharding mechanism cannot take into account the multi-dimensional business correlation, resulting in cost data entities with strong correlations being scattered and stored on heterogeneous physical nodes of the distributed cluster. When performing cross-project audits or horizontal comparison queries of material prices, the system generates network broadcast overhead across the entire cluster. Static range sharding, on the other hand, will cause write overload and write operation blocking on the physical storage nodes that carry aggregated data.

[0043] The multi-dimensional segmented logical organization and management system of this invention operates in this environment. The multi-dimensional feature parsing unit parses the time feature components representing the pricing cycle, the spatial feature components representing the project level, and the material feature components representing the cost category from the real-time entered cost data entities. The logical dimensionality reduction mapping unit converts each feature component into an equal-length binary feature sequence, and extracts binary bits from the binary feature sequences of each dimension according to the preset interleaving control matrix, generating a one-dimensional logical segmentation address corresponding to the engineering cost data entity. The specific one-dimensional logical segmentation address... The mathematical structure is as follows: ,in, These are the discretized dimensional feature values. For the interleaved order weights of the corresponding dimensions, For the index of the feature dimension, and The value range covers time, space, and material dimensions. To ensure lossless parsing of one-dimensional logical fragment addresses and maintenance of multi-dimensional prefix topology by the underlying physical storage nodes, based on the Morton code space-filling curve theory, the logical dimension reduction mapping unit uses a combination of bit shifting and binary logic to replace arithmetic weighting when converting system-level algebraic operations into hardware register-level low-level execution instructions, actually generating low-level binary aggregation instructions. The operation execution logic is limited to ,in, The final binary physical identifier is written after the representation conversion; The binary feature value after discretization of the i-th feature dimension is represented, and the value is in unsigned integer format. This characterizes the feature value of this dimension, which is allocated an independent bit shift offset step size in the set total bit width address space. The representation of bitwise XOR operation, the logical dimensionality reduction mapping unit is achieved by allocating non-overlapping shift offset steps. This approach ensures that each dimension's feature value independently occupies a discrete addressing bit range in the final binary sequence, preventing arithmetic carry from destroying the feature prefix. This addressing method deeply integrates the originally discrete business dimensions at the bit level through a bit-level interleaving mechanism, mapping data entities with similar business attributes to physically adjacent logical addressing spaces. The bit-level interleaving matrix control unit monitors the instantaneous write rate of the storage node. When the instantaneous write rate of the target physical storage node carrying material cost data under specific time and space dimensions reaches a preset threshold of 10000Hz, the bit-level interleaving matrix control unit identifies the target feature dimension that causes data aggregation and reduces the bit extraction step size of the corresponding target feature dimension in the bit-level interleaving matrix to correct the bit-level interleaving matrix.

[0044] The fragmentation and reconstruction routing unit performs logical address shifting on the generated initial fragmentation key based on the corrected bit-level interleaving matrix, generating address offsets and derived fragmentation keys. The specific address splitting and routing reconstruction execution path is constructed based on the consistent hashing mapping principle. The fragmentation and reconstruction routing unit generates a binary feature-preserving mask for the target feature dimension based on the base bit extraction matrix before correction. Perform as equation The defined feature masking operation, wherein, Characterizes the derived fragmentation key; Characterizes the initial slice key; The address offset obtained from load gradient calculation; The bitwise AND operation is used to block and remove the addressing prefix that leads to the node in the current overload state. The bitwise OR operation is used to merge newly generated offset address segments into the remaining shard key architecture. Before performing the aforementioned limited feature masking operation, the specific rules for the system to dynamically generate the target mask based on matrix data are established as follows: scan all logical address slots within the total bit width, write all bypass bits not belonging to the target feature dimension to a binary value of 1, and simultaneously overwrite the exclusive bit segments occupied only by the target feature dimension to a binary value of 0. This generates a structured mask with targeted masking capabilities. After establishing the derived shard key, the shard reconstruction routing unit queries the internal real-time maintenance to ensure consistency of the state variables of all physical storage nodes within the cluster. The global physical topology distribution table of the consistent hash ring structure uses the derived shard key as a location retrieval parameter to explore clockwise along the logical address of the consistent hash ring. It identifies the first physical storage node with working heartbeat feedback and a real-time write load gradient G value less than a preset threshold of 0.05, and establishes a mapping and rights confirmation relationship. This derived shard key guides data flow to other physical storage nodes with lower load levels within the cluster, achieving adaptive splitting of the address space at the logical level. During this physical mapping transfer process, the fixed scalar address offset calculated by the formula actually acts as a step jump factor on the consistent hash ring. Its core physical mechanism lies in forcing the originally congested fixed shard key to be removed from the target node. The bit pointer instantaneously jumps away from the hash-addressed sector mapped by the current high-load physical node. After the positioning pointer completes a fixed-length jump based on the offset, if the first clockwise physical node corresponding to the new landing point still does not meet the gradient threshold condition, the underlying probing mechanism of the distributed cluster starts a linear step scan along the hash ring until it reaches the first load-safe node and solidifies it as the final derived mapping target. The sharding reconstruction routing unit synchronously records the conversion logic between the initial sharding key and the derived sharding key in the mapping cache module, maintaining the local convergence characteristics of cross-dimensional joint query instructions, eliminating network scanning of irrelevant logical shards, and thus solving the storage problem. While maintaining logical consistency in data organization despite node write pressure, the system can still strictly adhere to the query convergence boundary after bitwise OR operations because the effective binary bit width of the address offset is fundamentally constrained within the target feature dimension bit field range that has been masked and cleared. The aforementioned bitwise OR operation only induces local address fission in this single congested dimension, while the bit distribution array in the high bits of the address, representing the service prefixes of other unoverloaded dimensions, is not affected by any level flipping interference throughout the operation. This ensures that multi-dimensional joint retrieval instructions can still rely on these robust shared high-bit prefixes to successfully penetrate and reach the adjacent storage topology domain fissioned from the parent node.

[0045] Example 2: In the functional verification process of the multi-dimensional fragmented logical organization and management system, this experiment constructed a distributed cluster environment containing 16 physical storage nodes. This environment was used to support an anonymized engineering cost dataset covering 1200 under-construction project units and 50,000 material classification dimensions. Gaussian white noise with a signal-to-noise ratio of 20dB was superimposed in the data injection channel to simulate signal jitter in a real industrial network environment. The total bit width of the logical dimensionality reduction mapping unit was set to a certain value. for The process of determining this parameter involves the logical address collision rate. overhead of routing table memory residency The trade-off between the two involves a mutually restrictive engineering contradiction, namely... Increasing the bit width reduces the probability of address collisions but leads to a linear increase in memory usage during route retrieval. Testing on storage nodes with a 128GB physical memory boundary revealed that when the bit width increased from 32 to 64, the collision rate decreased from 0.15 to 0.002, while memory consumption remained within the preset safe load range of 2.4GB. Therefore, 64 was determined as the bit width setting for this scenario, and the sampling period setting in the bit-level interleaving matrix was also determined. The 10ms setting is based on balancing the instantaneous write speed. To ensure the accuracy of the capture and the system monitoring load, and in accordance with the Nyquist sampling criterion, to prevent signal aliasing in the calculation of the load gradient, The reciprocal of the cost data entity arrival frequency must be greater than the cost data entity arrival frequency. Twice that of the data entities, since the highest arrival frequency of data entities in this test scenario is 45Hz, selecting 10ms can meet the real-time sensing requirements of sudden write hotspots.

[0046] After the experiment was started, the sample group of this invention received the real-time input of the steel reinforcement material cost entity. The multi-dimensional feature parsing unit parsed out the time, space and material feature components, and the logical dimension reduction mapping unit associated weights according to the preset dimensions. The binary sequences of each component are concatenated bit by bit to generate a one-dimensional logical fragment address. Control group 1 used a single-dimensional hash sharding method, while control group 2 used a static range sharding method. The test environment simulated the East China region entering the fourth quarter settlement cycle and the instantaneous write rate... When the frequency reached 11500Hz, exceeding the preset threshold of 10000Hz, the CPU utilization of the storage node carrying this business characteristic in control group 2 climbed to 94.22%, accompanied by a write suspension of up to 32.54s. While control group 1 maintained load balancing across physical nodes, during cross-project material unit price comparison queries, the query command initiated a network broadcast scan to all 16 nodes in the cluster, causing the query response latency to deteriorate from the normal 12.65ms to 215.82ms. The bit-level interleaved matrix control unit of this invention detected... After exceeding the threshold, the feature dimension causing the write hotspot is identified as the material feature dimension. Then, the bit extraction step size of the corresponding material dimension in the bit-level interleaving matrix is ​​reduced, so that the initial fragment key calculated by the logical dimension reduction mapping unit generates an address offset and is converted into a derived fragment key.

[0047] Data processing results show that after the bit extraction step size reduction operation is completed, the write operation flow that was originally concentrated on overloaded nodes is reset to the storage nodes with low load gradients. At this time, the overall write load standard deviation of the cluster decreases from 85.64 before adjustment to 4.18, showing a dynamic equilibrium state of writes under non-uniform data distribution. To verify the safety of parameter boundaries, this experiment sets up an out-of-range control group and correlates dimensions with weights. The measured values ​​were set to outliers exceeding the 0-1 range. The data showed that the effective information entropy of the one-dimensional logical shard address decreased by 75.82%, resulting in 65.23% address collisions and overlaps in the logical addressing space. This confirmed the necessity of limiting the weight coefficients and matrix step size range in this invention. Performance analysis confirmed that, in implementing load balancing, the sample group of this invention only accessed two storage nodes with specific addressing prefixes for cross-dimensional joint queries, achieving a 100% local convergence rate. This demonstrates that the bit-level interleaving mechanism, while addressing the write pressure on physical storage nodes, maintains the topological association characteristics of multi-dimensional business attributes in the logical addressing space.

[0048] Example 3: In a distributed storage scenario for provincial cross-regional transportation infrastructure audits, the system faces the objective condition of inconsistent hardware performance among nodes at different prefecture-level cities. Since the disk I / O throughput of some storage nodes is limited to a 6Gb / s serial interface standard, while emerging nodes use non-volatile fast transmission protocol interfaces, the multi-dimensional sharding logical organization and management system determines the threshold for instantaneous write rates through a bit-level interleaved matrix control unit. The determination process includes the load detection unit collecting the basic disk bandwidth parameters of the corresponding storage node under standard idle conditions. and the average capacity of engineering cost data entities The disk saturation warning coefficient is set to 0.85, and the system calculates the theoretical frequency carrying capacity limit of a single node based on physical performance boundaries. Based on mathematical mapping relationships A dynamic judgment criterion for this node was established, and the judgment threshold was calculated for a storage node with a disk bandwidth of 200MB / s and an average size of 20KB for engineering cost data entities. It is 8500Hz.

[0049] After the multi-dimensional feature parsing unit extracts the time feature components, spatial feature components, and material feature components from the real-time incoming engineering cost data entity, the logical dimensionality reduction mapping unit uses the interleaved control matrix... Perform bit extraction and determine the dimensional correlation weights of material feature dimensions. This determines the proportion of the high-order bits in the 64-bit binary logical address space, for instantaneous write rates. Reaching 9200Hz and exceeding the judgment threshold Under overload conditions, the bit-level interleaved matrix control unit identifies the target feature dimension that causes aggregation by extracting the step size of the corresponding dimension's bits in the interleaved control matrix. Revised to To adjust the bit order weights so that the address offset generated by the fragment reconstructing routing unit is... Based on logical structure Establish, among which, The load gradient is based on the formula. Calculations show that The feature bit width offset constant for the fragmentation key, which is a numerical reduction operation by decreasing the bit extraction step size parameter, essentially manifests as a reduction in the frequency of dense bit extraction for the congested dimension feature sequence in the physical execution logic of the underlying data sampling. This reduction in extraction frequency directly leads to the stretching and widening of the physical spacing between consecutively extracted adjacent bits in the original feature sequence during the splicing process. This effectively achieves a dynamic increase in the displacement span of the feature dimension in the final logical fragmentation address space, thus ensuring the logical consistency between the underlying operation reduction mechanism and the upper-layer address expansion effect. The dynamic adjustment of the data organization logic guides the newly added engineering cost data entities to the load gradient. For storage nodes with a value less than 0.05, the physical load and logical address space within the storage cluster are aligned in real time. The write bottlenecks originally caused by differences in physical specifications are offset by the step size correction action of the bit-level interleaving matrix, and the stability and load balance of the multi-dimensional cost data organization logic are maintained in the distributed environment.

[0050] In a storage cluster in a dynamic splitting state, the sharding reconfiguration routing unit maintains a global mapping table. A global mapping table is used to manage the routing pointers of the initial sharding key and derived sharding keys. The data structure includes a logical fragment identifier consisting of a 24-bit high-order address prefix and a corresponding physical address index; when the bit-level interleaving matrix control unit completes the bit extraction step size After the correction and generation of the derived fragmentation key, the fragmentation reconstruction routing unit will combine the high-order prefix of the derived fragmentation key with the corresponding address offset. Write to global mapping table This enables subsequent query commands targeting specific business characteristics to be retrieved from the global mapping table. It directly locates the target logical fragment after the split, eliminating the redundant operation of performing a full scan of the original physical nodes and newly added physical nodes during the address space reconstruction, and realizing the perception and accurate routing of the change in the organizational state of engineering cost data entities.

[0051] Example 4: In the initial deployment of the cross-regional transportation infrastructure project cost management system, due to the varying statistical distribution characteristics of business dimensions such as pricing cycle, geographical coordinates, and material attributes in the data source, the system initiates a pre-calibration procedure to determine the dimension association weights in the logical dimension reduction mapping unit. Specifically, the calibration procedure involves the multi-dimensional feature parsing unit extracting a set of feature vectors from 5,000 pre-stored cost business samples, and the bit-level interleaved matrix control unit calculating the information contribution rate of each feature dimension relative to the overall addressing space. Based on the discrete probability distribution of the binary feature sequences of each dimension, a weight allocation strategy is established so that when the logical dimension reduction mapping unit concatenates bits, the dimension with a specific information entropy weight is allocated to the high bit interval of the initial sharding key. The system establishes the topological occupancy ratio of each business dimension in the logical addressing space through this calibration procedure, thereby achieving a non-clustered distribution of sharding key generation in the early stage of data injection.

[0052] When the system is faced with a task of entering engineering cost data entities that include multi-level bills of materials and dynamic price adjustment coefficients, the multi-dimensional feature parsing unit converts textual material attributes and numerical time components into equal-width 64-bit binary sequences according to the standard discretization ladder. The bit-level interleaved matrix control unit determines the interleaved control matrix through controlled experiments. Bit extraction step size reference value The calibration procedure includes simulating address collision frequencies under different long gradients in a simulation environment. Select to make A step size value lower than 0.001 and a local convergence rate of sharding higher than 0.95 is used as the initial control parameter. Under this parameter setting, the logical dimensionality reduction mapping unit injects the binary sequences of each dimension into the logical addressing address space in bit order. The generated initial sharding key maintains the business association characteristics of the cost entity while realizing the adaptive distribution of the physical write load of the underlying storage node. The system eventually enters a continuous running state in which the multi-dimensional business characteristics and the physical sharding mapping logic are aligned in real time.

[0053] Example 5: In an integrated deployment scenario involving multiple heterogeneous pricing data sources, the system performs text mapping and labeling based on material attribute features. The multi-dimensional feature parsing unit performs word segmentation on 120,000 main material text description fields stored in the historical business database, establishing a mapping set between material text and integer values, i.e., an index dictionary. It is used to characterize the relationship between material text and integer values. The logical dimension reduction mapping unit converts the integer index code into a binary feature sequence with the minimum Hamming distance feature according to the Gray code conversion algorithm, so that the building material categories with high frequency of occurrence are assigned to the core area of ​​the numerical distribution, and the statistical distribution law of data entities is maintained in the generated initial partition key.

[0054] When the system performs weight initialization on the multidimensional address space consisting of pricing cycle, geographical location, and material category, the bit-level interleaved matrix control unit determines the dimension association weights based on the dispersion of each dimension feature in the business sample set. The process of determining this includes calculating the information entropy of the feature sequences in each dimension. Information entropy Used to quantify the The system normalizes the information entropy of each dimension to determine the uncertainty of information distribution in each dimension, and uses this as the initial weight allocation value. The logical dimensionality reduction mapping unit then assigns the weights accordingly. The descending order of binary bits in the bit-level interleaving matrix determines the priority of each dimension. In the generated 64-bit one-dimensional logical fragment address, the material category feature with high information entropy occupies the high bit segment of the address. A monotonic mapping relationship is established between the business feature distribution and the logical addressing topology. The cost data entity generates a local load convergence effect in the physical storage cluster. For the conversion process of the bit width distribution from the floating-point dimension associated weight to the terminal register, the logical dimension reduction system adopts a direct product strategy of rounding down. It performs the product operation of each normalized weight value and the total bit width constant 64 and truncates the decimal places to determine the initial allocation bit width constant. When the total number of bits allocated is less than the preset physical full bit width due to the stacking of rounding remainders, the system instruction will fill all the remaining empty bits into the dimension bit field with the highest information entropy indicator, thereby closing the conversion gap between floating-point parameters and discrete physical storage elements.

[0055] When the instantaneous write rate of the storage node Falling back to the judgment threshold And load gradient When the stable operating baseline of 0.02 is maintained for 300 seconds, the bit-level interleaved matrix control unit initiates a step-size recovery procedure to reduce the redundancy of the logic address space. This procedure involves the bit-level interleaved matrix control unit gradually increasing the interleaved control matrix in increments of 1. Bit extraction step size in the corresponding dimension After each step increase, a 60-second write uniformity monitoring is performed; if the overall write load standard deviation fluctuation of the cluster is found to be less than the preset 0.5 deviation value, the system continues to increase the step size until... After restoring to the initial bit width, the system spontaneously converges to the initial logical organization state with high business relevance after resolving sudden write peaks, maintaining the local convergence characteristics of cross-project cost audit queries while ensuring storage stability and load balance.

[0056] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.

Claims

1. A multi-dimensional fragmented logical organization and management system for engineering cost consulting business data, characterized in that, include: The multidimensional feature parsing unit is used to extract time feature components representing the pricing cycle, spatial feature components representing the project level, and material feature components representing the cost category from the input engineering cost data entity. The logical dimensionality reduction mapping unit is used to map the time feature components, spatial feature components and material feature components to a binary logical address space with a defined total bit width, and to concatenate the bits of each feature component according to the preset dimensional association weight to generate the initial fragmentation key. The bit-level interleaving matrix control unit is used to monitor the instantaneous write rate of the storage node and, based on the load gradient of the instantaneous write rate relative to a preset write threshold, adjust the bit extraction step size to correct the bit-level interleaving matrix. The fragmentation reconstruction routing unit is used to perform logical address shifting processing on the initial fragmentation key according to the corrected bit-level interleaving matrix, so that the initial fragmentation key generates an address offset to generate a derived fragmentation key that represents the split of the address space, and routes the engineering cost data entity to the logical fragment corresponding to the derived fragmentation key; wherein, the bit-level interleaving matrix control unit reduces the bit extraction step size of the target feature dimension, so that the initial fragmentation key generates an address offset when performing logical address shifting processing, thereby changing the distribution state of the engineering cost data entity among the storage nodes; The bit-level interleaving matrix control unit stores a basic bit extraction matrix, which defines the initial bit extraction step size for each feature component. Logical sharding maintains the business relevance of multidimensional data through derived sharding keys, so that cross-dimensional query commands are restricted to a subset of storage nodes with specific addressing prefixes when accessing storage nodes, thereby achieving local convergence of query requests in a distributed network.

2. The multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 1, characterized in that, When the multidimensional feature parsing unit performs feature extraction, it includes the following steps: Step 21: Identify the timestamp from the metadata of the engineering cost data entity and map it to a time feature component representing the pricing cycle; Step 22: Extract the project-level attributes of the engineering cost data entity and map them to a spatial feature component representing the project level; Step 23: Map the material attributes of the engineering cost data entity to a material feature component representing the cost category according to the industry classification code table.

3. The multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 1, characterized in that, When the logical dimensionality reduction mapping unit performs bit concatenation, the preset dimensionality association weight determines the bit distribution ratio of each feature component in the initial partitioning key.

4. The multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 1, characterized in that, The logic of the bit-level interleaving matrix control unit to adjust the bit extraction step size is as follows: Step 51: Calculate the deviation of the instantaneous write rate from the preset write threshold to determine the load gradient; Step 52: When the load gradient exceeds the preset gradient threshold, determine the target feature dimension that causes data aggregation. Step 53: For the target feature dimension, reduce the corresponding sampling frequency in the basic bit extraction matrix to achieve dynamic increase of the bit extraction step size.

5. A multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 1, characterized in that, When the fragmentation reconstruction routing unit performs logical address shift processing, it extracts the step size of the corrected bits as the offset parameter and performs binary mask operation on the initial fragmentation key to generate a derived fragmentation key that represents the nonlinear splitting of the address space.

6. A multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 1, characterized in that, The sharding reconstruction routing unit also includes a mapping cache module, which records the conversion correspondence between the initial sharding key and the derived sharding key, so that subsequent query commands can locate logical shards.

7. A multi-dimensional fragmented logical organization and management system for engineering cost consulting business data according to claim 6, characterized in that, The system also includes a feedback adjustment unit, which monitors the load balance of each storage node. After the load balance reaches a preset balance threshold, the instruction bit-level interleaving matrix control unit resets the bit extraction step size to the initial bit extraction step size.

Images