A dynamic data management method oriented to business complexity

Abstract

This invention relates to the field of database technology and discloses a dynamic data management method oriented towards business complexity. This method parses business logs to construct a real-time business interaction topology graph, calculates the structural entanglement degree representing logical coupling and the access discrete entropy representing the degree of hotspot dispersion, and generates a business state vector. It uses a time series model to predict future states, employs a recursive cost function considering lock blocking diffusion and resource contention effects to calculate execution costs, and generates policy instructions when the net utility meets a lag threshold. Responding to the instructions, it executes dynamic materialized views or transient vertical splitting operations including hot / cold separation, and performs nonlinear adaptive synchronous flow control based on real-time load. This invention quantifies business complexity through graph theory features and combines a predictive decision-making mechanism to achieve proactive data architecture optimization, effectively solving the performance degradation problem in highly coupled and hotspot scenarios.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a dynamic data management method oriented towards business complexity. Background Technology

[0002] As enterprise business logic becomes increasingly complex and its iteration speed accelerates, the underlying data storage architecture often struggles to adapt to changes in upper-layer applications. In real-world production environments, business systems typically face challenges such as fluctuating data access patterns, high coupling of transaction logic, and sudden hotspot access. Database administrators or automated operations and maintenance platforms need to adjust the data architecture promptly, such as creating views, splitting table structures, or adjusting indexes, to maintain system service levels.

[0003] Most existing database performance management methods rely on monitoring basic resource metrics such as CPU utilization, memory usage, or disk I / O throughput. This resource saturation-based approach often only reflects the result of system performance degradation, failing to delve into the root causes of business logic bottlenecks. For example, when a system lags, traditional static metadata analysis struggles to distinguish whether it's due to insufficient hardware resources, excessive multi-table joins in business transactions, or uneven data distribution leading to localized hotspot contention. Due to the lack of quantitative methods for analyzing the real-time interaction topology between business logic and data nodes, optimization strategies often lack specificity and struggle to accurately pinpoint structural bottlenecks.

[0004] Furthermore, existing automated optimization solutions typically employ a passive response mechanism, triggering adjustments only after the system load has exceeded a threshold. This delayed response poses risks in high-concurrency scenarios. More critically, existing decision-making mechanisms generally lack dynamic assessment of the cost of strategy execution. Performing data reconstruction operations (such as creating materialized views or splitting data tables) under high system load consumes significant computing resources and generates database locks. If the lock blocking and resource contention effects caused by modifying data structures are ignored, blindly executing optimization operations can lead to a sharp deterioration in system performance in the short term, and may even trigger service avalanches or repeated policy rollbacks and redoings.

[0005] During data reconstruction, data synchronization between the original and new data nodes is a crucial step. Existing data synchronization mechanisms typically employ fixed concurrency or rate limits, lacking the ability to detect real-time load in the production environment. During peak business periods, fixed synchronization tasks can easily compete with online business transactions for network bandwidth and disk I / O resources, leading to timeouts or service jitter in normal business request responses, making it impossible to achieve a smooth data architecture evolution while ensuring business continuity. Summary of the Invention

[0006] The purpose of this invention is to provide a dynamic data management method oriented towards business complexity, which solves the problems of existing technologies lacking real-time quantification of business logic coupling and data hotspot distribution, and lacking dynamic evaluation of execution costs and lock blocking effects in optimization decisions, thus causing system oscillations under high load.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A dynamic data management method oriented towards business complexity includes the following steps:

[0009] Construct a real-time business interaction topology diagram. The end-to-end awareness module monitors the transaction processing flow of the target business system and parses business logs to identify the set of data nodes accessed in a single transaction. Data node identifiers are extracted according to a preset granularity configuration, and the node set is stored using a hash-mapped adjacency list structure. By extracting global tracking identifiers or transaction session identifiers, operations with the same identifier are aggregated into atomic transactions.

[0010] When multiple data nodes are detected being accessed in the same atomic transaction, a business dependency relationship is determined between the nodes, and a directed connection edge is established. The weight update of the connection edge adopts an exponentially weighted moving average algorithm, which is based on the historical weight value of the previous time step and the normalized co-occurrence frequency of the current monitoring period, and is weighted and synthesized through a preset forgetting factor to dynamically reflect the changing trend of business logic.

[0011] Business complexity is quantified based on topological features. Graph theory features of the real-time business interaction topology are extracted to calculate the structural entanglement index, which represents the degree of coupling of business logic, and the access discrete entropy index, which represents the degree of dispersion of data hotspots.

[0012] The structural entanglement index is calculated based on the node's in-degree, out-degree, and adjacent edge weights. The specific process is as follows:

[0013] We calculate the weighted sum of the in-degree and out-degree of each data node to obtain the weighted sum of node connectivity. We then calculate the sum of the weights of all adjacent edges of each data node and perform a logarithmic operation to obtain the weight strength factor. Finally, we take the average of the product of the weighted sum of node connectivity and the weight strength factor as the final indicator. This indicator uses logarithmic operations to balance the influence of high-weight edges and quantifies the depth of business logic's dependence on data.

[0014] The access discrete entropy metric is calculated based on the Shannon entropy principle. By acquiring the underlying storage engine metadata, the data storage space is divided into multiple logical partitions. The number of access requests falling into each logical partition within the current time window is counted, and the access percentage probability is calculated. The expected value of the self-information of this access percentage probability is used as the access discrete entropy metric. When this metric approaches the logarithm of the number of logical partitions, the system is determined to be in a random access state.

[0015] Generate a business state vector and predict its trend. Merge the structural entanglement index, access discrete entropy index, and basic load index to generate the current business state vector. Obtain a historical business state vector sequence, construct a time sliding window sequence, normalize it, input it into a pre-trained time series prediction model, and output the predicted business state vector for the future preset time window.

[0016] Strategy decision-making is based on a recursive cost compensation mechanism. The theoretical performance benefits of candidate strategies for predicting business state vectors are evaluated; simultaneously, the strategy execution cost is calculated using a recursive cost function with the current structural entanglement index and basic load index as independent variables.

[0017] When calculating the execution cost of a strategy, the baseline is the basic resource consumption of the candidate strategy under no-load conditions:

[0018] The current structural entanglement index is used as an exponential term to calculate the first compensation factor, representing the lock blocking diffusion effect caused by modifying the data structure. The current basic load index is used as a linear term to calculate the second compensation factor, representing the resource contention effect under high load. The final strategy execution cost is determined by the product of the basic resource consumption and the two compensation factors mentioned above. A strategy switching instruction is generated only when the difference between the theoretical performance gain and the strategy execution cost exceeds a preset start threshold; the strategy is revoked when the net gain after revoking the strategy is lower than the stop threshold.

[0019] In response to the strategy switching command, perform data reconstruction operations.

[0020] When the instruction aims to reduce the structural entanglement index, it identifies strongly connected subgraphs in the topology graph where the connection weights exceed the aggregation threshold, extracts the relevant data table schema definitions, and constructs a dynamic materialized view through multi-table join logic. A change data capture task is initiated to synchronize changed data from the original data nodes to the dynamic materialized view to maintain consistency. Simultaneously, query rewriting rules are maintained at the database proxy layer to redirect query requests matching the view's coverage to the dynamic materialized view.

[0021] When the instruction aims to improve the access discrete entropy metric, the target data table's field set is divided into hot field groups and cold field groups based on field-level access statistics. A main table containing the hot field groups and an extension table containing the cold field groups are created at the physical storage layer, linked by a primary key. When processing update requests, if only the main table fields are being updated, only the main table rows are locked, allowing concurrent reads of the extension table fields.

[0022] An adaptive flow control mechanism is employed during data synchronization. Basic load metrics are periodically acquired, and their ratio to a system-defined load threshold is calculated. A preset flow shaping factor is used to perform a non-linear calculation on this ratio to determine the maximum permissible data synchronization rate. This rate decreases non-linearly as the basic load metrics increase, to avoid impacting online service performance during data reconstruction.

[0023] In summary, the present invention has at least one of the following beneficial technical effects:

[0024] 1. This invention constructs a real-time business interaction topology graph and calculates structural entanglement and access discrete entropy indices, enabling precise quantification of system complexity from two dimensions: business logic coupling and data access distribution. Unlike traditional optimization methods based on static metadata, this scheme utilizes graph theory features to capture dynamic dependencies between transactions in real time and uses entropy principles to identify hidden data hotspots. This allows the system to distinguish between performance bottlenecks caused by complex business logic and simple resource contention, thus providing accurate decision-making basis for subsequent data architecture adjustments.

[0025] 2. This invention introduces a decision-making mechanism combining time series forecasting and a recursive cost function, effectively mitigating the risks of strategy execution under high load. By predicting future business state vectors, the system achieves a shift from passive response to proactive management. The recursive cost function specifically considers the lock-blocking diffusion effect caused by structural entanglement and the resource contention effect caused by basic load, ensuring that reconfiguration is triggered only when the theoretical benefit covers the dynamic execution cost. This dual-threshold lag comparison mechanism avoids frequent oscillations in the system under critical states, guaranteeing the net utility of strategy switching.

[0026] 3. This invention provides a multimodal execution strategy that includes dynamic materialized views and transient vertical partitioning, and combines this with an adaptive flow control mechanism to ensure production stability. Materialized views are established for highly coupled scenarios to reduce multi-table join overhead, and vertical partitioning is performed for high-hotspot scenarios to isolate lock contention, achieving targeted solutions. Simultaneously, during the data reconstruction synchronization process, a nonlinear decay algorithm based on load feedback is used to control the synchronization rate, ensuring that background data synchronization tasks do not preempt critical resources of online services, maintaining system service stability during reconstruction. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a method according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart illustrating the deep learning-based business complexity trend prediction process of the present invention.

[0029] Figure 3 This is a schematic diagram of the strategy decision-making logic based on recursive cost compensation according to the present invention;

[0030] Figure 4 This is a schematic diagram of the dynamic execution process of the multimodal data management strategy of the present invention. Detailed Implementation

[0031] The following is in conjunction with the appendix Figure 1 - Appendix Figure 4 The present invention will be further described in detail below.

[0032] See attached document Figure 1 This invention provides a dynamic data management method oriented towards business complexity, comprising the following steps:

[0033] S100 constructs a real-time business interaction topology graph; the full-link perception module monitors the transaction processing flow of the target business system in real time, parses business logs to identify the set of data nodes accessed in a single transaction, and establishes or updates connection edges reflecting the co-occurrence relationship of nodes in the memory graph model, and dynamically updates the weight of connection edges according to the time decay algorithm.

[0034] S200, calculate the business complexity score based on topological features; the complexity calculation module extracts graph theory features based on the business interaction topology graph, calculates the structural entanglement index representing the degree of coupling of business logic, and the access discrete entropy index representing the degree of dispersion of data hotspots, and merges the above indicators with the basic load index to generate the business state vector at the current moment.

[0035] S300 uses deep learning to predict complexity trends. The trend prediction module obtains the historical business state vector sequence, inputs it into the pre-trained time series prediction model, and outputs the predicted business state vector within the future preset time window.

[0036] S400 makes strategy decisions based on a recursive cost compensation mechanism. The strategy decision module evaluates the theoretical performance benefits of candidate strategies for predicting business state vectors, calculates the strategy execution cost using a recursive cost function with the current structural entanglement degree and system load as independent variables, and generates a strategy switching instruction when the net benefit value exceeds a preset lag threshold.

[0037] S500 executes a multimodal data management strategy; the multimodal execution module responds to the strategy switching command and performs data reconstruction operations, including establishing dynamic materialized views or transient vertical splitting, and dynamically adjusts the data synchronization rate during the reconstruction process according to the real-time load.

[0038] The following will describe each of the above steps and their specific implementation principles in detail.

[0039] In the S100 step of constructing the real-time business interaction topology diagram, the system does not rely on static database schema definitions, but dynamically constructs a business interaction topology diagram reflecting the real-time status of the business through the end-to-end awareness module. This process is specifically divided into the following sub-steps:

[0040] S101, the end-to-end awareness module parses the collected business operation log stream to extract the data entities and their operation types involved in the business transactions. The system first identifies data nodes according to a preset granularity configuration. When configured at the table level, the system extracts the database table name involved in the SQL statement or API request as the node identifier; when configured at the column level, the system further parses the specific query field or update field as the node identifier. To ensure the global uniqueness of node identifiers, the system uses a combination of database name, table name or service name, and object name as the unique key for each node. The system stores the node set in memory using an adjacency list structure based on a hash map, which is updated in real time according to the parsing results. For the specific implementation of log parsing and data extraction, those skilled in the art can use conventional methods such as regular expression matching or abstract syntax tree (AST) analysis, which will not be elaborated here.

[0041] S102, the system identifies the relationships between data nodes based on business transactions or request context. The system extracts global trace IDs (TraceIDs) or transaction session IDs (SessionIDs) from the logs, aggregating operations with the same ID into an atomic transaction or call chain. If multiple data nodes are detected being read or modified within the same atomic transaction, or being accessed consecutively within the call chain of the same business request, the system determines that there is a business dependency between these data nodes and establishes directed connections between the corresponding nodes in the graph model. The direction of the connection is determined by the order of business logic calls or the direction of data flow; the establishment of the connection represents the coupling between data entities at the business logic level.

[0042] S103, the system dynamically updates the weights of connected edges based on a time decay mechanism. To ensure that the topology graph accurately reflects current business hotspots rather than being dominated by historical data, the system uses an Exponentially Weighted Moving Average (EWMA) algorithm to update the weights of connected edges. For any two data nodes that are connected, their current weight value is synthesized by weighting the historical weight value from the previous moment with the normalized co-occurrence frequency within the current monitoring period using a preset forgetting factor. This mechanism makes the weight update smooth and time-sensitive, and can adaptively adjust to changes in business traffic.

[0043] In step S200, which calculates the business complexity score based on topological features, the complexity calculation module further quantifies the business complexity from two dimensions—structural relevance and access discreteness—based on the constructed real-time business interaction topology graph, and generates a business state vector for subsequent decision-making. This process specifically includes the following sub-steps:

[0044] S201 calculates the structural entanglement degree metric, which represents the coupling depth of business logic. The complexity calculation module traverses the business interaction topology graph in memory and extracts the graph theory features of currently active data nodes. The structural entanglement degree is calculated based on the number of connections to a node and the weight strength of those connections, and is used to quantify the dependency complexity of business logic at the data level.

[0045] In the specific calculation process, for each node in the topology graph, the system obtains its in-degree and out-degree. The in-degree represents the frequency with which the data node is referenced by other business logic, and the out-degree represents the frequency with which the data node depends on other data. The system performs a weighted sum of the in-degree and out-degree, multiplies it by the logarithm of the sum of the weights of all adjacent edges of the node, and finally takes the average of the calculation results for all active nodes.

[0046] The formula for calculating the degree of structural entanglement is defined as follows:

[0047] ;

[0048] in, Indicates the degree of structural entanglement; This represents the total number of active nodes in the topology graph within the current time window; Indicates the first One data node; and Representing nodes respectively in-degree and out-degree; and These are the weighting coefficients for the in-degree and out-degree, respectively. ; Represents nodes The set of all nodes that have direct connections (including nodes connected by incoming edges and nodes connected by outgoing edges). Indicates the connection node With nodes The weights of the edges. The logarithmic function in the formula is used to reduce the non-linear impact of high-weight edges on the overall index.

[0049] S202 calculates the discrete entropy index, which characterizes the distribution of data access. Based on the entropy principle in information theory, this index measures the uniformity of data access traffic distribution across physical storage space, thereby determining whether hotspot contention exists in the current system.

[0050] The system first obtains the metadata of the underlying storage engine and divides the data storage space into... Each logical partition or physical shard. The end-to-end sensing module statistics are within the current time window. The system calculates the access request count for each partition. It uses the Shannon Entropy principle to calculate the discrete entropy of access. Specifically, the system calculates the expected value of the self-information of the probability of access to each partition. When this value is low, it indicates that access traffic is concentrated in a few partitions, and the system is in a hotspot access state; when this value approaches a certain level... When the access traffic is uniformly and randomly distributed, the system is in a random access state.

[0051] S203, Synthesize a multi-dimensional business state vector. To comprehensively describe the system's current operational state, the complexity calculation module uses the structural entanglement degree calculated above. Access Discrete Entropy Combined with system basic load indicators.

[0052] System basic load indicators The system calculates the overall load score using a weighted summation of multi-dimensional resource metrics. This is achieved by multiplying the CPU utilization, memory usage, and disk I / O throughput (after Min-Max Normalization) by preset weighting coefficients and then summing the results.

[0053] The final generated business state vector A standardized numerical sequence containing the above three dimensions is directly used as input data for subsequent deep learning trend prediction models. For the raw data acquisition of basic load metrics, those skilled in the art can use the performance counter interface provided by the operating system.

[0054] See attached document Figure 2 In step S300, which uses deep learning to predict business complexity trends, the trend prediction module, after obtaining the current business state vector, uses a time series prediction model to predict the future evolution of business complexity, thus providing forward-looking input for strategy decision-making. This process specifically includes the following sub-steps:

[0055] S301, Construct time series input samples. The trend prediction module maintains a fixed-length First-In-First-Out (FIFO) queue in memory to store historical business state vectors. Whenever the complexity calculation module generates a new business state vector... At this time, the vector is pushed to the tail of the queue, and the oldest vector at the head of the queue is removed. The system extracts consecutive vectors from the queue. A historical business state vector is used to construct a time-sliding window sequence as the input feature matrix for the prediction model. To ensure the numerical stability of the model input, the system employs an online standardization method, using the mean and variance within the sliding window to normalize the vector components.

[0056] S302 executes deep neural network inference. The trend prediction module feeds the input feature matrix into a Long Short-Term Memory (LSTM) network model. This model is used to capture long-term dependencies and non-linear patterns in changing business complexity. To adapt to dynamic changes in business patterns, the model supports periodic offline retraining or incremental updates based on new samples.

[0057] During inference, the feature matrix is ​​processed through the forget gate, input gate, and output gate mechanisms of the LSTM unit. The forget gate determines how much of the previous time step's unit state is retained, the input gate controls how much new information from the current time step is stored in the unit state, and the output gate controls the output of the hidden state based on the current unit state. Through the stacking of multiple layers of LSTM units, the model transforms the original time series into a hidden state vector containing temporal dependencies. The specific unit structure of the LSTM network and the backpropagation training algorithm are well-known techniques in this field and will not be elaborated upon here.

[0058] S303 generates a predicted business state vector for future timeframes. The fully connected output layer of the model maps the final hidden state vector of the LSTM hidden layer back to the dimensional space of the business state vector through a linear transformation, obtaining a future state vector containing structural entanglement, access discrete entropy, and predicted basic load values. This predicted vector is directly transmitted to the policy decision module as the basis for calculating recursive costs and evaluating the net utility of the policy.

[0059] See attached document Figure 3 In the S400 step, where strategy decision-making is based on a recursive cost compensation mechanism, the strategy decision-making module connects the trend prediction module and the end-to-end perception module. Based on the predicted future business state, it matches the optimal strategy from the multimodal data management strategy library and evaluates the feasibility of strategy switching through the recursive cost compensation mechanism. This process specifically includes the following sub-steps:

[0060] S401, retrieve candidate strategies and calculate theoretical performance gains. The strategy decision module has a built-in data management strategy library that stores various physical map reconstruction schemes. This library includes at least dynamic materialized view strategies for high structural entanglement and transient vertical splitting strategies for high access dispersion.

[0061] The module matches the predicted business state vector output by the trend prediction module with the applicable conditions of each strategy in the strategy library. For each candidate strategy... The system calculates its theoretical performance gain under the predicted state based on historical performance metadata. The theoretical performance gain is represented as the difference between the target component (such as structural entanglement) in the predicted business state value and the ideal benchmark value preset by the system, and is scaled by a performance conversion coefficient derived from historical statistical data.

[0062] This benefit value reflects the expected reduction in query response time or increase in throughput after the strategy is implemented.

[0063] S402, construct a recursive cost function and calculate the policy execution cost. The system introduces a recursive cost compensation mechanism to quantify the nonlinear amplification effect of the current system business complexity and basic load on the policy execution cost.

[0064] The system calculates and executes candidate strategies using a recursive cost function. Required dynamic cost This function uses the basic operation cost as the base and leverages the current degree of structural entanglement. and base load As a compensation factor for recursive amplification.

[0065] The formula for calculating the recursive cost function is defined as follows:

[0066] ;

[0067] in, For strategies pre-stored in the cost metadata table Basic resource consumption under no-load conditions; The current degree of structural entanglement is represented by its exponential term, which characterizes the lock blocking diffusion effect caused by modifying the data structure under strongly coupled business logic. The current baseline load characterizes the I / O amplification effect caused by resource contention; These are non-negative weighting coefficients trained based on historical system logs, used to adjust the amplification of the total cost by different dimensions.

[0068] S403, Perform net utility calculation and lag threshold determination. The strategy decision module calculates the weighted difference between the theoretical performance gain and the strategy execution cost to obtain the net utility value. The weight of the strategy execution cost in the difference calculation is adjusted by a cost sensitivity coefficient.

[0069] This coefficient is dynamically set based on the current system's Service Level Agreement (SLA) level. A higher SLA level results in a larger coefficient value, indicating a higher requirement for stability. To avoid frequent policy switching triggered under critical conditions, the module introduces a dual-threshold hysteresis comparison mechanism. Only when the calculated net utility value exceeds a preset threshold will a policy switch be initiated. When the net benefit of the strategy rollback falls below a preset stop threshold, the system generates a strategy switching instruction; only when the net benefit of the strategy rollback is lower than the preset stop threshold will the system switch the strategy. If this occurs, the system will rescind the policy. This mechanism ensures that the system only initiates a physical reconstruction of the underlying data structure when the expected benefits cover the dynamic execution costs.

[0070] See attached document Figure 4 In the S500 step of executing the multimodal data management strategy, the multimodal execution module responds to the strategy switching instructions generated by the strategy decision module, performs corresponding data reconstruction operations for different business complexity characteristics, and ensures the stability of production operations through an adaptive flow control mechanism. This process specifically includes the following sub-steps:

[0071] S501, execute a dynamic materialized view strategy based on reducing structural entanglement. When the received strategy instruction aims to reduce structural entanglement, the multimodal execution module traverses the business interaction topology graph and identifies strongly connected subgraphs where the connection weights exceed a preset aggregation threshold. The module extracts the schema definitions of all physical data tables involved in the subgraph and constructs a denormalized physical table containing redundant data, i.e., a dynamic materialized view, through multi-table join logic.

[0072] During the construction of the materialized view, the system initiates a Change Data Capture (CDC) task, subscribing to the incremental logs of the original data nodes and replaying data changes in the materialized view in real time to maintain near real-time consistency of the view data. To achieve transparency to upper-layer business logic, the system maintains a set of query rewriting rules at the database proxy layer. When the abstract syntax tree (AST) of an intercepted business query request matches the coverage of the materialized view, the system automatically redirects the query target to the materialized view, thus transforming the originally complex multi-table join query into a single-table scan operation. This process reduces lock wait time and deadlock probability during transaction processing, thereby reducing the system's structural entanglement index.

[0073] S502 executes a transient vertical partitioning and micro-segmentation domain locking strategy based on improving access dispersion. When the strategy instruction aims to improve access dispersion entropy to alleviate hotspot contention, the module performs a transient vertical partitioning operation. Based on the field-level access statistics of the end-to-end sensing module, the module divides the field set of the target data table into a high-frequency access hot field group and a low-frequency access cold field group.

[0074] The system performs table structure reconstruction at the physical storage layer, creating a main table containing only hot fields and an extended table containing cold fields, with the two linked one-to-one via the original primary key. Based on this physical partitioning, the system implements a micro-segmentation locking mechanism. When processing update requests, the database lock manager first parses the list of fields involved. If a business transaction only updates the hot fields in the main table, the lock manager only requests an exclusive lock on the main table record row, without blocking concurrent read operations on the cold fields in the extended table. This dual isolation at both the physical and logical levels increases the degree of dispersion of data access in the physical space, thereby effectively improving access discrete entropy.

[0075] S503 implements adaptive dual-write synchronous flow control based on real-time load. During the initialization and continuation of the aforementioned data reconstruction (materialized view construction or vertical splitting), consistency between the original data and the reconstructed data is required. To prevent the synchronization process from consuming excessive system resources and thus affecting online services, the module adopts an adaptive synchronization rate adjustment mechanism based on real-time load feedback.

[0076] The system periodically obtains the current basic load metrics. Based on this, the maximum allowed data synchronization rate is calculated. The calculation formula is defined as follows:

[0077] ;

[0078] in, The preset upper limit of the synchronization rate for the system; For the current base load; A load threshold is defined for the system. When the load reaches this value, the system will reduce the synchronization rate to zero to pause the synchronization operation. It is a flow shaping factor with a value greater than 1, used to control the steepness of the decrease in synchronization rate as the load increases.

[0079] This mechanism ensures that the data reconstruction process always runs as a low-priority background task, minimizing the impact on the performance of online services in the production environment.

Claims

1. A dynamic data management method oriented towards business complexity, characterized in that, Includes the following steps: S100: Monitor the transaction processing flow of the target business system in real time, parse the business logs to identify the set of data nodes accessed in a single transaction, and establish or update the connection edges that reflect the co-occurrence relationship of nodes in the memory graph model to construct a real-time business interaction topology graph. S200. Based on the real-time business interaction topology graph, extract graph theory features, calculate the structural entanglement index representing the degree of coupling of business logic and the access discrete entropy index representing the degree of dispersion of data hotspots, and merge the structural entanglement index, the access discrete entropy index and the basic load index to generate the business state vector at the current moment. S300. Obtain the historical business state vector sequence, input the business state vector sequence into a pre-trained time series prediction model, and output the predicted business state vector within a future preset time window. S400. Evaluate the theoretical performance benefits of the candidate strategy for the predicted business state vector, calculate the strategy execution cost using a recursive cost function with the current structural entanglement index and the basic load index as independent variables, and generate a strategy switching instruction when the difference between the theoretical performance benefits and the strategy execution cost exceeds a preset lag threshold. S500, in response to the strategy switching instruction, perform a data reconstruction operation, the data reconstruction operation including establishing a dynamic materialized view or performing transient vertical splitting.

2. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S100, constructing the real-time business interaction topology diagram specifically includes: Data node identifiers are extracted from the business logs according to the preset granularity configuration, and the node set is stored using a hash-mapped adjacency list structure. Extract the global tracking identifier or transaction session identifier from the business log, and aggregate operations with the same identifier into atomic transactions; When multiple data nodes are detected to be accessed in the same atomic transaction, it is determined that there is a business dependency relationship between the multiple data nodes, and a directed connection edge is established between the multiple data nodes. The weights of the connecting edges are updated using an exponentially weighted moving average algorithm. The current weight value of the connecting edge is synthesized by weighting the historical weight value of the previous moment and the normalized co-occurrence frequency in the current monitoring period using a preset forgetting factor.

3. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S200, calculating the structural entanglement index, which characterizes the degree of coupling of business logic, specifically includes: Traverse the real-time business interaction topology graph to obtain the in-degree and out-degree of each currently active data node; The weighted sum of the in-degree and the out-degree is obtained by weighting and summing the in-degree and the out-degree respectively. Calculate the sum of weights of all adjacent edges of the data node, and perform a logarithmic operation on the sum of weights to obtain the weight strength factor; The product of the weighted sum of the node connectivity and the weight strength factor is calculated, and the average of the product results for all active data nodes is taken to obtain the structural entanglement index.

4. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S200, calculating the access discrete entropy index, which characterizes the discreteness of data hotspots, specifically includes: Obtain the underlying storage engine's metadata and divide the data storage space into multiple logical partitions; Count the number of access requests falling into each of the logical partitions within the current time window, and calculate the access percentage probability of each of the logical partitions; The expected value of the self-information of the access proportion probability is calculated based on the Shannon entropy principle, and the expected value is used as the access discrete entropy index. When the access discrete entropy index approaches the logarithm of the number of logical partitions, the system is determined to be in a random access state.

5. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S300, outputting the predicted business state vector within a future preset time window specifically includes: A fixed-length first-in-first-out queue is maintained in memory to store the business state vector at historical moments; Extract the continuous historical business state vectors from the queue to construct a time sliding window sequence, and normalize the time sliding window sequence to obtain the input feature matrix; The input feature matrix is ​​input into a long short-term memory network model, and the input feature matrix is ​​processed through forget gate, input gate and output gate mechanisms to generate hidden state vectors. The hidden state vector is mapped back to the dimension space of the service state vector through a fully connected output layer, generating the predicted service state vector containing the predicted values ​​of structural entanglement, access discrete entropy, and basic load.

6. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S400, calculating the strategy execution cost using a recursive cost function with the current structural entanglement index and the basic load index as independent variables specifically includes: Obtain the basic resource consumption of candidate strategies under no-load conditions; The entanglement index of the structure at the current moment is used as an exponential term to calculate the first compensation factor, which represents the lock blocking diffusion effect caused by modifying the data structure. The basic load index at the current moment is used as a linear term to calculate the second compensation factor, which represents the input-output amplification effect caused by resource contention. The execution cost of the strategy is obtained by multiplying the basic resource consumption, the first compensation factor, and the second compensation factor.

7. The dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S500, the data reconstruction operation specifically includes: When the strategy switching instruction aims to reduce the structural entanglement index, identify strongly connected subgraphs in the real-time business interaction topology graph where the connection weight exceeds the aggregation threshold. Extract the data table schema definition involved in the strongly connected subgraph, construct the dynamic materialized view through multi-table join logic, and start the change data capture task to synchronize the change data of the original data node to the dynamic materialized view in real time, so as to maintain the consistency between the dynamic materialized view and the original data node. The database proxy layer maintains query rewriting rules. When a business query request matches the coverage of the dynamic materialized view, the query target is redirected to the dynamic materialized view.

8. A dynamic data management method oriented towards business complexity according to claim 6, characterized in that, In step S500, the data reconstruction operation further includes: When the strategy switching instruction aims to improve the access discrete entropy index, the field set of the target data table is divided into hot field group and cold field group based on field-level access statistics; In the physical storage layer, a main table containing the hot field group and an extended table containing the cold field group are created, and the main table and the extended table are associated through a primary key; When processing update requests, if the business transaction only updates fields in the main table, it only requests an exclusive lock on the record row of the main table, allowing concurrent read operations on fields in the extended table.

9. A dynamic data management method oriented towards business complexity according to claim 6, characterized in that, In step S500, the data reconstruction operation further includes: During data synchronization, the current basic load metrics are periodically acquired; Calculate the ratio of the basic load index to the system-defined load threshold; The ratio is calculated using a preset traffic shaping factor to determine the current maximum allowable data synchronization rate, such that the maximum data synchronization rate decreases non-linearly as the basic load index increases. The data synchronization process between the original data node and the reconstructed data node is controlled according to the maximum data synchronization rate.

10. A dynamic data management method oriented towards business complexity according to claim 1, characterized in that, In step S400, evaluating the theoretical performance gains of candidate strategies for the predicted service state vector and generating strategy switching instructions specifically includes: The difference between the target component in the predicted business state vector and the ideal benchmark value preset by the system is calculated, and the theoretical performance gain is determined by combining the historical performance conversion coefficient. A cost sensitivity coefficient is set according to the current system's service level agreement level. The weight of the strategy execution cost in the difference calculation is adjusted using the cost sensitivity coefficient to obtain the net utility value. A dual-threshold hysteresis comparison mechanism is adopted, which generates the strategy switching instruction only when the net utility value is greater than the start threshold, and cancels the strategy switching instruction only when the net benefit after canceling the strategy is lower than the stop threshold.

Images