A data storage method and device for mass transaction information

Abstract

The application discloses a data storage method and equipment for mass transaction information, through sensor network collection node data, based on communication record and topological calculation node distance and path jump number, combined with data queuing time and packet loss rate calculation node load, mark super threshold node and locate abnormal link, generate abnormal index, further calculate adjacent density and topological depth, get connection strength and path connectivity distribution, according to this, identify bottleneck node and build sub node set, through calculation path congestion degree update bottleneck set, generate congestion influence domain, when there is high load node in the domain, trigger congestion signal, extract domain path length and redundancy, combined with node history calculation preliminary congestion index and generate routing optimization vector, finally, through buffer overflow rate calculation risk coefficient, combined with flow peak value correlation analysis, output detailed feature data containing congestion degree, influence range and optimization suggestion, support network adjustment processing.

Description

Technical Field

[0001] This invention relates to the technical fields of financial data / transaction data processing and massive data storage and monitoring, and in particular to a data storage method and device for massive transaction information. Background Technology

[0002] In the digital economy era, massive data storage and efficient transmission are often deeply coupled engineering processes. With the advancement of comprehensive digitalization of financial services, research has found that the scale of transaction information carried by modern data processing nodes is growing exponentially. Cross-node and cross-regional transaction data migration and synchronization have become the norm. For example, transaction information from bank branches, especially transaction information data related to banks such as university management, such as large campus bank branches, contains diverse information and a huge amount of data.

[0003] Traditional data security perspectives often focus on encryption and access control mechanisms for static storage. However, the flow of transaction data—its transmission path—is also a critical link in risk aggregation and a performance bottleneck. When transaction data volume reaches the petabyte (PB) level, optimizing data storage methods transcends mere performance requirements and directly relates to the fundamental attributes of information security: among which, path scheduling is particularly crucial for ensuring security and availability. Through intelligent routing decisions, dynamic load balancing, and congestion avoidance mechanisms, the system must be able to automatically identify and avoid network congestion or faulty nodes, ensuring the timely delivery of critical transaction information such as real-time risk control instructions, security policy updates, and disaster recovery copies. This directly determines whether security policy delays or failure to achieve recovery time targets due to transmission delays or interruptions can be avoided.

[0004] However, current mainstream network path monitoring and scheduling solutions still have significant shortcomings when facing massive transaction scenarios. Existing methods mostly rely on static topology analysis and threshold judgment based on instantaneous indicators. They calculate basic parameters such as node load and path hop count by collecting node operation logs and communication records, and then combine clustering algorithms to analyze node distribution characteristics. These traditional methods expose two major flaws in real and complex transaction network environments:

[0005] First, the instability of node communication characteristics severely impacts the accuracy of path analysis. During transaction information transmission, inter-node communication is susceptible to interference from factors such as peak business hours and resource contention, causing significant fluctuations in communication intervals. This fluctuation leads to a drift in the node's coordinates within the feature space, akin to positioning measurements in an unstable medium, making it difficult to capture the true topological relationships. When a node's communication latency changes drastically due to a sudden surge in concurrency, the feature coordinates obtained through conventional measurements will deviate from its actual logical location, resulting in significant errors in the adjacency relationships calculated by density-based clustering algorithms. These errors are propagated and amplified at each level of the path decision-making process, ultimately leading to inaccurate identification of congested areas and hindering the effective implementation of optimization mechanisms.

[0006] Secondly, existing methods do not adequately address the comprehensive risk assessment of critical nodes. In financial transaction information networks, some core exchange nodes play an irreplaceable hub role. Traditional solutions often only focus on their instantaneous load status, lacking a comprehensive assessment of their global influence in the network, business criticality, and failure costs. Once such nodes are overloaded, simply adopting a detour strategy may not only significantly increase transmission latency due to path detours, affecting the timely delivery of real-time transaction data and security signaling, but may also transfer congestion pressure to other core nodes, triggering a cascading congestion effect and ultimately jeopardizing the security and availability of the entire transaction information network.

[0007] The aforementioned problems highlight the limitations of existing path optimization technologies in handling massive transaction information transmissions. Traditional methods, lacking the ability to predict the cost of path reconstruction after node failures, often result in superficial optimization decisions. This not only fails to systematically improve transmission efficiency but may also introduce new instabilities, posing a potential threat to the secure and timely transmission of transaction information. Therefore, there is an urgent need for a new data storage method that can balance transmission efficiency with security and stability for massive transaction information. Summary of the Invention

[0008] The purpose of this invention is to provide a data storage method and device for massive transaction information, thereby solving the aforementioned technical problems pointed out in the prior art.

[0009] This invention provides a data storage method for massive transaction information, comprising the following steps:

[0010] The system accesses the original transaction information of the online trading system in real time, and parses and extracts the original transaction information. At the same time, during the above process, each node generates and stores original operation log data and communication records, including data queuing time and transmission packet loss ratio indicators.

[0011] The system collects operation log data and communication record data of each node through a distributed sensor network. Based on the communication record data, it calculates the node spacing distribution data and path hop count. It extracts data queuing time and transmission packet loss ratio from the operation log data, and fuses and calculates node load data. When the node load data exceeds the first load threshold, it is marked as an over-threshold element. Abnormal links are located based on the over-threshold elements. The abnormal status index is obtained by calculating the ratio of the number of abnormal links to the total number of links.

[0012] Based on node spacing distribution data and path hop count, the node adjacency density and topology depth are calculated, and then the connection strength value is calculated and the path connectivity distribution is generated. Combined with data queuing time, transmission packet loss ratio and abnormal state indicators, the transmission bottleneck node is identified. The network sub-node set is extracted with the transmission bottleneck node as the core. The communication path is traversed in the network sub-node set and the congestion degree value of each communication path is calculated. When the congestion degree value exceeds the congestion threshold, the high load node is marked, the bottleneck node set is updated, and the influence sub-elements of the congestion influence domain are generated through clustering algorithm.

[0013] Based on the sub-elements of the congestion impact domain, when the over-threshold element is located within the sub-elements and the node load data exceeds the second load threshold, a system-level congestion signal is triggered.

[0014] After triggering a system-level congestion signal, the path length and redundancy information of the affected routing paths are extracted based on the path connectivity distribution. Combined with the historical bottleneck processing of nodes, preliminary congestion indicators are calculated, and a route optimization vector is generated.

[0015] Collect buffer overflow rate records for each node within the affected sub-elements of the congestion impact domain, calculate the ratio of the number of overflowing nodes to the total number of nodes to obtain the congestion risk coefficient, extract peak traffic data from the operation log data, and perform correlation analysis between the peak traffic data and the congestion risk coefficient and the routing optimization vector to generate congestion feature data.

[0016] In another aspect, the present invention also provides a data storage device for massive transaction information, the device comprising a memory and a processor, the memory storing a computer program, which, when executed by the processor, implements the steps of the above-described method for data storage of massive transaction information.

[0017] Compared with the prior art, the embodiments of the present invention have at least the following technical advantages:

[0018] Analysis of the data storage method for massive transaction information provided by this invention reveals that, in specific applications, this method first calculates abnormal states (quantifies overall link abnormality indicators) to achieve real-time monitoring and anomaly detection of network nodes and links; then, it further calculates node adjacency density using density calculation algorithms (such as DBSCAN) to characterize the tightness of connections between nodes and their surrounding nodes, identifying network hub nodes and transmission bottleneck nodes; taking all transmission bottleneck nodes as the core, it extracts a set of network child nodes containing neighboring nodes of the transmission bottleneck nodes from the node topology to achieve dimensionality reduction of the analysis scope; it traverses communication paths in the set of network child nodes, and calculates the congestion level value of each communication path by combining connection strength value, data queuing time, and transmission packet loss ratio; when the congestion level value exceeds a preset congestion threshold, it marks the corresponding node of the path as a high-load node and summarizes and updates it to the bottleneck node set;

[0019] Then, the scope of influence is further expanded. Based on the spatial distribution of the updated bottleneck node set in the network sub-node set, clustering algorithms are used to generate impact sub-elements of the congestion impact domain. By analyzing the impact sub-elements of the congestion impact domain and node load data, it is confirmed that there are severely overloaded nodes in the congestion impact domain, thus activating the subsequent in-depth analysis and optimization process. After the system-level congestion signal is triggered, based on the path connectivity distribution, the path length and path redundancy information of all affected routing paths in the impact sub-elements of the congestion impact domain are extracted to provide control processing for subsequent storage configuration.

[0020] Simultaneously, by collecting buffer overflow rate records of each node within the affected sub-elements of the congestion impact domain, the number of overflowing nodes is counted, and the congestion risk coefficient is calculated using the ratio of the number of overflowing nodes to the total number of nodes within the affected sub-elements. This reflects the server's risk of crashing at the resource level. Peak traffic data is extracted from the operation log data, and the peak traffic data, congestion risk coefficient, and routing optimization vector are matched and correlated to establish a correspondence table among the three. This assesses the congestion risk and optimization path selection under different traffic levels, and finally generates detailed congestion characteristic data, providing comprehensive reference information for storage optimization and adjustment.

[0021] In practice, by integrating factors such as latency jitter, load fluctuation, path redundancy, and virtual failure simulation of nodes in multi-path systems, the system accurately quantifies the connectivity density of nodes in the network topology (node ​​adjacency density), identifies structural central nodes (such as traffic hubs), and adaptively adjusts clustering parameters to make density calculations more closely reflect the actual network conditions, avoid interference from edge nodes, provide key topological features for subsequent calculation of connection strength values, distinguish between hubs and edge nodes, and improve the accuracy and robustness of node importance assessment in network congestion analysis. Attached Figure Description

[0022] Figure 1This is a schematic diagram of the main process of a data storage method for massive transaction information.

[0023] Figure 2 This is a schematic diagram of the topological hierarchy depth in a data storage method for massive transaction information.

[0024] Figure 3 This is a schematic diagram simulating a network sub-node set in a data storage method for massive transaction information;

[0025] Figure 4 A simplified network simulation diagram for a data storage method for massive transaction information;

[0026] Figure 5 This is a schematic diagram simulating the connection of key nodes in a data storage method for massive transaction information. Detailed Implementation

[0027] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment provides a data storage method for massive transaction information, including the following steps:

[0031] Step S10: Access the original transaction information (i.e., original transaction flow data) of the online transaction system in real time, parse the original transaction information, and extract information such as transaction identifier, timestamp, initiating node and receiving node; in the above process, each node generates and stores original operation log data and communication records including indicators such as data queuing time and transmission packet loss ratio.

[0032] The raw transaction information (i.e., raw transaction log data) originates from the online transaction system's database, message queue, or the memory of the business server. The transaction identifier is a unique ID for each transaction, used to distinguish and trace individual transactions. The timestamp indicates the precise time the transaction occurred, used to calculate transaction latency, traffic timing variations, etc. The initiating and receiving nodes identify the source and destination of transaction data packets, forming the basis for drawing data flow diagrams and calculating network topology paths. The main purpose of parsing the raw transaction information is to calculate business load characteristics (such as transactions per second (TPS) and peak transaction traffic) and to determine fluctuations in business volume.

[0033] The raw operational log data is generated by the operating systems and middleware on each server (node) running the trading system and written to log files in local storage. Communication records are captured and recorded by network devices (such as switches and routers) or network monitoring modules on the servers (such as packet capture tools and network probes). The communication records contain data about the transmission process of packets in the network. For example: the handshake time between nodes; retransmission records of data packets; the actual physical path of transmission (routing tracing results); for example: the distance between nodes (which can be roughly estimated through network latency) and the number of path hops.

[0034] The system collects operation log data and communication record data from each node through a distributed sensor network. Based on the source and destination address information in the communication record data, and combined with a preset server node connection topology, it calculates the node spacing distribution data. It extracts the communication path corresponding to each communication record data, counts the number of intermediate nodes in each communication path to generate the path hop count, extracts the data queuing time of each node from the operation log, and counts the transmission packet loss ratio of each node from the communication record. It merges the data queuing time and the transmission packet loss ratio to calculate the node load data corresponding to each node (i.e., node load data value = α × normalized value of data queuing time + β × transmission packet loss ratio). When the node load data exceeds a first load threshold, the node is marked as an over-threshold element, and abnormal links are located from each communication path based on the over-threshold element. It calculates the abnormal status index (i.e., the overall link abnormal index) by counting the ratio of the number of abnormal links to the total number of links. The parameters α and β are the corresponding weights, which are preset fixed weights.

[0035] It should be noted that in the above embodiments of this application, the node spacing distribution data is a dataset that quantifies the logical distance between any two nodes in the network. It is a logical distance based on network topology and performance, directly reflecting the cost or expense of data packet transmission between two nodes. An abnormal link is a part of the communication path. For example, in a communication record, the extracted communication path is A (e.g., source address information) → B → C → D (e.g., destination address information), that is, communication from node A to node B, from node B to node C, and from node C to node D. The number of intermediate nodes in each path is counted, which is to count the number of intermediate hops in the communication record data from the sending node (i.e., node A) to the destination node (i.e., node D), that is, hopping from node B and node C, i.e., the path hops of the above communication path A→B→C→D. The number of hops is 2. The above refers to locating abnormal links from each communication path based on the threshold element. The link is each forwarding path (or sub-path of communication) in the above communication path. That is, the above communication path A→B→C→D includes links A→B, B→C, and C→D. When node C is identified as the threshold element, the link through that node is called an abnormal link. That is, the located abnormal links are links B→C and C→D. It should also be noted that the number of path hops defined in this application specifically refers to the number of intermediate nodes in the communication path. In some network protocol contexts, the number of path hops may include source and destination nodes. Its value is equal to the 'number of path hops' defined in this application plus 2 (2 is the starting node and destination node in the communication path process). This application adopts this definition to focus more on the assessment of the relay burden of the path.

[0036] Step S20: Based on the node spacing distribution data and the path hop count, calculate the node adjacency density using a density calculation algorithm (such as DBSCAN) and calculate the topology depth using a depth-first traversal algorithm; calculate the connection strength value using the node adjacency density and the topology depth, and generate a path connectivity distribution based on the connection strength value; combine the data queuing time, the transmission packet loss ratio, and the abnormal state indicators to determine the transmission bottleneck node (the transmission bottleneck node is a node on a certain link, such as node C; the transmission bottleneck node determined in step S20 is a preliminary determination result, and subsequent processing operations are performed to conduct fine verification and discover potential associated bottlenecks); with all the transmission bottleneck nodes as the core, extract the network child node set containing the neighbor nodes of the transmission bottleneck node from the node topology (at this time, the network child node set not only determines a communication path ABCD, but also needs to extract other... The network consists of multiple communication paths, such as EFCP and UYCV, which pass through the bottleneck node C and form a set of all neighboring nodes. The communication paths are traversed within this set, and the congestion level of each path is calculated by combining the connection strength value, the data queuing time, and the packet loss ratio. When the current congestion level exceeds a preset congestion threshold, all nodes corresponding to the current communication path are marked as high-load nodes, and all high-load nodes are updated to the bottleneck node set. Based on the spatial distribution of the updated bottleneck node set within the network set, a clustering algorithm is used to generate influence sub-elements of the congestion influence domain (the influence sub-elements of the congestion influence domain are also called congestion influence subgraphs; the above process can also be described as spatial clustering of the updated bottleneck node set using a clustering algorithm, with each connected node cluster being called a congestion influence subgraph, and all subgraphs together constituting the congestion influence domain).

[0037] The congestion level value is calculated as follows:

[0038] ;

[0039] In the formula, Let be the congestion level of the communication path (currently traversed), and n be the number of links contained in the communication path (currently traversed). This is the average data queuing time on the p-th link (the communication path under the current traversal) (i.e., the average of the sum of the data queuing times of the two nodes on the corresponding p-th link). This represents the maximum average data queuing time across all links within the current set of child nodes in the network being traversed.

[0040] This represents the packet loss ratio of the p-th link in the current traversal of the communication path. Its value is calculated directly from the communication record data for each link (e.g., based on the number of successful and failed data packets transmitted on the link). It represents the maximum percentage of packet loss across all monitored links (within the current set of network child nodes being traversed). Let p be the connection strength value of the p-th link in the current traversal of the communication path. The maximum connection strength among all links (within the current set of child nodes in the network being traversed);

[0041] It should be noted that in the above embodiments of this application, before calculating the congestion level value in the above embodiments of this application, it is necessary to normalize each parameter to unify the units. The normalization method is common knowledge to those skilled in the art, and will not be described in detail here. First, the node spacing distribution data and the number of path hops are used to form a multi-dimensional feature space. The coordinates of the nodes in this feature space are input into the DBSCAN equal-density clustering algorithm. This algorithm calculates the number of other nodes within a specified radius around each node and outputs a quantified node adjacency density. The node adjacency density is a continuous or discrete value used to characterize the tightness of a node's connection with its surrounding nodes in the network topology. High adjacency density nodes are usually hubs for network traffic exchange, identifying structural centers in the network and providing key local topological features for subsequent calculation of connection strength values, enabling the server to distinguish between hub nodes and edge nodes.

[0042] Furthermore, starting from one or more pre-defined root nodes (such as core routers) in the network, a depth-first traversal algorithm (such as breadth-first search, BFS) is executed. By expanding layer by layer, the minimum number of hops required to reach every other node in the network from the root node (i.e., the aforementioned topological depth) is recorded. Figure 2 As shown, the topology hierarchy depth is an integer sequence that identifies the specific level of each node in the network hierarchy. The smaller the value, the closer the node is to the network core, thus defining the center-edge architecture of the network as a key dimension for subsequent evaluation of the global importance of nodes. It is used together with the node adjacency density to calculate the connection strength value. Furthermore, the node adjacency density (D) and the topology hierarchy depth (L) are fused through a predefined function.

[0043] For example, using the formula: Connection strength of node p The calculation method is as follows ,in, For node adjacency density, The core logic of the topology hierarchy depth is that nodes with higher adjacency density and lower topology hierarchy depth have greater connection strength values. The connection strength value mentioned above is a comprehensive index used to quantitatively evaluate the inherent communication potential and structural importance of any link (the connection between two adjacent nodes) in the network, transforming topology information into quantitative parameters that can be used for quality assessment, and providing basic data for generating path connectivity distribution. Furthermore, for each end-to-end communication path to be analyzed (such as A→B→C→D), the connection strength values ​​of each link contained in the path are aggregated (e.g., the minimum or average value is taken) to obtain a score representing the overall connectivity quality of the path. After performing this operation on all critical paths, a path connectivity distribution dataset for the entire network is formed (as shown in Table 1 below). The path connectivity distribution is a dataset that describes the theoretical smoothness of each communication path under ideal no-load conditions due to its own topology, thereby establishing a performance baseline for a path. When subsequently determining real-time bottlenecks, this distribution can be used to distinguish whether the congestion is caused by a weak path structure or by temporary excessive load, thus accurately locating the problem.

[0044] Table 1

[0045]

[0046] Furthermore, the data from four dimensions—data queuing time, packet loss ratio, and abnormal status indicators—are input into a rule-based multi-threshold classification model. A node is only identified as a transmission bottleneck node when it exceeds its threshold in multiple dimensions simultaneously. This outputs an initial set of transmission bottleneck nodes, which are critical fault points that exhibit severe anomalies in performance, load, and health status. This completes the transformation from monitoring the entire network status to initially identifying suspected fault points, providing a target range for the next step of detailed local analysis.

[0047] Furthermore, using all the aforementioned transmission bottleneck nodes as initial points (or seed points), in the global node topology, these transmission bottleneck nodes themselves, all their directly connected nodes (one-hop neighbors), and all connections between these transmission bottleneck nodes are extracted, collectively forming a local set of network sub-nodes. This set of network sub-nodes is a simplified, small-scale network topology graph (e.g., ...). Figure 3As shown, this includes all initial bottleneck nodes (i.e., the aforementioned transmission bottleneck nodes) and their directly connected network environments, thereby reducing the dimensionality of the analysis scope and shrinking computational resources from the complex entire network to the local node range where the problems are most concentrated, improving the efficiency and focus of subsequent analysis. Furthermore, within the aforementioned network sub-node set, all active communication paths are identified. For each path, combining the connection strength value of the links it passes through (structural potential), the data queuing time of the path nodes (real-time processing latency), and the transmission packet loss ratio (real-time link quality), a single congestion level value is calculated using an aggregation formula. This quantifies the actual operational obstruction status of a communication path at a specific time. The higher the value, the more unsmooth the real-time communication of the path, thus transforming the static congestion of the path... By integrating structural attributes with dynamic performance indicators, a precise and quantitative assessment of the path's operational status is achieved. Furthermore, the congestion level of each path is compared with a preset congestion threshold. Any path exceeding this threshold has all its nodes marked as high-load nodes. Subsequently, these newly discovered high-load nodes are updated to the initial transmission bottleneck node set (the initial transmission bottleneck node set is the set composed of all the aforementioned transmission bottleneck nodes), forming an updated bottleneck node set. This results in a more comprehensive and accurate set of fault nodes, thereby verifying and expanding the preliminary results. By using the real-time congestion status of the path, more related bottleneck nodes that may have been missed by global indicators are verified and discovered, improving the coverage of fault identification.

[0048] Finally, the nodes in the updated bottleneck node set are mapped to the node set of the network child node set. Clustering algorithms (such as DBSCAN) are then applied to this node set to cluster nodes based on their proximity in the topology space. Ultimately, the outer contours of the largest or densest clusters are calculated to form the influencing sub-elements of the congestion impact domain. These influencing sub-elements of the congestion impact domain are individual nodes within the congestion impact domain, visually representing the core node set of congestion and providing a spatial basis for partial node-based routing adjustments or resource scheduling.

[0049] Step S30: Based on the influence sub-elements of the congestion influence domain, when there is a node that exceeds the threshold element and the node load data exceeds the second load threshold, a system-level congestion signal is triggered.

[0050] Step S40: After the server detects a system-level congestion signal, based on the path connectivity distribution, extract the path length (path length is the number of links (i.e., hops) of all affected routing paths within the congestion impact sub-element of the congestion impact domain, based on the path connectivity distribution. For example, if the path A→B→C→D has 2 hops (intermediate nodes B and C), then the path length is 3 (three links: AB, BC, CD)). Also extract the path redundancy information (path redundancy information refers to whether there are functionally equivalent backup paths for a given communication path. For example, in the congestion impact sub-graph, there is a critical path P1: A→B→C→D. If there are also backup paths P2: A→E→F→D (not passing through B and C) and P3: A→G→H→D (not passing through B and C) in the network, then the path redundancy information of path P1 can be represented by a value of 2 (there are 2 backup paths). The higher the value, the more redundant the given communication path is in the network. The stronger the substitutability of the structure, the smaller the impact of its failure or congestion on the overall network (i.e.); collect the node processing bottleneck history recorded within a preset time window, and calculate the path length and path redundancy information in combination with the node processing bottleneck history to obtain the preliminary congestion index within the influence sub-element of the congestion influence domain; based on the preliminary congestion index, map the path length and path redundancy into a multi-dimensional route optimization vector through a vector generation algorithm (the elements of the route optimization vector represent the optimized path direction; when generating the route optimization vector, we consider the optimization potential that can be achieved through route adjustment, i.e., the reduction in path length and the utilization of path redundancy, rather than the static values ​​of path length and path redundancy themselves; specifically, the preliminary congestion index V=[V1,V2,V3] is used to weight the two optimization objectives of path length and path redundancy; the vector generation algorithm solves an optimization problem with the goal of maximizing V1×(path length reduction)+V2×(path redundancy utilization), thereby generating the corresponding route optimization vector).

[0051] First, a two-dimensional decision space is constructed, using path length reduction and path redundancy utilization as two optimization dimensions. Second, preliminary congestion indicators V1 and V2 are used as optimization weights for the two dimensions in this decision space. Finally, the direction vector is solved. =( , ), so that the objective function max(V1× +V2× The vector obtains the optimal solution under the constraints. This is the desired route optimization vector. Furthermore, path length and path redundancy are static attributes describing the current state of the path, while the path length reduction and path redundancy utilization used to calculate the initial congestion index represent optimization potential, describing the improvement space that can be achieved by adjusting the route. The goal of the route optimization vector is to guide traffic scheduling decisions. During optimization, it's necessary to consider the path length reduction (i.e., how many hops can be reduced by switching traffic from the current path to a shorter path) and the path redundancy utilization (i.e., how much redundant resources can be utilized by switching traffic from the current path to an alternative path). For example, assuming the current path P1 has a length of 5 and the alternative path P2 has a length of 3, then the path lengths P1 and P2 are 5 and 3 respectively, and the path length reduction is 2 if switching from P1 to P2.

[0052] Step S50: Collect and obtain the buffer overflow rate records of each node in the sub-element of the congestion impact domain, count the number of nodes in the sub-element of the congestion impact domain whose buffer overflow rate exceeds the preset overflow rate threshold, obtain the number of overflow nodes, and calculate the congestion risk coefficient by using the ratio of the number of overflow nodes to the total number of nodes in the sub-element of the congestion impact domain; extract the peak traffic data from the operation log data (the peak traffic data is the maximum data throughput of each node in the sub-element of the congestion impact domain, which is obtained from the operation log data within a preset time interval); match and correlate the peak traffic data with the congestion risk coefficient and the routing optimization vector (establish a correspondence table between the peak traffic data, the congestion risk coefficient, and the routing optimization vector, and evaluate the congestion risk and optimization path selection under different traffic levels based on the correspondence table), and finally generate detailed congestion feature data containing congestion degree, impact range, risk level, and optimization suggestions;

[0053] The preliminary congestion index is calculated as follows:

[0054] ;

[0055] In the formula, This represents the total number of all affected routing paths extracted from the path connectivity distribution and located within the affected sub-elements of the congestion impact domain (characterizing the communication range affected by the congestion problem). The path length of the affected route path a (its value is the number of links traversed by the path from the origin to the destination, i.e., the path hop count plus one; for example, the path A→B→C has 2 hops and a path length of...). (3), representing the inherent transmission delay basis of the path. The maximum path length among all affected routing paths. This represents the total number of nodes in the updated bottleneck node set within the affected sub-elements of the congestion impact domain (representing the number of core failure points in the congestion impact domain). This represents the path redundancy of the b-th node in the updated set of bottleneck nodes within the congestion impact domain. Within the sub-elements of the congestion impact domain, the total number of nodes that have bottlenecks recorded within a preset time period. Within the impact sub-element of the congestion impact domain, the bottleneck frequency of the c-th node among the nodes that have recorded the bottlenecks in the past preset time period (its value is the number of times that node was recorded by the server as an over-threshold element or associated with an abnormal link set in step S10 within the preset time period; this value quantifies the instability and vulnerability of the node in history).

[0056] In the above formula, The baseline of average transmission efficiency of communication paths within the congestion impact domain is characterized by a higher value, which means that the communication paths within the congestion impact domain are generally longer and the inherent delay of data transmission is greater.

[0057] It characterizes the tolerance of the network structure of the congestion impact domain to bottleneck nodes. The higher the value, the more backup paths the network has, and the stronger the structural robustness, even if a bottleneck occurs.

[0058] It characterizes the historical reliability of nodes within the congestion impact domain. The higher the value, the more it indicates that the congestion impact domain is a set of "old problem" nodes that repeatedly have performance issues, and the risk of recurrence is high.

[0059] The calculation of the preliminary congestion index in the above embodiments of this application encapsulates real-time path structure characteristics and historical performance in a unified mathematical object. The server can calculate the preliminary congestion index based on which component is dominant (e.g., Extra High The specific value (e.g., low value) determines whether the optimization strategy focuses on "shortening the path" or "increasing redundancy utilization," thus providing accurate and comprehensive input for the subsequent generation of route optimization vectors.

[0060] It should be noted that in the above embodiments of this application, by first traversing each node in the sub-elements of the congestion impact domain, reading the node load data value one by one, and comparing the load data value of each node with a preset second load threshold that is higher than the first load threshold in S10, when it is found that the load data of at least one node exceeds the second load threshold, a system-level congestion signal is triggered. This signal is a Boolean value or a specific event identifier, indicating that there are indeed nodes with extremely high loads in the initially defined congestion impact domain, thereby confirming the active congestion impact domain from the suspected node set, thus completing the conversion from node set location to congestion confirmation, and serving as a decision switch to start the subsequent in-depth analysis and optimization process;

[0061] After a congestion signal is triggered, the server uses the sub-elements of the congestion impact domain as spatial filtering conditions to filter out all routing paths whose origin, destination, or path nodes are located within the affected sub-elements from the generated path connectivity distribution (a dataset containing all network paths and their connection strength values). Subsequently, two key indicators are extracted from the attributes of these paths: path length (the number of hops or the total distance traversed by the path) and path redundancy (the number of functionally equivalent backup paths). The server outputs a feature subset of the affected routing paths, which quantitatively describes the structural characteristics of all critical paths within the congestion impact domain, providing targeted input data for subsequent calculations and transforming the topology distribution into path-level features that can be used for quantitative evaluation.

[0062] Furthermore, by querying the database, the historical records of node processing bottlenecks within a preset time window (e.g., the frequency and history of nodes being marked as exceeding the threshold, continuously updated in S10) are collected. These bottleneck histories are then fused with path characteristics to generate a preliminary congestion index. This index characterizes the overall congestion pressure of the congestion impact domain across both the current path state and historical bottleneck tendencies, thus providing a preliminary assessment of congestion severity. Combining dynamic real-time characteristics with static historical performance lays the foundation for generating more refined optimization strategies. Furthermore, based on the aforementioned preliminary congestion index, [the following steps are taken]. Through vector generation algorithms (a mathematical method that maps scalar or multidimensional indicators to direction vectors), the two key features of path length and path redundancy are mapped to a multidimensional space. The vector generation algorithm determines a main direction that can simultaneously optimize (shorten) path length and (utilize) path redundancy along this direction. The direction and length of this routing optimization vector in the predefined policy space represent the optimization direction in which data traffic should be preferentially guided (e.g., the vector points to a group of nodes with high redundancy and short paths), thereby transforming the conclusions of the analysis into a mathematical compass, providing clear control instructions for automated traffic scheduling systems;

[0063] Furthermore, the buffer overflow rate records of all nodes within the affected sub-elements of the congestion impact domain are collected from the operation logs. The overflow rate of each node is compared with a preset overflow rate threshold, and the number of nodes exceeding the threshold is counted to obtain the number of overflow nodes. Finally, the ratio of this number to the total number of nodes within the affected sub-elements of the congestion impact domain is calculated to obtain the congestion risk coefficient. This coefficient represents the prevalence of the node buffer resources within the congestion impact domain being nearly exhausted or already overflowing, reflecting the server's risk of collapse at the resource level. This provides a key early warning indicator for server stability. The higher the risk coefficient, the greater the possibility of localized packet loss, a sharp increase in latency, or even network paralysis.

[0064] Furthermore, peak traffic data (i.e., the maximum data throughput of all nodes within the congestion impact domain within a preset time interval) is extracted from the operation log data. A corresponding table is constructed with peak traffic data and congestion risk coefficient as query conditions and route optimization vector as the output result. Through table lookup and analysis, the optimal path selection should be adopted under different traffic levels and risk levels. Based on all the above analysis and calculation results, a final report is compiled and generated, outputting detailed congestion characteristic data, comprehensively describing the current degree of congestion (combining indicators and peak values), geographical impact range (impact sub-elements), potential risks (risk coefficients), and specific solutions (optimization vectors).

[0065] Examples of the technical solutions adopted in the embodiments of this application are illustrated below:

[0066] First, a simplified network consisting of 6 nodes is constructed within the server, with its physical connections and logical hierarchy as shown in the topology. Figure 4 As shown (assuming the core layer consists of nodes A and B; the aggregation layer consists of nodes C and D; and the access layer consists of nodes E and F);

[0067] Preset thresholds: First load threshold = 0.7, Second load threshold = 0.8, Congestion threshold = 0.75, Overflow rate threshold = 5%;

[0068] Distributed sensors collect operation logs and communication records from each node. Based on the communication records, node spacing distribution data is calculated (i.e., the node spacing distribution data describes the distribution relationship of nodes in the topology graph. According to the shortest path hop count in the topology graph, the logical distance between any two nodes in the network is calculated to form a node spacing distribution dataset). For example, nodes A and B are close, while A and F are far apart. The number of intermediate nodes in the communication path is counted to generate the path hop count. For example, the path A→C→E→F has 2 hop counts. The data queuing time sequence is extracted from the operation logs. For example, the queuing time of node C increases significantly to 120ms. The packet loss ratio of each node is counted from the communication records. For example, the packet loss ratio of node C is 3%. Then, anomalies are located. The data queuing time and packet loss ratio of node C are fused to calculate its node load data as 0.82. Among all load data, the load data of node C (0.82) exceeds the first load threshold (0.7) and is marked as an over-threshold element. Based on the over-threshold element C, the abnormal link is located from the communication path. For example, the paths B→C and C→D along route C are identified as abnormal links, and the statistical abnormality index is 2 (number of abnormal links) / 10 (total number of links) = 0.2.

[0069] Further, the processing operation of step S20 is performed. Based on the node spacing distribution data and the number of path hops, the node adjacency density is calculated using the DBSCAN algorithm. Nodes A, B, and C are identified as high-density regions due to their close connections. Starting from the root node A, the topology depth is calculated using the BFS algorithm. The depth of nodes A and B is 1 (core), and the depth of nodes C and D is 2 (convergence). Using the node adjacency density (high) and the topology depth (small), the connection strength value of node C is calculated to be very high. Based on the connection strength values ​​of all links, the path connectivity distribution is generated. The path A→C→D has a high link connection strength value and a high score in the distribution. Combined with the high data queuing time, high packet loss ratio, and abnormal state index (0.2), node C is determined to be the transmission bottleneck node. Taking node C as the core, all its one-hop neighbors (B,D,E) are extracted to form a network sub-node set, which includes nodes {B,C,D,E} and the links between them. Traversing the path B→C→D in the network sub-node set, and combining its link connection strength value, data queuing time, and packet loss ratio, the congestion level value C=0.88 is calculated. This congestion level value (0.88) exceeds the preset congestion threshold (0.75). Therefore, nodes B and D on the path are marked as high-load nodes and merged with the initial transmission bottleneck node C, updating the bottleneck node set to {B,C,D}.

[0070] Based on the distribution of the updated bottleneck node set {B,C,D} in the network sub-node set, clustering is performed using the DBSCAN algorithm. Since these three nodes are topologically closely connected, they are clustered into a cluster. The outer contour of the cluster is calculated to generate the influence sub-elements of the congestion influence domain. These influence sub-elements cover the set of nodes from the core to the convergence layer of the network.

[0071] Furthermore, among all node load data, the node load data of node C (located within the affected sub-element of the congestion impact domain) is 0.82, exceeding the second load threshold (0.8), triggering a system-level congestion signal. Then, based on the path connectivity distribution, the characteristics of the affected paths within the affected sub-element are extracted. For example, the path length of path A→B→C is 2, and its path redundancy is 1 (there is a backup path A→D→C).

[0072] Calculate the initial congestion index V: A (total number of affected routes) = 5, PL max (Maximum path length) = 3, B (total number of updated bottleneck nodes) = 3 (B, C, D), C (number of nodes with historical bottleneck records) = 2 (C, D), calculate vector V:

[0073] ((2+2+3+2+3) / (5) 3) = 12 / 15 = 0.8, (1+0+1) / 3 ≈ 0.67, (5+2) / 2 = 3.5);

[0074] Among them, 0.8 indicates that the path is generally long and the transmission efficiency baseline is poor; 0.67 indicates that the network's tolerance for bottleneck nodes is average; and 3.5 indicates that nodes C and D have serious historical problems, and the reliability of this node set is low. The preliminary congestion index V=[0.8,0.67,3.5]. 0.8 and 3.5 dominate, indicating that the problem stems from long paths and high historical risks.

[0075] Based on V, the vector generation algorithm maps path length and path redundancy to a route optimization vector [0.9, -0.2, 0.1]. This vector strongly points in the direction of shortening the main path (A→B→C), that is, guiding traffic to the alternative path. The overflow rate of the node buffer within the impact sub-element was collected, and it was found that the overflow rate of node C was 8% (exceeding the 5% threshold), the number of overflowing nodes was 1, and the congestion risk coefficient = 1 / 3≈0.33, indicating that there is a certain risk of resource collapse in this node set.

[0076] Generate the final report: Extract the peak traffic data of 1.2Gbps from the operation log data; establish a corresponding relationship table and perform matching. When the peak traffic is >1Gbps and the risk coefficient is >0.3, the strategy indicated by the routing optimization vector [0.9,-0.2,0.1] should be adopted.

[0077] Detailed congestion characteristic data is finally generated:

[0078] Congestion level: Severe (based on indicator V and peak value);

[0079] Scope of impact: Core layer and convergence layer (based on the impact sub-elements of the congestion impact domain);

[0080] Risk level: Medium (based on a congestion risk coefficient of 0.33);

[0081] Optimization suggestion: Partially schedule the backbone traffic passing through nodes B→C→D to the backup paths A→D→C and A→B→E to shorten the path and avoid nodes with high historical risk.

[0082] In the specific implementation process of the above-described embodiments of this application, in step S20, "based on the node spacing distribution data and the path hop count, the node adjacency density is calculated using a density calculation algorithm (such as DBSCAN)". Simply using a clustering algorithm to calculate the node adjacency density may cause distortion of the coordinate distribution of nodes in the feature space due to some reason such as communication interval jitter between nodes or data packet transmission delay fluctuations during communication, thereby causing adjacency density calculation errors, and ultimately leading to excessively large errors in the subsequent calculation of connection strength values ​​and the generation of influence sub-elements of the congestion influence domain.

[0083] Analyzing the above S10-S50 process, firstly, the operation log data and communication record data of the nodes are collected through a distributed sensor network. Based on the source and destination address information in the communication records and combined with the pre-set server node connection topology, the node spacing distribution data is calculated, the logical distance between any two nodes in the network is quantified, the communication path corresponding to each communication record is extracted, the number of intermediate nodes in the path is counted to generate the path hop count to assess the path relay burden, the queuing time is extracted from the operation log, and the transmission packet loss ratio is counted from the communication records. The two are then fused to calculate the node load data. When the node load data exceeds the first load threshold, the node is marked as an over-threshold element, and abnormal links are located from the communication path based on the over-threshold element. The ratio of the number of abnormal links to the total number of links is counted, and the abnormal status index (overall link abnormal index) is calculated. This enables real-time monitoring and anomaly detection of network nodes and links. By quantifying the topology and dynamic performance, basic data is provided for subsequent analysis.

[0084] Furthermore, based on node spacing distribution data and path hop counts, node adjacency density is calculated using density calculation algorithms (such as DBSCAN) to characterize the tightness of connections between nodes and their surrounding nodes, identifying network hub nodes; the topology depth is calculated using depth-first traversal algorithms to define the network's center-edge architecture; connection strength values ​​are calculated using node adjacency density and topology depth; and path connectivity distribution is generated based on these connection strength values ​​(this path connectivity distribution reflects the theoretical smoothness of each communication path under ideal no-load conditions). Combined with data queuing time, transmission packet loss ratio, and abnormal state indicators, transmission bottleneck nodes are identified; and so on. With the bottleneck node as the core, a set of network child nodes containing the neighboring nodes of the bottleneck node is extracted from the node topology to reduce the dimensionality of the analysis. Communication paths are traversed in the set of network child nodes, and the congestion level of each communication path is calculated by combining connection strength value, data queuing time and transmission packet loss ratio. When the congestion level exceeds a preset congestion threshold, the node corresponding to the path is marked as a high-load node, and the data is summarized and updated in the bottleneck node set. Based on the spatial distribution of the updated bottleneck node set in the set of network child nodes, the influence sub-elements of the congestion influence domain are generated by clustering algorithm, realizing the transformation from point fault location to area influence range definition.

[0085] Furthermore, based on the sub-elements of the congestion impact domain, when an element exceeding the threshold is located within the sub-elements and the node load data exceeds the second load threshold, a system-level congestion signal is triggered to confirm the existence of severely overloaded nodes within the congestion impact domain, activating subsequent in-depth analysis and optimization processes. After the system-level congestion signal is triggered, based on the path connectivity distribution, the path length and path redundancy information of all affected routing paths within the sub-elements of the congestion impact domain are extracted. The node processing bottleneck history recorded within a preset time window is collected, and combined with the path length and path redundancy information, a preliminary congestion index is calculated to quantify the overall congestion pressure of the congestion impact domain in terms of the current path status and historical bottleneck tendencies. Based on the preliminary congestion index, the path length and path redundancy are mapped into a multi-dimensional route optimization vector through a vector generation algorithm, with its elements representing the optimized path direction, providing control instructions for path storage and scheduling.

[0086] Simultaneously, by collecting buffer overflow rate records of each node within the affected sub-elements of the congestion impact domain, the number of overflowing nodes is counted, and the congestion risk coefficient is calculated using the ratio of the number of overflowing nodes to the total number of nodes within the affected sub-elements. This reflects the server's risk of crashing at the resource level. Peak traffic data is extracted from the operation log data, and the peak traffic data, congestion risk coefficient, and routing optimization vector are matched and correlated to establish a correspondence table among the three. This assesses the congestion risk and optimization path selection under different traffic levels, and finally generates detailed congestion characteristic data containing congestion degree, impact range, risk level, and optimization suggestions, providing comprehensive reference information for storage path adjustment.

[0087] Specifically, in step S20, based on the node spacing distribution data and the path hop count, the node adjacency density is calculated using a density calculation algorithm (such as DBSCAN), including the following steps:

[0088] Step S21: Traverse each node, select the communication path that passes through the node from all communication paths, and use it as the first communication path; calculate the number of character jumps for each communication path; calculate the average number of character jumps for all first communication paths corresponding to the node as the average number of path jumps for the node.

[0089] It should be noted that in the above embodiments of this application, the calculation of the role jump count of the node is as follows: if the node is at the beginning or end of the first communication path, the role jump count is 0 (because it represents the endpoint of communication, not an intermediate jump point); if the node is inside the first communication path, the role jump count is the path jump count of the first communication path.

[0090] The average path hop count described above represents the average frequency and depth at which the node plays the role of a relay station in network communication. The lower the average path hop count, the more likely the node is to be an endpoint or close to an endpoint in communication; the higher the value, the more frequently the node becomes a core hub for data forwarding.

[0091] Step S22: Extract multiple node pairs based on the communication path (a node pair refers to a node pair in the communication record data where there is data exchange between node 1 and node 2, that is, the two corresponding nodes in the above link are called a node pair); extract the time interval between the sending of continuous data packets (continuous data packets are multiple consecutive data packets) and the receiving of data packets between the node pairs from the communication record data (the time interval is the sequence of time intervals between the sending and receiving of each data packet); calculate the standard deviation of the delay fluctuation of the time interval;

[0092] It should be noted that the delay fluctuation standard deviation in the above embodiments of this application (the delay fluctuation standard deviation is the average value of the interval time series, which can be called the average delay, and then the delay standard deviation is calculated using the average delay. The calculation of the standard deviation is common knowledge to those skilled in the art and will not be described in detail in this application, and is the delay fluctuation standard deviation) characterizes the stability of the communication link between node pairs. The larger the delay fluctuation standard deviation, the more severe the communication delay fluctuation between the node pairs, and the worse the stability.

[0093] Step S23: For each node, the mean standard deviation of the delay jitter of all node pairs (that is, the node pairs consisting of two nodes corresponding to all links through that node) is used to obtain the comprehensive jitter impact factor for each node.

[0094] It should be noted that the comprehensive jitter impact factor in the above embodiments of this application reflects the overall communication stability of the local network where the node is located. In the subsequent density clustering algorithm, this factor is used to dynamically adjust the search radius. Nodes with poor stability (large comprehensive jitter impact factor value) need a larger search radius to accommodate the uncertainty of their coordinates. For example, if the standard deviation of the latency fluctuation between node C and its neighbors D, E, and F is 1.5ms, 2.0ms, and 2.5ms respectively, then the comprehensive jitter impact factor of node C is (1.5+2.0+2.5) / 3=2.0ms. The higher this value, the worse the network communication stability of the area where node C is located.

[0095] Step S24: Based on the average path jump count and node spacing distribution data, map each node into a two-dimensional feature space point;

[0096] It should be noted that in the above embodiments of this application, the horizontal coordinate of the two-dimensional feature space point is the normalized average distance between each node and other nodes in the node spacing distribution data, indicating the average logical position of the node; the vertical coordinate is the average number of path hops of each node; the mapping processing of the above embodiments of this application transforms the network topology information into a data form that can be used for clustering algorithms, wherein the coordinates of each node comprehensively reflect its spatial position and functional role.

[0097] Step S25: Adaptively set the dynamic radius of each node in the density calculation algorithm using the comprehensive jitter influence factor of each node;

[0098] The formula for calculating the dynamic radius is:

[0099] ;

[0100] In the formula, Let be the dynamic radius of the i-th node in the density calculation algorithm. The basic neighborhood radius (pre-set by the network topology scale, its value is set according to the overall distribution range of the coordinates of the nodes in the two-dimensional feature space, usually a certain proportion of the median of the Euclidean distance between all nodes (such as 10%)). Let be the comprehensive jitter impact factor of the i-th node. The jitter compensation coefficient (which maps the comprehensive jitter impact factor to the radius increment, is a normal number pre-calibrated based on the historical stable state of the network, used to adjust the intensity of the impact of communication jitter on the neighborhood radius, and can be determined by grid search method with the robustness of clustering results during the stable period as the objective)

[0101] It should be noted that the above embodiments of this application are for a communication with poor stability (i.e., the above-mentioned...). Nodes with high values ​​may have significant drift errors in their coordinates in the feature space due to communication delay fluctuations. This can be mitigated by assigning them a larger neighborhood radius. > This enables the density calculation algorithm to accommodate the coordinate uncertainty caused by instantaneous instability and time delay fluctuations, avoiding the incorrect exclusion of its original neighbors from the cluster. It is equivalent to providing each node with a fuzzy search range that matches its communication stability.

[0102] Step S26: Based on the dynamic radius of each node, cluster each two-dimensional feature space point to obtain multiple clusters; count the total number of nodes contained in the corresponding cluster for each node to obtain the node adjacency density of each node.

[0103] It should be noted that, in the above embodiments of this application, when analyzing node adjacency density, the statistical analysis of communication interval jitter is combined. The standard deviation of the arrival time interval of consecutive data packets in the communication record data is calculated and introduced as a weighting factor into the neighborhood radius parameter compensation of the density calculation algorithm, thereby eliminating the interference of instantaneous communication instability on topology analysis and accurately analyzing and obtaining node adjacency density.

[0104] Specifically, in step S24, each node is mapped to a two-dimensional feature space point based on the average path hop count and node spacing distribution data, including the following steps:

[0105] Step S241: Extract the node spacing distance from node i to each node in the current server topology from the node spacing distribution data. Calculate the basic x-coordinate of each node based on the node spacing distance of each node using a normalized average. Use the average path hop count corresponding to each node as the basic y-coordinate.

[0106] Step S242: Extract the CPU utilization time series of each node from the running log data, and calculate the coefficient of variation (i.e., the ratio of standard deviation to mean) of the CPU utilization time series as the load fluctuation factor.

[0107] It should be noted that the CPU utilization time series of the node in the above embodiments of this application refers to the percentage of CPU workload of the node in a continuous time, which represents the busyness of the node's own computing resources. High CPU utilization indicates that the node is processing computing tasks at full capacity.

[0108] Step S243: Calculate the proportion of communication paths containing node i that have functionally equivalent backup paths, and use this as the path redundancy factor for node i.

[0109] It should be noted that in the above embodiments of this application, the functionally equivalent backup path refers to a communication path from the same source node to the same destination node that does not pass through node i. For example, if there are two communication paths in the current network, namely path 1A→B→C→D and path 2E→F→C→G, for node C, the communication paths including it are path 1 and path 2. For path 1, we search for whether there is another communication path starting from node A and ending at node D that does not pass through node C. For example, we find path A→H→I→D. That is to say, there is one backup path for path 1. For path 2, we cannot find a communication path starting from node E and ending at node G. That is, path 2 has no backup path; then, by dividing the number of communication paths with backup paths (in the example above, this number is 1 for node C) by the total number of communication paths containing node i (in the example above, this number is 2 for node C), we obtain the path redundancy factor of node i. The higher the path redundancy factor, the more it means that even if this node fails, most of the communication passing through this node has a detour, this node is not very critical, and its path redundancy factor is high. Conversely, it means that this node is the only bridge on many communication paths, and once it is congested or fails, a large amount of communication will be interrupted, this node is very critical, and its path redundancy factor is low.

[0110] Step S244: Correct the basic horizontal and vertical coordinates based on the comprehensive jitter impact factor, path redundancy factor, and load fluctuation factor to obtain the corrected horizontal and vertical coordinates.

[0111] The above-mentioned correction abscissa is calculated as follows:

[0112] ;

[0113] In the formula, Let be the corrected x-coordinate of node i. Let i be the basic x-coordinate of node i. The comprehensive jitter impact factor of node i. Let be the load fluctuation factor of node i;

[0114] The above-mentioned corrected ordinate is calculated as follows:

[0115] ;

[0116] In the formula, Let be the corrected ordinate of node i. Let i be the basic ordinate of node i. The path redundancy factor for node i;

[0117] In the above formula, Stability decay coefficient (0 < <1, used to weaken the credibility of highly jittery node coordinates). For load fluctuation compensation coefficient ( >0, used to compensate for coordinate offset caused by high CPU fluctuations). Redundancy enhancement coefficient ( >0, used to enhance the functional weight of low-redundancy critical nodes), the above stability decay coefficient Load fluctuation compensation coefficient Redundancy enhancement coefficient The values ​​of these factors are all obtained based on empirical values;

[0118] The above-described embodiments of this application suppress coordinate drift of unstable nodes by combining jitter impact factor and load fluctuation factor, and enhance the functional role weight of low-redundancy key nodes by path redundancy factor, so that the mapping result reflects both the steady-state topology characteristics of the nodes and the importance of the network structure.

[0119] Step S245: Obtain the two-dimensional feature space points of each node based on the corrected abscissa and corrected ordinate.

[0120] It should be noted that in the above embodiments of this application, the normalized average value is calculated using the node spacing distribution data as the basic horizontal axis, and the average number of path jumps is directly used as the basic vertical axis to establish the initial position of the node in the feature space. The horizontal axis reflects the average logical position of the node, and the vertical axis reflects its functional role as a data transfer hub, providing a reference coordinate system for subsequent correction.

[0121] Furthermore, this embodiment extracts the CPU utilization time series from the operation log and calculates the coefficient of variation to obtain the load fluctuation factor, quantifying the stability of the node's own computing resources, and transforming the dynamic load characteristics of the node into quantifiable parameters, providing load dimension input data for subsequent coordinate correction; this embodiment calculates the path redundancy factor by statistically analyzing the proportion of functionally equivalent backup paths in the communication paths containing the node, assessing the structural criticality of the node in the network topology, identifying critical nodes lacking backup paths, and providing a basis for structural importance for subsequent correction of node functional role weights;

[0122] The above steps take into account several core influencing factors. Based on the comprehensive jitter impact factor, load fluctuation factor, and path redundancy factor, the basic horizontal and vertical coordinates are synergistically corrected. When correcting the horizontal coordinate, the position drift of unstable nodes is suppressed by the stability attenuation coefficient and the load fluctuation compensation coefficient. When correcting the vertical coordinate, the functional role of low-redundancy key nodes is strengthened by the redundancy enhancement coefficient. This achieves the fusion of topological characteristics, stability characteristics, and structural characteristics, generating accurate coordinates that can simultaneously reflect the steady-state characteristics of nodes and the importance of the network.

[0123] The mapping process described in the embodiments of this application is used to transform network topology information into a data form that can be used in clustering algorithms, wherein the coordinates of each node comprehensively reflect its spatial location and function; the above steps provide a technical basis for realizing node adjacency density calculation. Furthermore, steps 241-244 also provide detailed reference information and a technical basis for finally calculating the connection strength value using node adjacency density and topology hierarchy depth, and for finally determining the transmission bottleneck node.

[0124] Example 2

[0125] This invention provides a data storage method for massive transaction information in embodiment two. The operation execution logic of the method is the same as that of embodiment one above, and will not be repeated here. The difference is that, in the specific implementation process of the above embodiments of this application, the technicians found that when optimizing the core financial transaction network, there are a small number of hub nodes (such as core exchange nodes) in these networks that carry most of the critical traffic of the entire network and are structurally irreplaceable. When these nodes are overloaded, the conventional detour strategy based on instantaneous state is extremely risky. Simple detour may not only significantly increase the delay due to the detour, but may also instantly transfer the load pressure to other equally vulnerable core nodes, causing systemic risks.

[0126] Therefore, the analysis of these core hub nodes needs to go beyond their instantaneous state to simulate and quantify "how great the communication cost and stability risk of the entire network would be if this node were lost", so as to provide decision-makers with the ultimate basis for whether to detour, when to detour, and how to detour. See steps S241' to S245' below for details.

[0127] Specifically, in step S24, each node is mapped to a two-dimensional feature space point based on the average path hop count and node spacing distribution data, including the following steps:

[0128] Step S241': Extract the node spacing distance from node i to each node in the current server topology from the node spacing distribution data. Calculate the basic x-coordinate of each node based on the normalized average of the node spacing distances. Use the average path hop count corresponding to each node as the basic y-coordinate.

[0129] Step S242': Based on the preset server node topology and node spacing distribution data, perform a virtual failure operation on each node in the network; after node i is virtually failed, calculate a new optimal path for all communication paths passing through node i using a path planning algorithm, and ensure that the new path does not pass through node i (because node i has already performed a virtual failure); calculate the path stretching rate by calculating the ratio of the average path length of all new paths to the average path length of the original paths; at the same time, extract all direct neighbor nodes of node i (direct neighbor nodes refer to nodes in the preset server node topology that have a direct physical or logical connection with the current node and can communicate without passing through any other intermediate nodes), and calculate the average growth rate of the node load data of these neighbor nodes before and after the virtual failure operation (the calculation method is the same as in S10, which integrates data queuing time and transmission packet loss ratio), as the neighbor load increment;

[0130] It should be noted that in the above embodiments of this application, the virtual failure operation is a simulation operation performed in the server memory and does not affect the actual operation of the network. The path stretching rate quantifies the additional transmission delay caused by bypassing node i, while the neighbor load increment quantifies the local load impact caused when the traffic pressure is transferred from node i to its surrounding nodes. The node spacing distribution data serves as a cost metric for path planning, ensuring that the new path calculation is based on the logical distance and transmission cost between nodes. These two indicators together constitute the dynamic basis for evaluating the network role of node i.

[0131] Step S243': Combine the path stretching rate and the neighbor load increment to calculate the detour criticality index of node i; normalize the neighbor load increment separately to obtain the detour vulnerability index of node i;

[0132] Among them, the key index for detours (K) i The formula for calculating K is obtained through linear weighting. Formula: K i = λ1 S i + λ2 ΔL i S i S represents the path stretching ratio of node i. i ≥ 1, the larger the value, the more severe the path detour caused by the detour. ΔL i ΔL represents the load increment of node i's neighbors.i A value ≥ 0 indicates a more dramatic increase in the load of neighboring nodes and a higher local risk. λ1 and λ2 are preset fusion weight coefficients, which typically satisfy λ1 + λ2 = 1 for standardized fusion, or their relative magnitudes are set empirically. λ1 reflects the emphasis on "global structural cost," while λ2 reflects the emphasis on "local stability risk." This formula fuses two indicators with different dimensions (multiplier and growth rate) into a single scalar exponent K, representing the overall bypass cost of node i, through a weighted sum. i Of course, K i The higher the value, the greater the overall cost and risk of bypassing the node.

[0133] It should be noted that in the above embodiments of this application, the detour criticality index is a comprehensive scalar. The higher the value, the more core node i is in the network path structure, and the greater the system cost required to bypass it. The detour vulnerability index specifically characterizes the robustness of the local network environment in which node i is located. The higher the value, the more likely the set of neighboring nodes of node i is to be overloaded when subjected to additional traffic, thus making the operation risk of detouring node i extremely high.

[0134] Among them, the bypass vulnerability index (Vul i The calculation involves normalizing ΔLi. The calculation formula is: Vul i = ΔLi / ΔLmax (where ΔLmax is the maximum value of ΔLi for all nodes in this simulation). Output: The calculated standardized vulnerability index.

[0135] Specifically, ΔLmax represents the ΔL of all evaluated nodes (such as nodes within the congestion domain or key nodes in the entire network) within the current analysis period. i The maximum value in the set. ΔLmin represents the minimum value in the corresponding set. In practice, if the load on all nodes is increasing, ΔL_min is often set to 0 to simplify calculations. This bypass vulnerability index (Vul i The calculation formula maps the neighbor load increment of node i to the interval [0, 1]. i = 0 indicates that its neighbors are the least sensitive to additional traffic (most robust), Vul i = 1 indicates that its neighbors are most sensitive (most vulnerable) to additional traffic.

[0136] Step S244': Correct the basic abscissa and basic ordinate based on the detour criticality index and the detour vulnerability index to obtain the corrected abscissa and corrected ordinate;

[0137] The corrected x-coordinate is calculated as follows:

[0138] ;

[0139] The corrected ordinate is calculated as follows:

[0140] ;

[0141] In the formula, For the corrected x-coordinate, For the corrected ordinate, Based on the horizontal axis, Based on the ordinate, The circumvention vulnerability index (Vul) of node i i ), This is the vulnerability impact coefficient (preset value). The detour criticality index (i.e., K) for node i i ), This is the critical impact coefficient (preset value);

[0142] The correction logic in the above-described embodiments of this application is as follows: based on the results of the simulated rerouting test, the position of the node in the feature space is shifted to a core offset, bypassing the vulnerability index. For nodes with high elevations, their corrected abscissas will increase significantly, shifting to the right in the feature space. This reflects a higher risk to local network stability when performing detours around these nodes, thus classifying them as high-risk management areas in subsequent clustering, and thus reducing the detour critical index. A node with a high ordinate will have its corrected ordinate significantly increased, indicating that the node has an indispensable core position in the global connectivity of the network, making it stand out as a core hub node in the feature space.

[0143] Step S245': Obtain the two-dimensional feature space points of each node based on the corrected abscissa and corrected ordinate.

[0144] It should be noted that: the above-described embodiments of this application introduce path rerouting to simulate stress testing, transforming the baseline of node mapping from the current state to the impact of potential operations. The resulting two-dimensional feature space points not only reflect the structural characteristics of the network, but also embed the expected costs and risks of adopting detour optimization strategies for different nodes. The node adjacency density obtained by clustering these space points, as well as the subsequently calculated connection strength values ​​and the generated congestion impact domain, will directly serve the formulation of optimization strategies: for nodes with high correction ordinates and low abscissas, it is recommended to adopt proactive path detours; for nodes with both high correction ordinates and abscissas, it is necessary to issue warnings and recommend more conservative optimization measures such as resource allocation or traffic shaping.

[0145] Examples of the technical solutions adopted in the embodiments of this application are provided below:

[0146] The current server contains the following: Figure 5 The six key nodes shown are A, B, C, D, E, and F, with a pre-set vulnerability impact coefficient ω=0.2 and a criticality impact coefficient ψ=0.3. The basic abscissas of each node are calculated from the node spacing distribution data: Node A: 0.1 (network core, close to other nodes), Node B: 0.3, Node C: 0.4, Node D: 0.8 (network edge), Node E: 0.5, Node F: 0.7. The basic ordinates are the average number of path hops for each node obtained from communication records: Node A: 0.2 (often used as a communication endpoint), Node B: 1.8 (frequently used as a relay node), Node C: 1.5, Node D: 0.3, Node E: 2.2 (critical relay hub), Node F: 0.4.

[0147] Then, taking node E as an example, a virtual failure handling operation is performed. The original path through E is: B→E→C, with a path length of 2. After the virtual failure, the new path is: B→A→C, with a path length of 3. Its path stretching rate is 3 / 2 = 1.5.

[0148] Then, node E's direct neighbors are nodes B and C. Before the virtual failure, the neighbor load data was 0.65 for node B (calculated based on a data queuing time of 80ms and a packet loss rate of 2%), and 0.60 for node C. After the virtual failure, the neighbor load data was 0.82 for node B (which took over part of the original traffic of E), and 0.78 for node C. Therefore, the calculation of the neighbor load increment is as follows:

[0149] Neighbor load increment = [(0.82-0.65) / 0.65 + (0.78-0.60) / 0.60] / 2 ≈ 0.28;

[0150] Furthermore, by fusing the path stretching ratio (1.5) and the neighbor load increment (0.28), we obtain:

[0151] Calculation formula: =1.5×0.6+0.28×0.4=1.012 (assuming weights of 0.6 and 0.4 respectively), the bypass vulnerability index is the normalized value of the neighbor load increment = 0.28 (which is within a reasonable range);

[0152] Furthermore, the coordinates are corrected, that is:

[0153] Corrected x-axis: 0.5 + 0.2 × 0.28 = 0.556;

[0154] Corrected ordinate: 2.2 + 0.3 × 1.012 = 2.504;

[0155] Therefore, the two-dimensional feature space point of node E is (0.556, 2.504).

[0156] In summary, the present invention proposes a data storage method for massive transaction information. This method first calculates anomaly indicators (quantifying overall link anomaly indicators) to achieve real-time monitoring and anomaly detection of network nodes and links. Then, it further calculates node adjacency density using a density calculation algorithm (such as DBSCAN) to characterize the tightness of connections between nodes and their surrounding nodes, identifying network hub nodes and transmission bottleneck nodes. Using all transmission bottleneck nodes as the core, it extracts a set of network child nodes containing neighboring nodes of the transmission bottleneck nodes from the node topology, achieving dimensionality reduction in the analysis scope. It traverses communication paths within the set of network child nodes, combining connection strength values, data queuing time, and packet loss ratios to calculate the congestion level of each communication path. When the congestion level exceeds a preset congestion threshold, the corresponding node of that path is marked as a high-load node, and this is aggregated and updated in the bottleneck node set.

[0157] Then, the scope of influence is further expanded. Based on the spatial distribution of the updated bottleneck node set in the network sub-node set, clustering algorithms are used to generate impact sub-elements of the congestion impact domain. By analyzing the impact sub-elements of the congestion impact domain and node load data, it is confirmed that there are severely overloaded nodes within the congestion impact domain, activating the subsequent in-depth analysis and optimization process. After the system-level congestion signal is triggered, based on the path connectivity distribution, the path length and path redundancy information of all affected routing paths within the impact sub-elements of the congestion impact domain are extracted to provide control processing for subsequent path storage scheduling. At the same time, the data storage method for massive transaction information proposed in this embodiment is also the technical foundation for realizing large-scale massive data storage path detection and adjustment.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; those skilled in the art can modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for storing massive amounts of transaction information, characterized in that, The following steps are included: The system accesses the original transaction information of the online trading system in real time, and parses and extracts the original transaction information. At the same time, during the above process, each node generates and stores original operation log data and communication records, including data queuing time and transmission packet loss ratio indicators. The system collects operation log data and communication record data of each node through a distributed sensor network. Based on the communication record data, it calculates the node spacing distribution data and path hop count. It extracts data queuing time and transmission packet loss ratio from the operation log data and fuses and calculates node load data. When the node load data exceeds the first load threshold, it is marked as an over-threshold element. Abnormal links are located based on the over-threshold elements. The abnormal status index is obtained by calculating the ratio of the number of abnormal links to the total number of links. Based on the node spacing distribution data and the path hop count, the node adjacency density is calculated using a density calculation algorithm, and the topology depth is calculated using a depth-first traversal algorithm. The connection strength value is calculated using the node adjacency density and the topology depth, and a path connectivity distribution is generated based on the connection strength value. The transmission bottleneck node is identified by combining the data queuing time, the transmission packet loss ratio, and the abnormal state indicators. Using all the transmission bottleneck nodes as the core, a set of network child nodes containing the neighbor nodes of the transmission bottleneck nodes is extracted from the node topology. Communication paths are traversed within this set of network child nodes, and the congestion level of each communication path is calculated by combining the connection strength value, the data queuing time, and the transmission packet loss ratio. When the current congestion level exceeds the preset congestion threshold, all nodes corresponding to the current communication path are marked as high-load nodes, and all high-load nodes are aggregated and updated to the bottleneck node set; based on the spatial distribution of the updated bottleneck node set in the network sub-node set, the influence sub-elements of the congestion influence domain are generated by clustering algorithm. Based on the sub-elements of the congestion impact domain, when the over-threshold element is located within the sub-elements and the node load data exceeds the second load threshold, a system-level congestion signal is triggered. After triggering a system-level congestion signal, the path length and redundancy information of the affected routing paths are extracted based on the path connectivity distribution. Combined with the historical bottleneck processing of nodes, preliminary congestion indicators are calculated, and a route optimization vector is generated. Collect buffer overflow rate records of each node within the sub-elements of the congestion impact domain, calculate the ratio of the number of overflowing nodes to the total number of nodes to obtain the congestion risk coefficient, extract peak traffic data from the operation log data, and perform correlation analysis between peak traffic data and congestion risk coefficient and routing optimization vector to generate congestion feature data. The node adjacency density is calculated using a density calculation algorithm based on the node spacing distribution data and the path hop count, including the following steps: Iterate through each node, select the communication path that passes through the node from all communication paths, and use it as the first communication path; calculate the number of character jumps for the node for each communication path; calculate the average number of character jumps for the node for all first communication paths corresponding to the node. Multiple node pairs are extracted based on the communication path; the time interval between the sending and receiving of continuous data packets between node pairs is extracted from the communication record data; the standard deviation of the delay fluctuation of this time interval is calculated. The mean standard deviation of the delay jitter for all node pairs corresponding to each node is used to obtain the comprehensive jitter impact factor for each node. Based on the average path jump count and node spacing distribution data, each node is mapped to a two-dimensional feature space point. The dynamic radius of each node in the density calculation algorithm is adaptively set by utilizing the comprehensive jitter influence factor of each node. Based on the dynamic radius of each node, the two-dimensional feature space points are clustered to obtain multiple clusters; the total number of nodes contained in the corresponding cluster for each node is counted to obtain the node adjacency density of each node.

2. The data storage method for massive transaction information according to claim 1, characterized in that, The process involves calculating node spacing distribution data and path hop counts based on communication record data, extracting data queuing time and packet loss ratio from operation log data, fusing node load data, marking nodes as exceeding a first load threshold when their load data exceeds a first load threshold, locating abnormal links based on these elements, and calculating the ratio of the number of abnormal links to the total number of links to obtain an abnormal status index. This process includes the following steps: The system collects operation log data and communication record data from each node through a distributed sensor network. Based on the source and destination address information in the communication record data, and combined with a preset server node connection topology, it calculates the node spacing distribution data. It extracts the communication path corresponding to each communication record data, counts the number of intermediate nodes in each communication path to generate the path hop count, extracts the data queuing time of each node from the operation log, and counts the transmission packet loss ratio of each node from the communication record. It merges the data queuing time and the transmission packet loss ratio to calculate the node load data corresponding to each node. When the node load data exceeds a first load threshold, the node is marked as an over-threshold element, and abnormal links are located from each communication path based on the over-threshold element. It calculates the abnormal state index by calculating the ratio of the number of abnormal links to the total number of links.

3. The data storage method for massive transaction information according to claim 2, characterized in that, After triggering a system-level congestion signal, the path length and redundancy information of the affected routes are extracted based on the path connectivity distribution. Combined with the historical bottleneck processing data of each node, preliminary congestion indicators are calculated, and a route optimization vector is generated. The process includes the following steps: After the server detects a system-level congestion signal, it extracts the path length and path redundancy information of all affected routing paths within the affected sub-elements of the congestion impact domain based on the path connectivity distribution. The system collects the node processing bottleneck history recorded within a preset time window, and calculates the path length and path redundancy information by combining the node processing bottleneck history to obtain the preliminary congestion index within the influence sub-elements of the congestion influence domain. Based on the preliminary congestion index, the system maps the path length and path redundancy into a multi-dimensional route optimization vector through a vector generation algorithm.

4. The data storage method for massive transaction information according to claim 3, characterized in that, Collect buffer overflow rate records for each node within the affected sub-elements of the congestion impact domain, calculate the ratio of the number of overflowing nodes to the total number of nodes to obtain the congestion risk coefficient, extract peak traffic data from the operation log data, and perform correlation analysis between the peak traffic data and the congestion risk coefficient and routing optimization vector to generate congestion feature data. The process includes the following steps: Collect and obtain the buffer overflow rate records of each node in the sub-element of the congestion impact domain, count the number of nodes in the sub-element of the congestion impact domain whose buffer overflow rate exceeds the preset overflow rate threshold, obtain the number of overflow nodes, and calculate the congestion risk coefficient by using the ratio of the number of overflow nodes to the total number of nodes in the sub-element of the congestion impact domain. Extract peak traffic data from the runtime log data; The peak traffic data is matched and correlated with the congestion risk coefficient and the route optimization vector to generate congestion feature data.

5. A method for storing massive amounts of transaction information according to claim 4, characterized in that, The congestion level value is calculated as follows: For each link on the communication path, the ratio of the average data queuing time of the link to the maximum queuing time of all links is added to the ratio of the transmission packet loss ratio of the link to the maximum packet loss ratio of all links. Then, the ratio of the connection strength value of the link to the maximum connection strength value of all links is subtracted. The sum of the sums of all links on the communication path is then divided by the total number of links in the path. The average value obtained is the congestion level value of the communication path. The calculation of the preliminary congestion index includes three parts: First, the sum of the path lengths of all affected routing paths within the congestion impact domain is calculated and then divided by the product of the total number of affected routing paths and the maximum path length among all affected routing paths; Second, the sum of the path redundancy of all nodes in the bottleneck node set within the domain is calculated and then divided by the total number of nodes; Third, the sum of the bottleneck frequencies of all nodes within the congestion impact domain that have experienced bottlenecks is calculated and then divided by the total number of these nodes; Finally, the calculation results of the first, second, and third parts together constitute the preliminary congestion index vector.

6. A method for storing massive amounts of transaction information according to claim 5, characterized in that, Based on the average path hop count and node spacing distribution data, each node is mapped to a two-dimensional feature space point, including the following steps: Extract the node spacing distance from node i to each node in the current server topology from the node spacing distribution data. Calculate the basic x-coordinate of each node by normalizing the node spacing distance based on the node spacing distance. The average number of path jumps corresponding to each node is used as the basic vertical axis. Extract the CPU utilization time series of each node from the operation log data, and calculate the coefficient of variation of the CPU utilization time series as the load fluctuation factor. The proportion of communication paths containing node i that have functionally equivalent backup paths is calculated and used as the path redundancy factor for node i. Based on the comprehensive jitter impact factor, combined with the path redundancy factor and the load fluctuation factor, the basic horizontal and vertical coordinates are corrected to obtain the corrected horizontal and vertical coordinates. The two-dimensional feature space points of each node are obtained based on the corrected abscissa and corrected ordinate.

7. A method for storing massive amounts of transaction information according to claim 5, characterized in that, Based on the average path hop count and node spacing distribution data, each node is mapped to a two-dimensional feature space point, which also includes the following steps: Extract the node spacing distance from node i to each node in the current server topology from the node spacing distribution data. Calculate the basic x-coordinate of each node by normalizing the node spacing distance based on the node spacing distance. The average number of path jumps corresponding to each node is used as the basic vertical axis. Based on pre-set server node topology and node spacing distribution data, a virtual failure operation is performed on each node in the network. After node i is virtually failed, a new optimal path is calculated for all communication paths passing through node i using a path planning algorithm, ensuring that the new path does not pass through node i. The path stretching rate is calculated by comparing the average path length of all new paths with the average path length of the original paths. At the same time, all direct neighbor nodes of node i are extracted, and the average growth rate of the node load data of these neighbor nodes before and after the virtual failure operation is calculated as the neighbor load increment. By combining the path stretching rate and the neighbor load increment, the detour criticality index of node i is calculated; The neighbor load increment is normalized separately to obtain the bypass vulnerability index of node i; The basic horizontal and vertical coordinates are corrected based on the detour criticality index and the detour vulnerability index to obtain the corrected horizontal and vertical coordinates. The two-dimensional feature space points of each node are obtained based on the corrected abscissa and corrected ordinate.

8. A data storage device for massive transaction information, characterized in that, The device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the data storage method for massive transaction information as described in any one of claims 1-7.

Images