A method for intelligent exchange and integration of heterogeneous data in digital enterprises

Abstract

The present invention relates to a method for intelligent exchange and integration of heterogeneous data of a digital enterprise, comprising the following steps: obtaining communication frequency data between nodes in a distributed system; dynamically adjusting the timestamp array dimension and compression ratio of a vector clock according to a node event counter; adding a timestamp to the timestamp array of each compressed vector clock; dynamically clipping the timestamp array of the vector clock according to the incremental update result; identifying the vector clocks with high frequency access in the system, and caching their timestamp arrays in a distributed memory cache system; monitoring the event concurrency index and vector clock distribution of the distributed system in real time, and dynamically optimizing the vector clock storage and event sorting in the distributed system through concurrency-aware adaptive adjustment. The method of the present invention effectively reduces the storage overhead of the vector clock, improves the efficiency of event sorting, and ensures the accuracy of causal relationship judgment, thereby significantly improving the overall performance of the distributed system.

Description

Technical Field

[0001] The present invention relates to the technical field of heterogeneous data conversion, and in particular to a method for intelligent exchange and integration of heterogeneous data of a digital enterprise. Background Art

[0002] In large-scale distributed systems, in order to ensure causal consistency, vector clocks are usually used to record the order of events. However, with the continuous expansion of system scale and the increase in the number of nodes, the storage overhead of vector clocks also increases. Especially in high-concurrency, multi-node enterprise-level data exchange scenarios, massive data synchronization requests will cause vector clocks to occupy a large amount of storage space, which poses a severe challenge to the scalability and performance of the system. How to reduce the storage space occupied by vector clocks through compression or clipping technology while ensuring causal consistency has become a key issue that needs to be solved urgently. On the one hand, compression technology can reduce the size of vector clocks through encoding or dictionary mapping, but the compression process itself will also bring additional computing overhead, affecting the real-time performance of the system; on the other hand, clipping technology can regularly clean up expired or no longer needed vector clock information, but the clipping granularity is difficult to grasp, and excessive clipping may destroy causal dependencies and affect data consistency; at the same time, the introduction of compression and clipping technology will also increase system complexity, bringing new challenges to operation and maintenance and fault location; therefore, how to balance storage overhead and system performance and find the optimal vector clock compression and clipping strategy is a technical problem that needs to be overcome in large-scale distributed systems. Summary of the invention

[0003] In order to solve the problems existing in the above-mentioned prior art, the present invention aims to provide a method for intelligent exchange and integration of heterogeneous data in a digital enterprise. The method for intelligent exchange and integration of heterogeneous data in a digital enterprise effectively reduces the storage overhead of vector clocks, improves the efficiency of event sorting, and ensures the accuracy of causal relationship judgment, thereby significantly improving the overall performance of the distributed system.

[0004] The method for intelligent exchange and integration of heterogeneous data of a digital enterprise described in the present invention comprises the following steps:

[0005] S1. Obtain communication frequency data between nodes in a distributed system, determine the activity level of each node according to a preset node activity threshold, and obtain node activity level information;

[0006] S2. For nodes with different activity levels, dynamically adjust the timestamp array dimension and compression ratio of the vector clock according to the node event counter, and compress the vector clock of the low-activity node;

[0007] S3, adding a timestamp to the timestamp array of each compressed vector clock, and performing vector clock synchronization on the newly added or updated vector clock received by the receiving node to obtain an incremental update result of the vector clock;

[0008] S4. Analyze the partial order relationship between each element in the timestamp array according to the incremental update result, identify and determine the redundant elements that have no impact on the causal dependency, and dynamically trim the timestamp array of the vector clock;

[0009] S5. Based on the vector clock pruning, identify the vector clocks with high frequency access in the system, cache their timestamp arrays in the distributed memory cache system, and set the cache expiration time and elimination strategy;

[0010] S6. Monitor the event concurrency index and vector clock distribution of the distributed system in real time, dynamically adjust the intensity of vector clock compression and clipping according to the preset event concurrency threshold, and dynamically optimize the vector clock storage and event sorting in the distributed system through concurrency-aware adaptive adjustment.

[0011] Preferably, the step S1 specifically includes:

[0012] Receiving communication record information carrying a source node identifier and a target node identifier, wherein the communication record information is generated by a node in a distributed system within a preset time period;

[0013] Calculate the communication frequency between node pairs according to the communication record information to obtain a node communication frequency matrix;

[0014] A clustering algorithm is used to classify the total communication frequency of nodes, and the nodes are divided into three activity levels: high, medium, and low. The cluster center is used as the activity threshold of each level. The activity threshold includes a high activity threshold, a medium activity threshold, and a low activity threshold.

[0015] Determine the activity level of the node as follows:

[0016] Obtaining the total communication frequency of the node, and comparing the total communication frequency of the node with the activity threshold;

[0017] If the total communication frequency of the node is higher than the high activity threshold, it is determined to be a high activity node;

[0018] If the total communication frequency of the node is lower than the low activity threshold, it is determined to be a low activity node;

[0019] If the total communication frequency of the node is between the high activity threshold and the low activity threshold, it is determined to be a medium-activity node.

[0020] Preferably, the step S2 specifically includes:

[0021] Obtain the activity level information of the node and the average event frequency in the past, and determine the event counter value of each node based on the information;

[0022] Adopting an adaptive compression algorithm based on frequency analysis, calculating the initial timestamp array dimension and compression ratio of the vector clock according to the event counter value;

[0023] Monitor changes in event counters through a fixed-size sliding window mechanism. If the rate of change exceeds a preset threshold, adjust the timestamp array dimension and compression ratio of the vector clock.

[0024] For low-activity nodes, the vector clock is compressed using a compression ratio. During the compression process, the timestamps of high-frequency events are retained and the timestamps of low-frequency events are merged.

[0025] When communicating between nodes, the sender attaches the compressed vector clock to the message. The receiver decompresses the received vector clock according to the sender's activity level and compression ratio, and uses linear interpolation method to estimate the merged low-frequency event timestamps during the decompression process.

[0026] Preferably, the step S3 specifically includes:

[0027] Get the compressed vector clock timestamp array, calculate the hash value of the timestamp array through the hash algorithm, and determine the version number;

[0028] The receiving node obtains the local vector clock version number and the received vector clock version number, and compares the two version numbers using a bitwise XOR operation;

[0029] If the XOR result is not 0, it is determined whether the vector clock received by the receiving node is newly added or updated;

[0030] If the received vector clock is newly added or updated, an incremental transmission mechanism is used to synchronize the vector clocks between nodes;

[0031] According to the position of the change in the timestamp array represented by the bitmap, the changed timestamp value is transmitted;

[0032] The receiving node recovers the complete updated vector clock using the bitmap information based on the received incremental data;

[0033] Using the bit-by-bit comparison method, a newer timestamp value is selected to update the local vector clock to obtain the incremental update result of the vector clock.

[0034] Preferably, the step S4 specifically includes:

[0035] Construct a directed graph based on the elements of the timestamp array, perform topological sorting on the directed graph, and obtain a dependency graph between the elements;

[0036] The depth-first search algorithm is implemented recursively to traverse the dependency graph and calculate the influence range of the element;

[0037] If the influence range of an element is smaller than a preset threshold, it is determined to be a potential redundant element;

[0038] Obtaining the update frequency of the potential redundant elements, monitoring them through a sliding window mechanism, and calculating the average update frequency of the elements using an exponential moving average method;

[0039] If the average update frequency is lower than the dynamic pruning threshold, the element is determined to be a redundant element;

[0040] If the average update frequency is higher than or equal to the dynamic pruning threshold, the element is determined to be a non-redundant element;

[0041] Perform a removal operation on the redundant elements, use a compressed array method to move the non-redundant elements to the front of the array in sequence, and update the index mapping of the elements at the same time;

[0042] Recalculating the array dimension of the non-redundant elements to generate a pruned vector clock timestamp array;

[0043] Obtain the partial order relationship of the vector clocks before and after clipping, compare the partial order relationship of the vector clocks before clipping with the partial order relationship of the vector clocks after clipping, and verify whether the vector clocks after clipping can correctly represent the event causal relationship.

[0044] Preferably, the step S5 specifically includes:

[0045] Obtain the access frequency of the vector clock, and use a sliding window algorithm to identify the high-frequency access vector clock according to the access frequency;

[0046] Distribute the timestamp array of the frequently accessed vector clock to the distributed memory cache cluster;

[0047] During the vector clock update process, the distributed cache is updated synchronously according to the double-write consistency protocol;

[0048] When determining the sequence of events, vector clock data is obtained from the distributed cache. If the distributed cache does not hit, the vector clock data is read from the persistent storage.

[0049] Monitor the hit rate and access delay of the distributed cache, and dynamically adjust the cache strategy based on the hit rate and access delay. The adjustment process includes updating the cache expiration time and elimination strategy.

[0050] Preferably, the step S6 specifically includes:

[0051] Obtain the system event data collected by the lightweight agent in real time on each node, calculate the number of events per unit time based on the system event data, and obtain the event concurrency index;

[0052] Using the vector clock elements in the system event data, the mean, variance, maximum value and minimum value of the vector clock elements are calculated to obtain the distribution statistics of the vector clock elements;

[0053] Inputting the event concurrency index and the vector clock element distribution statistics into a pre-trained gradient boosting decision tree model;

[0054] Outputting the vector clock compression ratio and clipping threshold in the current system state through the gradient boosting decision tree model, and storing the vector clock compression ratio and clipping threshold in a distributed configuration center;

[0055] If the event concurrency index is greater than a preset event concurrency threshold, increasing the vector clock compression ratio and the clipping threshold;

[0056] If the event concurrency index is less than a preset event concurrency threshold, reducing the vector clock compression ratio and the clipping threshold;

[0057] The node obtains the vector clock compression ratio and clipping threshold from the distributed configuration center, and recompresses and clips the locally stored vector clock according to the vector clock compression ratio and clipping threshold.

[0058] In subsequent event processing, the vector clock compression ratio and clipping threshold are used to store and sort the vector clocks;

[0059] Obtain the storage space occupancy and sorting accuracy of the node, and transmit the storage space occupancy and sorting accuracy to the central monitoring node;

[0060] The central monitoring node optimizes the gradient boosting decision tree model according to storage space occupancy and sorting accuracy.

[0061] The method for intelligent exchange and integration of heterogeneous data of a digital enterprise described in the present invention has the following advantages:

[0062] A digital enterprise heterogeneous data intelligent exchange and integration method of the present invention can significantly reduce the use of storage space and computing resources by dynamically adjusting the timestamp array dimension and compression ratio of the vector clock according to the activity level of the node; the vector clock of the low-activity node is compressed, reducing unnecessary redundant data, thereby improving the overall resource utilization efficiency of the system; adding and synchronizing the compressed vector clock with the timestamp ensures that the newly added or updated vector clock can be quickly and accurately synchronized to the receiving node, thereby enhancing the real-time performance and data consistency of the system; by analyzing the partial order relationship in the timestamp array, identifying and cutting redundant elements that have no effect on the causal dependency, further reducing the storage burden of the vector clock, improving the data processing efficiency and reducing the complexity of system maintenance; The accessed vector clock timestamp array is cached in a distributed memory cache system, and the cache expiration time and elimination strategy are set, so that the system can cope with the growth of data volume more flexibly. The cache mechanism reduces the frequent access to the disk or database, and improves the response speed and scalability of the system; the event concurrency index and vector clock distribution of the distributed system are monitored in real time, and the compression and clipping strength of the vector clock are dynamically adjusted according to the preset threshold. This adaptive adjustment mechanism enables the system to automatically optimize the storage and event sorting strategies according to different workloads, and improves the stability and reliability of the system; therefore, this method effectively reduces the storage overhead of the vector clock, improves the event sorting efficiency, and ensures the accuracy of causal relationship judgment, thereby significantly improving the overall performance of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a method for intelligent exchange and integration of heterogeneous data of a digital enterprise described in the present invention. DETAILED DESCRIPTION

[0064] like Figure 1 As shown, the method for intelligent exchange and integration of heterogeneous data of a digital enterprise described in the present invention comprises the following steps:

[0065] S1. Obtain communication frequency data between nodes in a distributed system, determine the activity level of each node according to a preset node activity threshold, and obtain node activity level information;

[0066] S2. For nodes of different activity levels, dynamically adjust the timestamp array dimension and compression ratio of the vector clock according to the node event counter, and compress the vector clock of the low-activity node; specifically, dynamically adjust the timestamp array dimension and compression ratio of the vector clock according to the node event counter by adopting an adaptive clock compression method; compressing the vector clock of the low-activity node is to reduce the vector clock stored in the low-activity node;

[0067] S3, adding a timestamp to the timestamp array of each compressed vector clock, and performing vector clock synchronization on the newly added or updated vector clock received by the receiving node to obtain an incremental update result of the vector clock;

[0068] S4. Analyze the partial order relationship between each element in the timestamp array according to the incremental update result, identify and determine the redundant elements that have no effect on the causal dependency, and dynamically trim the timestamp array of the vector clock; specifically, dynamically trimming the timestamp array of the vector clock is achieved by adopting an adaptive trimming algorithm to remove redundant elements;

[0069] S5. Based on the vector clock clipping, the vector clocks with high frequency access in the system are identified, their timestamp arrays are cached in the distributed memory cache system, and the cache expiration time and elimination strategy are set; specifically, the timestamp arrays are cached in the distributed memory cache system by adopting a distributed cache method, and the cache is updated synchronously when the vector clock is updated, so as to improve the judgment performance of the event sequence;

[0070] S6. Monitor the event concurrency index and vector clock distribution of the distributed system in real time, dynamically adjust the intensity of vector clock compression and clipping according to the preset event concurrency threshold, and dynamically optimize the vector clock storage and event sorting in the distributed system through concurrency-aware adaptive adjustment.

[0071] Furthermore, in this embodiment, step S1 specifically includes:

[0072] Receiving communication record information carrying a source node identifier and a target node identifier, where the communication record information is generated by a node in the distributed system within a preset time period;

[0073] The communication frequency between the node pairs is calculated according to the communication record information to obtain a node communication frequency matrix; the node communication frequency matrix includes the communication frequency between each node pair;

[0074] A clustering algorithm is used to classify the total communication frequency of nodes, and the nodes are divided into three activity levels: high, medium, and low. The cluster center is used as the activity threshold of each level. The activity thresholds include high activity threshold, medium activity threshold, and low activity threshold.

[0075] Determine the activity level of the node as follows:

[0076] Obtain the total communication frequency of the node and compare it with the activity threshold;

[0077] If the total communication frequency of a node is higher than the high activity threshold, it is determined to be a high activity node;

[0078] If the total communication frequency of a node is lower than the low activity threshold, it is judged as a low activity node;

[0079] If the total communication frequency of a node is between the high activity threshold and the low activity threshold, it is determined to be a medium-active node; that is, if the total communication frequency of a node is between the medium activity threshold and the high activity threshold, or between the low activity threshold and the medium activity threshold, it can be determined to be a medium-active node;

[0080] Specifically, the communication record data between nodes in a preset time period is obtained from the distributed system, and a log collection component is deployed on each node to record the timestamps of sending and receiving messages, source node information, and target node information;

[0081] Calculate the communication frequency between each pair of nodes based on the timestamp information in the communication records and generate a node communication frequency matrix;

[0082] For the total communication frequency of each node in the communication frequency matrix, the K-means clustering algorithm is used to divide the nodes into three activity levels: high, medium, and low;

[0083] The total communication frequency of each node is used as the feature vector, the number of clusters is set to 3, and the K-means algorithm is run to obtain three cluster centers;

[0084] According to the distance from each node to the cluster center, it is divided into the nearest cluster to obtain the activity level of the node;

[0085] The cluster center is used as the activity threshold of each level, and the total communication frequency of each node is compared with the activity threshold;

[0086] If the total communication frequency of a node is higher than the high activity threshold, it is determined to be a high activity node;

[0087] If the total communication frequency of a node is lower than the low activity threshold, it is judged as a low activity node;

[0088] If the total communication frequency of a node is between the high activity threshold and the low activity threshold, it is determined to be a medium-active node;

[0089] According to the activity level of each node and the data in the node communication frequency matrix, the average communication frequency between nodes of different activity levels is calculated;

[0090] Generate a communication mode statistical report that reflects the node activity distribution characteristics. The communication mode statistical report includes: the number and proportion of nodes at each activity level, the average communication frequency matrix between nodes at each level, and the average number of connections between nodes at each level. These indicators can fully reflect the node activity distribution characteristics and communication mode;

[0091] Here is an example:

[0092] In a distributed system, communication record data is obtained by deploying log collection components on each node;

[0093] These components record the sending and receiving timestamps of the message, the source node and the destination node identifiers. For example, when node A sends a message to node B, the log record format is "1632547890,A,B,send", and when receiving, it is "1632547891,B,A,receive";

[0094] After collecting 30 days of communication data, the communication frequency between nodes is calculated. Assuming that the system has 100 nodes, a 100x100 communication frequency matrix is ​​generated. In the matrix, the element M[i][j] represents the number of times node i sends a message to node j.

[0095] Next, the K-means clustering algorithm is used to divide the nodes into three activity levels: high, medium, and low. The total communication frequency of each node (i.e., the sum of the elements in this row in the matrix) is used as the feature vector, and the number of clusters k is set to 3;

[0096] The K-means algorithm randomly selects three initial center points and iterates until convergence. Assume that the three cluster centers finally obtained are 5000, 2000, and 500, respectively. These values ​​are used as the thresholds of high, medium, and low activity.

[0097] According to the total communication frequency of each node, it is divided into the nearest cluster. For example, a node with a total communication frequency of 5200 will be classified as a highly active node, while a node with a total communication frequency of 1800 will be classified as a medium active node.

[0098] After the classification is completed, a communication mode statistical report is generated. The report includes the number and proportion of nodes at each activity level, for example, 20 highly active nodes, accounting for 20%, 50 medium active nodes, accounting for 50%, and 30 low active nodes, accounting for 30%, the average communication frequency matrix between nodes of each level (3x3 matrix, element [i][j] represents the average communication frequency of nodes at level i to nodes at level j), and the average number of connections of nodes at each level, for example, highly active nodes communicate with an average of 80 other nodes, medium active nodes communicate with an average of 40 other nodes, and low active nodes communicate with an average of 10 other nodes. These indicators comprehensively reflect the node activity distribution characteristics and communication patterns, and provide data support for system optimization and load balancing.

[0099] Furthermore, in this embodiment, step S2 specifically includes:

[0100] Obtain the activity level information of the node and the average event frequency in the past, and determine the event counter value of each node based on the information;

[0101] Adopting an adaptive compression algorithm based on frequency analysis, the initial timestamp array dimension and compression ratio of the vector clock are calculated according to the event counter value;

[0102] Monitor changes in event counters through a fixed-size sliding window mechanism. If the rate of change exceeds a preset threshold, adjust the timestamp array dimension and compression ratio of the vector clock.

[0103] For low-activity nodes, the vector clock is compressed using a compression ratio. During the compression process, the timestamps of high-frequency events are retained and the timestamps of low-frequency events are merged.

[0104] When communicating between nodes, the sender attaches the compressed vector clock to the message. The receiver decompresses the received vector clock according to the sender's activity level and compression ratio, and uses linear interpolation to estimate the merged low-frequency event timestamps during the decompression process.

[0105] Specifically, according to the node activity level information and the average event frequency in the past 24 hours, the event counter value of each node is obtained, and the initial timestamp array dimension and compression ratio of each node vector clock are calculated through an adaptive compression algorithm based on frequency analysis;

[0106] A high-activity node sets a larger initial dimension and a smaller compression ratio, while a low-activity node sets the opposite. A sliding window mechanism with a fixed size of 1 hour is used to calculate the rate of change of the number of events in the window every 10 minutes and monitor the changes in the node event counter;

[0107] If the rate of change exceeds the preset threshold, the timestamp array dimension and compression ratio of the node vector clock are dynamically adjusted;

[0108] For low-activity nodes, their vector clocks are compressed using a compression ratio. A frequency-based compression method is used during the compression process to retain the timestamps of high-frequency events and merge the timestamps of low-frequency events, thereby reducing the number of stored timestamps.

[0109] When communicating between nodes, the sender attaches the compressed vector clock to the message, and the receiver decompresses the received vector clock based on the sender's activity level and compression ratio;

[0110] During the decompression process, the merged timestamp information is restored according to the compression ratio. For the merged low-frequency event timestamps, the linear interpolation method is used to estimate the original timestamps and restore the complete timestamp information.

[0111] Here is an example:

[0112] In a distributed system, suppose there are 100 nodes, of which 20 are highly active nodes, 50 are moderately active nodes, and 30 are low-activity nodes;

[0113] The adaptive compression algorithm based on frequency analysis first calculates the average event frequency of each node in the past 24 hours. For example, a high-activity node generates 1,000 events per hour on average, a medium-activity node generates 500 events, and a low-activity node generates 100 events per hour on average;

[0114] Based on these data, the algorithm sets the initial timestamp array dimension of 100 for high-activity nodes and a compression ratio of 0.9, the dimension of 50 for medium-activity nodes and a compression ratio of 0.5, and the dimension of 20 for low-activity nodes and a compression ratio of 0.2.

[0115] The system uses a sliding window of 1 hour and calculates the rate of change of the number of events in the window every 10 minutes;

[0116] If the number of events of a low-activity node increases from 100 to 200 in the last hour, the change rate is 100%, which exceeds the preset threshold of 80%, the system adjusts the dimension of its timestamp array from 20 to 30, and increases the compression ratio from 0.2 to 0.3; for vector clock compression of low-activity nodes, assuming that the original timestamp array is [1,2,3,4,5,6,7,8,9,10] and the compression ratio is 0.5, then the high-frequency event timestamps [1,3,5,7,9] are retained after compression;

[0117] When nodes communicate, the sender appends the compressed vector clock [1,3,5,7,9] to the message. The receiver learns that the sender is a low-activity node and the compression ratio is 0.5. It then decompresses the message. During the decompression process, the missing timestamps 2, 4, 6, 8, and 10 are estimated using linear interpolation. For example, the estimated value of timestamp 2 is (1+3) / 2=2. Finally, the complete timestamp array [1,2,3,4,5,6,7,8,9,10] is restored.

[0118] Furthermore, in this embodiment, step S3 specifically includes:

[0119] Get the compressed vector clock timestamp array, calculate the hash value of the timestamp array through the hash algorithm, and determine the version number;

[0120] The receiving node obtains the local vector clock version number and the received vector clock version number, and compares the two version numbers using a bitwise XOR operation;

[0121] If the XOR result is not 0, it is determined whether the vector clock received by the receiving node is newly added or updated;

[0122] If the received vector clock is newly added or updated, an incremental transmission mechanism is used to synchronize the vector clocks between nodes;

[0123] According to the position of the change in the timestamp array represented by the bitmap, the changed timestamp value is transmitted;

[0124] The receiving node recovers the complete updated vector clock using the bitmap information based on the received incremental data;

[0125] Using the bit-by-bit comparison method, a newer timestamp value is selected to update the local vector clock to obtain the incremental update result of the vector clock;

[0126] Specifically, after the vector clock is compressed, the hash value of the timestamp array is calculated using the SHA-256 hash algorithm, the first 8 bytes are taken as the version number, and the version number is added to the compressed vector clock to obtain a compressed vector clock with a version number;

[0127] The receiving node obtains the version numbers of the local vector clock and the received vector clock, and compares the difference between the two version numbers using a bitwise XOR operation. If the XOR result is not 0, it determines that the received vector clock is newly added or updated, and outputs the judgment result;

[0128] If the result of the judgment is new or updated, the incremental transmission mechanism is adopted, and the changed position in the timestamp array is represented by a bitmap, and only the changed timestamp value is transmitted;

[0129] The receiving node uses the bitmap information to restore the complete updated vector clock based on the received incremental data, and uses the bit-by-bit comparison method to select the newer timestamp value to update the local vector clock to obtain the incremental update result of the vector clock.

[0130] To handle data loss or disorder problems that may occur during network transmission, a sequence number and a check number are added during incremental transmission. The receiver verifies data integrity through checksums and reorders incremental data based on the sequence number to ensure the reliability of incremental updates.

[0131] Here is an example:

[0132] In a distributed system, assuming that the compressed vector clock of node A is [1,3,5,7,9], the SHA-256 algorithm is used to calculate the hash value, and a 64-bit hexadecimal string "a1b2c3d4e5f6g7h8..." is obtained. The first 8 bytes "a1b2c3d4" are taken as the version number.

[0133] Node A combines the version number with the compressed vector clock to form a compressed vector clock with a version number "a1b2c3d4:[1,3,5,7,9]";

[0134] After receiving this information, node B performs a bitwise XOR operation on its version number "a1b2c3d4" and the local version number "b2c3d4e5". If the result is not 0, it is judged as a new addition or update.

[0135] Node B creates a 32-bit bitmap "10101", indicating that the 1st, 3rd, and 5th timestamps have changed, and only transmits the changed timestamp values ​​[1,5,9];

[0136] Node A adds sequence number 1 and 32-bit CRC checksum 0xA5B6C7D8 to the incremental data during transmission;

[0137] After receiving the incremental data, Node B verifies the data integrity through CRC checksum, and uses the bitmap "10101" to insert the incremental data [1,5,9] into the corresponding position of the local vector clock [2,4,6,8,10] to obtain the updated vector clock [1,4,5,8,9];

[0138] Finally, node B compares the updated vector clock with the local vector clock bit by bit, selects the newer timestamp value, and finally obtains the incremental update result [1,4,5,8,9].

[0139] Furthermore, in this embodiment, step S4 specifically includes:

[0140] Construct a directed graph based on the elements of the timestamp array, perform topological sorting on the directed graph, and obtain a dependency graph between the elements;

[0141] The depth-first search algorithm is implemented recursively to traverse the dependency graph and calculate the influence range of the element;

[0142] If the influence range of an element is smaller than a preset threshold, it is determined to be a potential redundant element;

[0143] Obtain the update frequency of potential redundant elements, monitor them through a sliding window mechanism, and use the exponential moving average method to calculate the average update frequency of the elements;

[0144] If the average update frequency is lower than the dynamic pruning threshold, the element is determined to be a redundant element;

[0145] If the average update frequency is higher than or equal to the dynamic pruning threshold, the element is determined to be a non-redundant element;

[0146] Remove redundant elements, use the compressed array method to move non-redundant elements to the front of the array, and update the index mapping of the elements;

[0147] Recalculate the array dimensions for non-redundant elements to generate a pruned vector clock timestamp array;

[0148] Obtain the partial order relationship of the vector clocks before and after clipping, compare the partial order relationship of the vector clocks before clipping with the partial order relationship of the vector clocks after clipping, and verify whether the vector clocks after clipping can correctly represent the causal relationship of events;

[0149] Specifically, according to the incremental update result of the vector clock, a directed graph between the elements of the timestamp array is constructed, and topological sorting is performed using the Kahn algorithm. An in-degree table and an adjacency table are constructed to represent the dependency relationship between the elements, and a dependency graph between the elements is obtained.

[0150] A recursive method is used to implement the depth-first search algorithm to traverse the dependency graph, maintain the access mark array to avoid repeated access, and calculate the influence range of each element, that is, the number of affected nodes. If the influence range of the element is less than the preset threshold, it is marked as a potential redundant element and output;

[0151] A sliding window mechanism with a size of 1 hour and a step length of 10 minutes is used to monitor the update frequency of potential redundant elements. The update frequency of elements is calculated by the exponential moving average method, and the pruning threshold is dynamically adjusted. If the update frequency of an element is lower than the threshold, it is confirmed as a redundant element.

[0152] Remove the confirmed redundant elements, use the compressed array method to move the non-redundant elements to the front of the array in sequence, update the corresponding index mapping, recalculate the array dimension, and generate the pruned vector clock timestamp array;

[0153] By comparing the partial order relationship of the vector clocks before and after clipping, it is verified whether the clipped vector clock can still correctly represent the causal relationship of the event, thus ensuring the correctness of the clipping operation;

[0154] Here is an example:

[0155] In a distributed system, suppose the vector clock increment update result of a node is [5,3,7,2,6];

[0156] First, a directed graph between the elements of the timestamp array is constructed, and the Kahn algorithm is used for topological sorting to construct the in-degree table [0,1,0,2,1] and the adjacency list {0:[1,3],2:[3,4],4:[1]} to obtain the dependency graph;

[0157] Then, we use a recursive method to implement depth-first search and maintain the access mark array [0,0,0,0,0]. We start searching from the unvisited nodes and calculate the impact range: 0 affects 3 nodes, 1 affects 1 node, 2 affects 3 nodes, 3 affects 1 node, and 4 affects 2 nodes.

[0158] The preset threshold is 2, marking 1 and 3 as potential redundant elements, and using a sliding window of 1 hour and 10 minutes step length to monitor the update frequency of these two elements;

[0159] The update frequency is calculated by the exponential moving average method, α=0.2, update frequency=α×current value+(1-α)×historical average value, assuming that the update frequencies of 1 and 3 are 0.05 times / minute and 0.02 times / minute respectively, and the dynamic clipping threshold is 0.03 times / minute, confirming that 3 is a redundant element;

[0160] Remove element 3, compress the array to [5,3,7,6], and update the index mapping {0:0,1:1,2:2,4:3};

[0161] Finally, we compare the partial order relations before and after pruning to verify whether [5, 3, 7, 6] still correctly represents the event causal relationship and ensure the correctness of the pruning operation.

[0162] Furthermore, in this embodiment, step S5 specifically includes:

[0163] Obtain the access frequency of the vector clock, and use a sliding window algorithm to identify the high-frequency access vector clock according to the access frequency;

[0164] Distribute the timestamp array of the frequently accessed vector clock to the distributed memory cache cluster;

[0165] During the vector clock update process, the distributed cache is updated synchronously according to the double-write consistency protocol;

[0166] When determining the sequence of events, vector clock data is obtained from the distributed cache. If the distributed cache does not hit, the vector clock data is read from the persistent storage.

[0167] Monitor the hit rate and access delay of the distributed cache, and dynamically adjust the cache strategy based on the hit rate and access delay. The adjustment process includes updating the cache expiration time and elimination strategy.

[0168] Specifically, the access frequency of each vector clock is calculated through a sliding window statistical algorithm, and the number of accesses is counted using a 1-hour time window and a 5-minute sliding step. The high-frequency access vector clocks in the system are identified according to the preset access frequency threshold, and a high-frequency vector clock list is generated.

[0169] The consistent hashing algorithm is used to distribute and store the timestamp array of the high-frequency vector clock into the distributed memory cache cluster. The virtual node technology is used to improve the data distribution balance. The addition and deletion of cache nodes are processed through the rehash operation. The cache expiration time and the elimination strategy based on the least recently used principle are set.

[0170] Use the double-write consistency protocol to synchronously update the distributed cache when the vector clock is updated. Adopt the strategy of updating the cache first and then the database. Use distributed locks to ensure the consistency of concurrent updates. Use incrementing integers as version numbers to solve the concurrent update problem. Ensure the consistency of cache data and persistent storage. Send notifications to subscribers after the update is completed.

[0171] When judging the order of events, vector clock data is obtained from the distributed cache first. If the cache does not hit, it is read from the persistent storage. At the same time, the cache preheating mechanism is triggered, and the asynchronous thread pool is used to process the cache preheating task, and the read data is asynchronously written to the cache.

[0172] Monitor cache hit rate and access latency in real time, and optimize system performance by dynamically adjusting cache strategies;

[0173] Here is an example:

[0174] In a distributed system, suppose there are 1,000 vector clocks. The access frequency is calculated using a sliding window statistical algorithm. A 1-hour time window is used and statistics are collected every 5 minutes to record the number of times each vector clock is accessed within the window. For example, vector clock A has been accessed 500 times in the last hour, and its access frequency is 8.33 times / minute.

[0175] The access frequency threshold is set to 5 times / minute, and 100 high-frequency access vector clocks are identified;

[0176] The consistent hashing algorithm is used to distribute these 100 high-frequency vector clocks to 10 cache nodes. Each physical node corresponds to 100 virtual nodes, and a total of 1000 virtual nodes are evenly distributed on the hash ring. For example, the hash value of vector clock A is 0x3F2E, which is assigned to physical node 3 corresponding to virtual node 0x3F00.

[0177] Set the cache expiration time to 30 minutes and use the LRU strategy to eliminate the least recently used data;

[0178] When the vector clock is updated, for example, A is updated from [1,2,3,4,5] to [1,2,3,4,6], the distributed lock "lock_A" is first acquired, then the data in cache node 3 is updated, the version number is incremented from 100 to 101, the database is updated, and finally the lock is released and a notification is sent;

[0179] When determining the event sequence, cache node 3 is queried first. If a hit is not found, the data is read from the database and an asynchronous thread is started to write the data into the cache.

[0180] Through monitoring, we found that the cache hit rate was 95% and the average access delay was 5ms. Based on these indicators, we dynamically adjusted the cache capacity and expiration time to continuously optimize system performance.

[0181] Furthermore, in this embodiment, step S6 specifically includes:

[0182] Obtain the system event data collected by the lightweight agent in real time on each node, calculate the number of events per unit time based on the system event data, and obtain the event concurrency index;

[0183] Using the vector clock elements in the system event data, the mean, variance, maximum value and minimum value of the vector clock elements are calculated to obtain the distribution statistics of the vector clock elements;

[0184] Input event concurrency metrics and vector clock element distribution statistics into a pre-trained gradient boosting decision tree model;

[0185] Output the vector clock compression ratio and clipping threshold in the current system state through the gradient boosting decision tree model, and store the vector clock compression ratio and clipping threshold in the distributed configuration center;

[0186] If the event concurrency index is greater than the preset event concurrency threshold, the vector clock compression ratio and clipping threshold are increased; that is, the compression and clipping strength are increased in high concurrency to reduce the storage space occupied by the vector clock;

[0187] If the event concurrency index is less than the preset event concurrency threshold, the vector clock compression ratio and clipping threshold are reduced; that is, the compression strength is appropriately relaxed at low concurrency to improve the accuracy of event sequence judgment;

[0188] The node obtains the vector clock compression ratio and clipping threshold from the distributed configuration center, and recompresses and clips the locally stored vector clock according to the vector clock compression ratio and clipping threshold.

[0189] In subsequent event processing, the vector clock compression ratio and clipping threshold are used to store and sort the vector clocks;

[0190] Obtain the storage space occupancy and sorting accuracy of the node, and transmit the storage space occupancy and sorting accuracy to the central monitoring node;

[0191] The central monitoring node optimizes the gradient boosting decision tree model based on storage space occupancy and sorting accuracy;

[0192] Specifically, by deploying a lightweight agent on each node to collect event data in the system in real time, the number of events per unit time is calculated to obtain the event concurrency index, and the distribution of elements of the vector clock is statistically analyzed, including mean, variance, maximum, minimum and other statistical quantities. These monitoring data are transmitted through the message queue and aggregated to the central monitoring node;

[0193] The monitoring data is input into a pre-trained gradient boosting decision tree model. The model uses features such as event concurrency and vector clock element distribution statistics in historical data to learn the relationship between concurrency and the optimal compression and pruning strategy. It outputs the optimal vector clock compression ratio and pruning threshold under the current system state and stores the results in the distributed configuration center.

[0194] According to the preset event concurrency threshold and the compression and trimming parameters output by the model, the compression algorithm parameters and trimming algorithm parameters of the vector clock are dynamically adjusted. The compression algorithm adopts a frequency-based compression method to adjust the threshold for retaining high-frequency elements. The trimming algorithm adjusts the judgment criteria for removing redundant elements. The compression ratio and trimming threshold are increased when concurrency is high, and reduced when concurrency is low.

[0195] Each node periodically extracts the latest compression and pruning parameters from the distributed configuration center, recompresses and prunes the locally stored vector clocks, and applies the new parameters to store and sort the vector clocks in subsequent event processing.

[0196] Collect adjusted system performance indicators, including storage space usage and sorting accuracy, and feed them back to the central monitoring node for continuous optimization of prediction models and adjustment strategies;

[0197] Here is an example:

[0198] In a distributed system with 100 nodes, a lightweight agent is deployed on each node to collect event data every second. Assume that the system event concurrency is 1000 events / second at a certain moment, the vector clock element distribution has a mean of 50, a variance of 10, a maximum of 100, and a minimum of 10;

[0199] These data are transmitted to the central monitoring node through the RabbitMQ message queue. The pre-trained XGBoost model (an implementation of the gradient boosting decision tree) receives these features and outputs the optimal compression ratio of 0.8 and the pruning threshold of 20. These parameters are stored in the ZooKeeper distributed configuration center;

[0200] The system detects that the current concurrency exceeds the preset threshold of 800 events / second, triggering parameter adjustment: the compression algorithm will retain elements with a frequency greater than 50 times / minute, and the pruning algorithm will remove redundant elements with an impact range of less than 5 nodes;

[0201] 100 nodes pull new parameters from ZooKeeper every 10 seconds and reprocess the local vector clock. For example, the original vector clock [10, 20, 30, 40, 50, 60, 70, 80, 90, 100] becomes [50, 60, 70, 80, 90, 100] after compression and pruning.

[0202] System monitoring shows that storage space usage has been reduced by 40% and sorting accuracy is 98%. These performance indicators are collected through the Prometheus monitoring system and stored in the InfluxDB time series database for subsequent model optimization, forming a closed-loop feedback mechanism.

[0203] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "lateral, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention.

[0204] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A method for intelligent exchange and integration of heterogeneous data in a digital enterprise, characterized in that: The following steps are involved: S1. Obtain communication frequency data between nodes in a distributed system, determine the activity level of each node according to a preset node activity threshold, and obtain node activity level information; S2. For nodes with different activity levels, dynamically adjust the timestamp array dimension and compression ratio of the vector clock according to the node event counter, and compress the vector clock of the low-activity node; S3, adding a timestamp to the timestamp array of each compressed vector clock, and performing vector clock synchronization on the newly added or updated vector clock received by the receiving node to obtain an incremental update result of the vector clock; S4. Analyze the partial order relationship between each element in the timestamp array according to the incremental update result, identify and determine the redundant elements that have no impact on the causal dependency, and dynamically trim the timestamp array of the vector clock; S5. Based on the vector clock pruning, identify the vector clocks with high frequency access in the system, cache their timestamp arrays in the distributed memory cache system, and set the cache expiration time and elimination strategy; S6. Real-time monitoring of event concurrency indicators and vector clock distribution of distributed systems, dynamic adjustment of vector clock compression and clipping strength according to preset event concurrency thresholds, and dynamic optimization of vector clock storage and event sorting in distributed systems through concurrency-aware adaptive adjustment; The step S6 specifically includes: Obtain the system event data collected by the lightweight agent in real time on each node, calculate the number of events per unit time based on the system event data, and obtain the event concurrency index; Using the vector clock elements in the system event data, the mean, variance, maximum value, and minimum value of the vector clock elements are calculated to obtain the distribution statistics of the vector clock elements; Inputting the event concurrency index and the vector clock element distribution statistics into a pre-trained gradient boosting decision tree model; Outputting the vector clock compression ratio and clipping threshold in the current system state through the gradient boosting decision tree model, and storing the vector clock compression ratio and clipping threshold in a distributed configuration center; If the event concurrency index is greater than a preset event concurrency threshold, increasing the vector clock compression ratio and the clipping threshold; If the event concurrency index is less than a preset event concurrency threshold, reducing the vector clock compression ratio and the clipping threshold; The node obtains the vector clock compression ratio and clipping threshold from the distributed configuration center, and recompresses and clips the locally stored vector clock according to the vector clock compression ratio and clipping threshold. In subsequent event processing, the vector clock compression ratio and clipping threshold are used to store and sort the vector clocks; Obtain the storage space occupancy and sorting accuracy of the node, and transmit the storage space occupancy and sorting accuracy to the central monitoring node; The central monitoring node optimizes the gradient boosting decision tree model according to storage space occupancy and sorting accuracy.

2. According to claim 1, the method for intelligent exchange and integration of heterogeneous data of digital enterprises is characterized by: The step S1 specifically includes: Receiving communication record information carrying a source node identifier and a target node identifier, wherein the communication record information is generated by a node in a distributed system within a preset time period; Calculate the communication frequency between node pairs according to the communication record information to obtain a node communication frequency matrix; A clustering algorithm is used to classify the total communication frequency of nodes, and the nodes are divided into three activity levels: high, medium, and low. The cluster center is used as the activity threshold of each level. The activity threshold includes a high activity threshold, a medium activity threshold, and a low activity threshold. Determine the activity level of the node as follows: Obtaining the total communication frequency of the node, and comparing the total communication frequency of the node with the activity threshold; If the total communication frequency of the node is higher than the high activity threshold, it is determined to be a high activity node; if the total communication frequency of the node is lower than the low activity threshold, it is determined to be a low activity node; If the total communication frequency of the node is between the high activity threshold and the low activity threshold, it is determined to be a medium-activity node.

3. According to claim 1, the method for intelligent exchange and integration of heterogeneous data of digital enterprises is characterized by: The step S2 specifically includes: Obtain the activity level information of the node and the average event frequency in the past, and determine the event counter value of each node based on the information; Adopting an adaptive compression algorithm based on frequency analysis, calculating the initial timestamp array dimension and compression ratio of the vector clock according to the event counter value; Monitor the changes of event counters through a fixed-size sliding window mechanism. If the change rate exceeds the preset threshold, adjust the timestamp array dimension and compression ratio of the vector clock. For low-activity nodes, the vector clock is compressed using a compression ratio. During the compression process, the timestamps of high-frequency events are retained and the timestamps of low-frequency events are merged. When communicating between nodes, the sender attaches the compressed vector clock to the message. The receiver decompresses the received vector clock according to the sender's activity level and compression ratio, and uses linear interpolation method to estimate the merged low-frequency event timestamps during the decompression process.

4. According to claim 1, the method for intelligent exchange and integration of heterogeneous data of digital enterprises is characterized by: The step S3 specifically includes: Get the compressed vector clock timestamp array, calculate the hash value of the timestamp array through the hash algorithm, and determine the version number; The receiving node obtains the local vector clock version number and the received vector clock version number, and compares the two version numbers using a bitwise XOR operation; If the XOR result is not 0, it is determined whether the vector clock received by the receiving node is newly added or updated; If the received vector clock is newly added or updated, an incremental transmission mechanism is used to synchronize the vector clocks between nodes; According to the position of the change in the timestamp array represented by the bitmap, the changed timestamp value is transmitted; The receiving node recovers the complete updated vector clock using the bitmap information based on the received incremental data; Using the bit-by-bit comparison method, a newer timestamp value is selected to update the local vector clock to obtain the incremental update result of the vector clock.

5. According to claim 1, the method for intelligent exchange and integration of heterogeneous data of digital enterprises is characterized by: The step S4 specifically includes: Construct a directed graph based on the elements of the timestamp array, perform topological sorting on the directed graph, and obtain a dependency graph between the elements; The depth-first search algorithm is implemented recursively to traverse the dependency graph and calculate the influence range of the element; If the influence range of an element is smaller than a preset threshold, it is determined to be a potential redundant element; Obtaining the update frequency of the potential redundant elements, monitoring them through a sliding window mechanism, and calculating the average update frequency of the elements using an exponential moving average method; If the average update frequency is lower than the dynamic pruning threshold, the element is determined to be a redundant element; If the average update frequency is higher than or equal to the dynamic pruning threshold, the element is determined to be a non-redundant element; Perform a removal operation on the redundant elements, use a compressed array method to move the non-redundant elements to the front of the array in sequence, and update the index mapping of the elements at the same time; Recalculating the array dimension of the non-redundant elements to generate a pruned vector clock timestamp array; Obtain the partial order relationship of the vector clocks before and after clipping, compare the partial order relationship of the vector clocks before clipping with the partial order relationship of the vector clocks after clipping, and verify whether the vector clocks after clipping can correctly represent the event causal relationship.

6. According to claim 1, the method for intelligent exchange and integration of heterogeneous data of digital enterprises is characterized by: The step S5 specifically includes: Obtain the access frequency of the vector clock, and use a sliding window algorithm to identify the high-frequency access vector clock according to the access frequency; Distribute the timestamp array of the frequently accessed vector clock to the distributed memory cache cluster; During the vector clock update process, the distributed cache is updated synchronously according to the double-write consistency protocol; When determining the sequence of events, vector clock data is obtained from the distributed cache. If the distributed cache does not hit, vector clock data is read from persistent storage. Monitor the hit rate and access delay of the distributed cache, and dynamically adjust the cache strategy based on the hit rate and access delay. The adjustment process includes updating the cache expiration time and elimination strategy.

Images