Financial industry distributed system data consistency automatic management platform and method

Abstract

The invention discloses a financial industry distributed system data consistency automatic management platform and method, and the platform is provided with a heterogeneous data access adaptation unit which can achieve the standardized packaging of heterogeneous data according to a metadata dynamic adaptation protocol; the distributed data storage unit adopts a partition fragmentation and composite index strategy to store data; and the abnormal data feature extraction unit is combined with the optimized depth map neural network to mine abnormal features, and is matched with the heterogeneous data association relationship modeling unit to construct a data association model. A consistency conflict detection unit detects conflicts according to the data consistency, a conflict resolution unit generated by a dynamic strategy generates a solution according to a service scene, and a data consistency state monitoring and feedback unit realizes closed-loop control. According to the platform and the method, through the depth map neural network and the anomaly screening model, the data consistency management problem of the financial distributed system is effectively solved, and the management efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of financial industry data management, and in particular to a platform and method for automated management of data consistency in distributed systems in the financial industry. Background Art

[0002] As the financial industry accelerates its digital transformation and distributed systems become increasingly used in financial business processing, the importance of data consistency management is becoming increasingly prominent. Financial business involves massive amounts of multi-dimensional, heterogeneous data, such as transaction records, customer information, and market data, all stored across diverse systems and nodes. Traditional centralized data management models are no longer able to adapt to the high-concurrency, low-latency business requirements of distributed architectures. Building an automated data consistency management platform has become a pressing issue for the industry.

[0003] Existing technologies have many shortcomings when dealing with data consistency management in distributed systems in the financial industry. On the one hand, there are deficiencies in heterogeneous data processing capabilities. Traditional methods are unable to efficiently process the complex and diverse heterogeneous data in the financial field and are unable to accurately adapt to the formats and protocols of different data sources, resulting in low data access efficiency and poor standardization, affecting the basic quality of data consistency management. On the other hand, there is a lack of intelligent means for data anomaly detection and conflict resolution. Existing technologies mostly rely on preset rules to judge data anomalies, making it difficult to explore potential correlation features between data. Faced with complex and changeable financial data anomalies, conflicts cannot be detected in a timely and accurate manner. In addition, the generation of conflict resolution strategies lacks dynamism and pertinence, making it difficult to meet the timeliness and accuracy requirements of financial services for data consistency management. Summary of the Invention

[0004] In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides a data consistency automation management platform and method for distributed systems in the financial industry.

[0005] The technical solution adopted by the present invention is an automated data consistency management platform for distributed systems in the financial industry, comprising:

[0006] Heterogeneous data access adapter unit, which can communicate with various heterogeneous data sources in the financial industry's distributed system through a multi-protocol conversion interface. It dynamically adjusts the data receiving format and transmission protocol based on the metadata characteristics of the data source, and standardizes the packaging of financial multi-dimensional heterogeneous data with different structures and storage methods;

[0007] Distributed data storage unit: This unit adopts a partitioned and sharded storage strategy, combining the business attributes and time series characteristics of financial data to store standardized and encapsulated data in a distributed storage cluster. By combining hash indexes with range indexes, it constructs a data storage index structure for fast data storage and retrieval.

[0008] An abnormal data feature extraction unit, which transforms the multi-dimensional heterogeneous financial data stored in distributed storage units into graph-structured data. Using an optimized deep graph neural network, it performs multi-layer convolution operations on the graph-structured data to extract abnormal feature vectors from the data. Through a node feature propagation and aggregation mechanism, it mines potential correlation features between data.

[0009] Heterogeneous data association modeling unit: This unit builds an association model between data based on the extracted abnormal feature vectors, combined with the business rules of financial data and historical data. It uses a directed acyclic graph to represent the dependencies and influence paths between data entities, and quantifies the association weights between data nodes.

[0010] The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold;

[0011] A conflict resolution unit for dynamic policy generation. This unit dynamically generates conflict resolution solutions based on detected data consistency conflicts, financial business scenarios, and data priorities, using a preset conflict resolution policy library. These solutions include data update policies, data rollback policies, and data synchronization policies.

[0012] The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects the status information and conflict resolution results of each data node, and feeds back the monitoring data to the heterogeneous data access adaptation unit, distributed data storage unit, abnormal data feature extraction unit, heterogeneous data association relationship modeling unit, consistency conflict detection unit, and dynamic strategy generation conflict resolution unit to perform closed-loop control of data consistency management.

[0013] Furthermore, the abnormal data feature extraction unit uses an optimized deep graph neural network to extract abnormal features, and its model formula is:

[0014]

[0015] Among them, H (l) Represents the node feature matrix of the lth layer. The nodes correspond to financial data entities, and the features include the numerical attributes and timestamp parameters of the financial data. To add the self-connected adjacency matrix, A represents the relationship between financial data entities, I N is the identity matrix; for The diagonal node degree matrix of W (l)is the trainable weight matrix of the lth layer; σ is the activation function; through multi-layer iterative calculation, the output abnormal feature vector H (L) , L is the total number of network layers.

[0016] Furthermore, the heterogeneous data association relationship modeling unit constructs a data association relationship model using the following formula:

[0017] E ij =α·Sim(x i , x j )+β·Weight(x i , x j )

[0018] Among them, E ij Represents financial data node x i with x j The association weight between i , x j ) is the node feature similarity calculated based on cosine similarity, and the features include financial transaction amount, transaction time, and customer credit rating parameters; Weight(x i , x j ) is the node association weight preset according to financial business rules; α and β are weight adjustment coefficients, which are determined through historical data training and are used to balance the impact of feature similarity and preset weights on the association weight.

[0019] Furthermore, the consistency conflict detection unit uses a financial multi-dimensional heterogeneous data anomaly screening model to perform conflict detection, and its formula is:

[0020]

[0021] Among them, C k represents the conflict detection index of the kth data node; f i (x k ) is the data node x k The i-th financial data feature value, including account balance and transaction frequency; is the mean value of the feature within the normal data range; is the standard deviation of the feature; ω i is the weight coefficient of each feature, which is determined according to the importance of financial business; when C k When the conflict threshold τ is exceeded, it is determined that the data node has a consistency conflict.

[0022] Furthermore, the conflict resolution unit of the dynamic policy generation takes into account the timeliness and business priority of financial data when generating the data update policy. The formula is:

[0023]

[0024] Among them, P u Indicates data update priority; T cur is the current time, T last is the last update time of the data; ΔT is the data update cycle threshold; γ is the time decay coefficient; Priority(x) is the business priority of data x, which is determined according to the financial business type and the amount involved. The update priority of each conflicting data is calculated by this formula to determine the data update order.

[0025] Furthermore, the abnormal data feature extraction unit further improves the optimized deep graph neural network and introduces the attention mechanism, the formula of which is:

[0026]

[0027] Among them, Att i is the attention weight of the i-th node, which is used to measure the importance of the node in abnormal feature extraction; W a is the weight matrix of the attention mechanism; Represents the feature vector of the i-th node in the l-th layer.

[0028] Furthermore, the heterogeneous data association relationship modeling unit considers the dynamic change characteristics of financial data when constructing the association relationship model and adopts a time sliding window mechanism. The formula is:

[0029]

[0030] in, Represents the financial data node x at time t i with x j The association weight between and They are node x at time s i with x j The characteristic vector of ; m is the sliding window size; through the time sliding window mechanism, the association weights between data nodes are dynamically updated to adapt to the dynamic changes of financial data.

[0031] Furthermore, the consistency conflict detection unit introduces the fluctuation range of historical data when calculating the conflict detection index, and the formula is:

[0032]

[0033] Among them, C' k represents the conflict detection index of the kth data node after improvement; Fluct(x k ) is based on the data node x kThe volatility factor calculated based on the historical data fluctuations reflects the degree of fluctuation of the data characteristics.

[0034] Furthermore, the conflict resolution unit of the dynamic policy generation considers the relationship between network bandwidth and data volume when generating the data synchronization policy. The formula is:

[0035]

[0036] Among them, S sync represents the data synchronization priority; Q is the amount of data to be synchronized; B is the current network bandwidth; λ is the bandwidth impact coefficient; Urgency(x) is the urgency of data x, which is determined according to financial business needs.

[0037] The method for automatically managing data consistency in a distributed system in the financial industry includes the following steps:

[0038] Step S1: Establish communication connections with various heterogeneous data sources in the financial industry distributed system through the heterogeneous data access adapter unit, dynamically adjust the data receiving format and transmission protocol based on the metadata characteristics of the data source, and perform standardized packaging of financial multi-dimensional heterogeneous data;

[0039] Step S2: The standardized and encapsulated data is transferred to the distributed data storage unit. According to the partition and sharding storage strategy, the data is stored in the distributed storage cluster in combination with the business attributes and time series characteristics of the financial data, and a data storage index structure combining hash index and range index is constructed.

[0040] Step S3: The abnormal data feature extraction unit constructs the stored data into graph structure data, uses an optimized deep graph neural network to perform multi-layer convolution operations on the graph structure data, and extracts abnormal feature vectors in the data through a node feature propagation and aggregation mechanism;

[0041] Step S4: The heterogeneous data association relationship modeling unit constructs an association relationship model between data based on the extracted abnormal feature vectors, combined with financial data business rules and historical data, and uses a directed acyclic graph to represent the dependency and influence path between data entities, and quantifies the association weights between each data node;

[0042] Step S5: The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold;

[0043] Step S6: The conflict resolution unit for dynamic policy generation dynamically generates conflict resolution solutions from a preset conflict resolution policy library based on the detected data consistency conflicts, combined with financial business scenarios and data priorities, including data update policies, data rollback policies, and data synchronization policies.

[0044] Step S7: The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects status information and conflict resolution results of each data node, and feeds back the monitoring data to each related unit to perform closed-loop control of data consistency management.

[0045] Beneficial effects: The present invention proposes an automated data consistency management platform and method for distributed systems in the financial industry. The platform is equipped with a heterogeneous data access adaptation unit. With the help of a multi-protocol conversion interface and a dynamic adaptation mechanism for metadata features, it can quickly access various types of financial heterogeneous data sources, standardize and encapsulate data with different structures and storage methods, greatly improve data access efficiency and standardization, and lay a solid foundation for data consistency management. In terms of addressing the lack of intelligence in data anomaly detection and conflict resolution, the platform has built a complete intelligent processing chain with the help of an optimized deep graph neural network and a financial multi-dimensional heterogeneous data anomaly screening model. The abnormal data feature extraction unit constructs financial data into a graph structure, and through multi-layer convolution operations and node feature propagation, it mines potential correlation features between data and accurately extracts abnormal feature vectors; the heterogeneous data association relationship modeling unit combines business rules and historical data to quantify the association weights of data nodes; the consistency conflict detection unit accurately identifies data consistency conflicts by calculating feature similarity and weight deviation; the conflict resolution unit generated by dynamic strategies dynamically generates diversified conflict resolution solutions based on business scenarios and data priorities. In addition, the data consistency status monitoring and feedback unit realizes a closed management loop, ensuring efficient and accurate data consistency management, and significantly improving the data management level and business stability of the financial industry's distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a composition diagram of the platform unit of the present invention;

[0047] Figure 2 The figure is a flow chart of the method steps of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that, unless there is a conflict, the embodiments in this application and the features described in the embodiments can be combined with each other. The application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, the financial industry distributed system data consistency automation management platform and method include:

[0050] Heterogeneous data access adapter unit, which can communicate with various heterogeneous data sources in the financial industry's distributed system through a multi-protocol conversion interface. It dynamically adjusts the data receiving format and transmission protocol based on the metadata characteristics of the data source, and standardizes the packaging of financial multi-dimensional heterogeneous data with different structures and storage methods;

[0051] Specifically, this unit is the primary link for the financial industry's distributed system data consistency automation management platform to achieve heterogeneous data integration. Its core technical parameters are reflected in the diversity and flexibility of the multi-protocol conversion interface, which supports multiple network transmission protocols such as TCP / IP, HTTP, FTP, and common data format protocols such as XML, JSON, and CSV. Through real-time analysis of data source metadata characteristics, including data structure (such as the table structure of relational databases, storage mode of unstructured data), data type (numeric type, character type, time type, etc.) and data encoding method, the data receiving format and transmission protocol are dynamically adjusted. The significance lies in breaking the format barriers and protocol differences between different data sources in the financial system, ensuring that various types of heterogeneous data can be connected to the platform with unified standards, and laying the foundation for subsequent data consistency management.

[0052] In terms of implementation, the unit first establishes communication connections with various heterogeneous data sources within the financial industry's distributed systems. Using network detection and metadata reading technologies, it acquires relevant feature information from these data sources. Based on a pre-set rule base and algorithmic model, it converts the format of incoming data, for example, converting table data in a relational database into a data object format that complies with platform standards. It also adapts the transmission protocol to ensure efficient and stable data transmission to the platform. This entire process requires no human intervention, automating and intelligently accessing heterogeneous data and improving both efficiency and accuracy.

[0053] Distributed data storage unit: This unit adopts a partitioned and sharded storage strategy, combining the business attributes and time series characteristics of financial data to store standardized and encapsulated data in a distributed storage cluster. By combining hash indexes with range indexes, it constructs a data storage index structure for fast data storage and retrieval.

[0054] Specifically, this unit mainly solves the problem of efficient storage and fast retrieval of financial data in a distributed storage environment. Its technical parameters include the basis for the division of partition and sharding storage strategies and the construction rules of hash indexes and range indexes. The partition and sharding strategy is based on the business attributes of financial data (such as customer business type, transaction category) and time series characteristics (transaction time, data update time), and the data is stored in different nodes of the distributed storage cluster. The hash index quickly locates the key attributes of the data (such as customer ID, transaction serial number), and the range index is suitable for query scenarios based on time range, numerical interval and other conditions. This storage method not only improves the scalability of the storage system, but also meets the high performance requirements of financial business for data reading and writing, ensuring the real-time nature of data consistency management.

[0055] During the implementation process, after the standardized encapsulated data is transferred to the unit, the data is first allocated to the corresponding storage node according to the preset partitioning strategy. During the storage process, the hash index and range index structure are synchronously constructed to record the storage location and key feature information of the data. When the upper-level application initiates a data query request, the unit quickly locates the storage node where the target data is located through the index based on the query conditions, thereby achieving efficient data retrieval. At the same time, the redundant backup mechanism of the distributed storage cluster is used to ensure the security and reliability of the data, prevent data loss due to single point failure, and provide a stable storage environment for data consistency management.

[0056] An abnormal data feature extraction unit, which transforms the multi-dimensional heterogeneous financial data stored in distributed storage units into graph-structured data. Using an optimized deep graph neural network, it performs multi-layer convolution operations on the graph-structured data to extract abnormal feature vectors from the data. Through a node feature propagation and aggregation mechanism, it mines potential correlation features between data.

[0057] Specifically, this unit focuses on mining potential abnormal features from multi-dimensional heterogeneous financial data, which is a key step in achieving data consistency conflict detection. Its core technology is to construct financial data into a graph structure, in which data entities correspond to the nodes of the graph, and the associations between data entities correspond to the edges of the graph. Through an optimized deep graph neural network, multi-layer convolution operations are performed on the graph structured data. Each layer of convolution operation aggregates the feature information of the node and its neighboring nodes, and through the node feature propagation and aggregation mechanism, the complex dependencies and potential patterns between the data are captured. In this process, the network continuously learns the feature distribution of normal data, so that it can identify abnormal feature vectors that deviate from the normal pattern. This unit is of great significance for timely detection of anomalies in financial data and ensuring data consistency.

[0058] In specific implementation, the multi-dimensional, heterogeneous financial data stored in distributed storage units is first converted into a graph structure. The graph's edge connections and weights are determined based on the business relationships between data entities. This constructed graph data is then fed into an optimized deep graph neural network. The network uses forward propagation to update and extract node features at each layer. After iterative operations across multiple layers, the network ultimately outputs vectors that characterize data anomalies. These anomaly feature vectors serve as an important basis for subsequent data relationship modeling and conflict detection, providing strong support for accurately determining data consistency.

[0059] Heterogeneous data association modeling unit: This unit builds an association model between data based on the extracted abnormal feature vectors, combined with the business rules of financial data and historical data. It uses a directed acyclic graph to represent the dependencies and influence paths between data entities, and quantifies the association weights between data nodes.

[0060] Specifically, this unit aims to reveal the intrinsic connections between financial data, quantify the degree of association between data entities, and provide a model basis for data consistency conflict detection. The technical point is to combine the extracted abnormal feature vectors, and build a correlation relationship model between data based on the business rules of financial data (such as transaction logic, risk control rules) and historical data. The dependency relationship and influence path between data entities are represented in the form of a directed acyclic graph, and the degree of association between each data node is quantified by calculating the feature similarity between nodes and the association weights preset based on business rules. This modeling method can clearly present the complex correlation network between financial data and help the platform accurately grasp the influencing factors of data consistency status.

[0061] During implementation, the system first receives the abnormal feature vectors output by the abnormal data feature extraction unit and, combined with financial domain knowledge, determines the association rules and influence directions between data entities. It then uses a feature similarity calculation algorithm and preset weights to calculate the association weights between each data node. By continuously updating and optimizing model parameters, the association model adapts to the dynamic nature of financial data. The resulting association model serves as input to the consistency conflict detection unit, providing an accurate model basis for detecting data consistency conflicts and improving the accuracy and reliability of conflict detection.

[0062] The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold;

[0063] Specifically, this unit is the core module for determining whether there is a consistency conflict in financial data, and its functional implementation relies on the financial multi-dimensional heterogeneous data anomaly screening model. The model evaluates the data consistency status by calculating the feature similarity and association weight deviation between data nodes. Specifically, the association relationship model constructed by the heterogeneous data association relationship modeling unit is used as input. Based on the feature information and association weights of each data node in the model, the actual features of the data nodes are compared and analyzed with the feature differences under the normal mode, as well as the degree of deviation between the current association weight and the historical stable weight. When these differences and deviations exceed the preset conflict threshold, it is determined that the data node has a consistency conflict, so that potential problems in financial data can be discovered in a timely manner.

[0064] In terms of implementation, this unit performs traversal checks on data nodes in the distributed system, either periodically or when data changes. For each data node, based on the computational logic of the anomaly screening model, it extracts the relevant node's feature information and associated weight data, calculates feature similarity, and analyzes weight deviations. The calculated results are compared with preset conflict thresholds to output consistency conflict determination results for the data node. These determination results serve as the basis for subsequent conflict resolution, providing accurate problem location and analysis for the conflict resolution unit generated by the dynamic policy, ensuring timely and effective resolution of data consistency conflicts.

[0065] A conflict resolution unit for dynamic policy generation. This unit dynamically generates conflict resolution solutions based on detected data consistency conflicts, financial business scenarios, and data priorities, using a preset conflict resolution policy library. These solutions include data update policies, data rollback policies, and data synchronization policies.

[0066] Specifically, after detecting a data consistency conflict, this unit is responsible for generating targeted conflict resolution solutions to ensure the consistency of financial data. Its technical principles are based on a preset conflict resolution strategy library, combining financial business scenarios and data priorities to dynamically generate the most appropriate conflict resolution solution. The strategy library contains a variety of policy templates, such as data update strategies, data rollback strategies, and data synchronization strategies. Each strategy has corresponding applicable conditions and execution rules. When generating a strategy, the unit comprehensively considers factors such as data timeliness, business importance, and data volume. Using an algorithmic model, it calculates the applicability scores of different strategies and selects the strategy with the highest score as the final solution.

[0067] During implementation, upon receiving the conflict determination result from the consistency conflict detection unit, the unit first analyzes the business scenario of the conflicting data, determining information such as the data's business type and the business processes involved. It also assesses the data's priority, for example, determining its importance based on factors such as transaction amount and customer level. Then, based on a pre-set policy generation algorithm, it selects eligible policies from the conflict resolution policy library and dynamically adjusts policy parameters to generate a specific conflict resolution solution. Finally, the generated solution is sent to the relevant data processing module for execution, effectively resolving data consistency conflicts and restoring the consistency of financial data.

[0068] The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects the status information and conflict resolution results of each data node, and feeds back the monitoring data to the heterogeneous data access adaptation unit, distributed data storage unit, abnormal data feature extraction unit, heterogeneous data association relationship modeling unit, consistency conflict detection unit, and dynamic strategy generation conflict resolution unit to perform closed-loop control of data consistency management.

[0069] Specifically, this unit is a key component in achieving closed-loop control of data consistency management. It is responsible for real-time monitoring of the consistency status of data in the distributed system and feeding this information back to other functional units. Its technical implementation relies on the real-time collection and analysis of status information from each data node, including data storage status, update time, and consistency conflict resolution results. Through the comprehensive processing and analysis of this information, a real-time monitoring view of the data consistency status is constructed, enabling the timely identification of potential problems or anomalies in the data consistency management process.

[0070] In terms of implementation, this unit establishes communication connections with other functional units to obtain status information of each data node regularly or in real time. The collected information is processed using data analysis algorithms, such as statistically calculating the frequency of consistency conflicts and analyzing the execution effect of conflict resolution strategies. The analysis results are presented in the form of monitoring reports or status indicators, and the relevant information is fed back to other functional units such as the heterogeneous data access adaptation unit and the distributed data storage unit according to preset feedback rules. Other units adjust and optimize their own operating parameters and processing strategies based on the feedback information, thereby achieving continuous improvement and closed-loop control of data consistency management, and ensuring the efficient operation of data consistency management in distributed systems in the financial industry.

[0071] Preferably, in the abnormal data feature extraction unit, an optimized deep graph neural network is used to extract abnormal features, and its model formula is:

[0072]

[0073] Among them, H (l) Represents the node feature matrix of the lth layer. The nodes correspond to financial data entities, and the features include the numerical attributes and timestamp parameters of the financial data. To add the self-connected adjacency matrix, A represents the relationship between financial data entities, I N is the identity matrix; for The diagonal node degree matrix of W (l) is the trainable weight matrix of the lth layer; σ is the activation function; through multi-layer iterative calculation, the output abnormal feature vector H (L) , L is the total number of network layers.

[0074] Specifically, an optimized deep graph neural network is used to extract anomaly features, constructing financial data into a graph structure where nodes correspond to data entities and features include numerical attributes, timestamps, and other parameters. Through multi-layer convolution operations and node feature propagation and aggregation, the network automatically learns the underlying correlation patterns in the data and outputs anomaly feature vectors. This technology breaks through the traditional method's reliance on explicit rules and can capture complex nonlinear relationships in financial data, improving the accuracy of anomaly detection. In implementation, the graph structure data is first constructed, then iterated through a multi-layer network to ultimately output a vector representing the data's anomaly features, providing a key basis for subsequent conflict detection.

[0075] Preferably, in the heterogeneous data association relationship modeling unit, the formula for constructing the data association relationship model is:

[0076] E ij =α·Sim(x i , x j )+β·Weight(x i , x j )

[0077] Among them, E ij Represents financial data node x i with x j The association weight between i , x j ) is the node feature similarity calculated based on cosine similarity, and the features include financial transaction amount, transaction time, and customer credit rating parameters; Weight(x i , x j ) is the node association weight preset according to financial business rules; α and β are weight adjustment coefficients, which are determined through historical data training and are used to balance the impact of feature similarity and preset weights on the association weight.

[0078] Specifically, when constructing a data association model, feature similarity and pre-set weights based on business rules are comprehensively considered. The feature similarity between nodes is calculated based on cosine similarity, taking into account parameters such as transaction amount, time, and credit rating. This is combined with the pre-set weights assigned by business rules, and the influence of these two factors is balanced through weight adjustment coefficients. This fusion approach considers the inherent feature associations of the data while incorporating financial business knowledge, making the model more aligned with actual business needs. During implementation, feature similarity and pre-set weights are first calculated, and then the final association weights are determined through coefficient adjustment, forming a model that reflects the true dependencies between the data.

[0079] Preferably, in the consistency conflict detection unit, a financial multi-dimensional heterogeneous data anomaly screening model is used to perform conflict detection, and its formula is:

[0080]

[0081] Among them, C k represents the conflict detection index of the kth data node; f i (x k ) is the data node x k The i-th financial data feature value, including account balance and transaction frequency; is the mean value of the feature within the normal data range; is the standard deviation of the feature; ω i is the weight coefficient of each feature, which is determined according to the importance of financial business; when C k When the conflict threshold τ is exceeded, it is determined that the data node has a consistency conflict.

[0082] Specifically, the conflict detection unit incorporates a standardized index calculation method. For each data node, the deviation of multiple financial feature values ​​from the mean within the normal range is calculated and normalized using the standard deviation. Each feature is assigned a weight coefficient based on its business importance, and the accumulated value is used to generate the conflict detection index. When the index exceeds a threshold, a conflict is determined. This method accurately identifies data consistency conflicts by quantifying the degree of feature deviation. During implementation, the conflict index for each node is calculated periodically or in real time, compared to a preset threshold, and the detection result is output.

[0083] Preferably, in the conflict resolution unit for dynamic policy generation, the timeliness of financial data and business priority are considered when generating the data update policy, and the formula is:

[0084]

[0085] Among them, P u Indicates data update priority; T cur is the current time, T lastis the last update time of the data; ΔT is the data update cycle threshold; γ is the time decay coefficient; Priority(x) is the business priority of data x, which is determined according to the financial business type and the amount involved. The update priority of each conflicting data is calculated by this formula to determine the data update order.

[0086] Specifically, when generating data update policies, we consider both data timeliness and business priorities. We prioritize data updates by calculating the ratio of the difference between the current time and the last update time to the update cycle, combining the time decay coefficient with business priorities. This approach prioritizes important and time-sensitive data and effectively allocates system resources. During implementation, we calculate the update priority of each conflicting data using a formula, and execute updates in order of priority, improving the efficiency of data consistency repairs.

[0087] Preferably, the abnormal data feature extraction unit further improves the optimized deep graph neural network and introduces an attention mechanism, the formula of which is:

[0088]

[0089] Among them, Att i is the attention weight of the i-th node, which is used to measure the importance of the node in abnormal feature extraction; W a is the weight matrix of the attention mechanism; Represents the feature vector of the i-th node in the l-th layer; the aggregation process of node features is adjusted through the attention mechanism to enhance the ability to extract abnormal features.

[0090] Specifically, an attention mechanism is introduced into graph neural networks. Attention weights are calculated for each node to adjust the feature aggregation process. Attention weights measure the importance of a node in extracting anomaly features, allowing the network to focus on key data features and suppress irrelevant information. This improvement enhances the model's sensitivity to and ability to extract anomaly features, making it particularly useful for sparse yet significant anomaly patterns in financial data. In implementation, attention weights are calculated at each network layer and applied to feature aggregation, optimizing anomaly feature extraction.

[0091] Preferably, the heterogeneous data association relationship modeling unit considers the dynamic change characteristics of financial data when constructing the association relationship model and adopts a time sliding window mechanism. The formula is:

[0092]

[0093] in, Represents the financial data node x at time t i with x j The association weight between and They are node x at time s i with x j The characteristic vector of ; m is the sliding window size; through the time sliding window mechanism, the association weights between data nodes are dynamically updated to adapt to the dynamic changes of financial data.

[0094] Specifically, a dynamic association model is constructed using a sliding window mechanism. At each point in time, association weights are calculated based on data from the past. By averaging the feature similarities of the data within the window with pre-set weights, changes in data relationships are dynamically reflected. This mechanism adapts to the time-varying nature of financial data, making the association model more timely and accurate. During implementation, an appropriate window size is set, and the data within the window is continuously updated over time, adjusting the association weights in real time.

[0095] Preferably, the consistency conflict detection unit introduces the fluctuation range of historical data when calculating the conflict detection index, and the formula is:

[0096]

[0097] Among them, C' k represents the conflict detection index of the kth data node after improvement; Fluct(x k ) is based on the data node x k The fluctuation factor calculated based on the historical data fluctuation reflects the degree of fluctuation of the data characteristics; by introducing the fluctuation factor, data consistency conflicts can be detected more accurately.

[0098] Specifically, the conflict detection indicator calculation method has been improved by introducing a historical data volatility factor, which reflects the historical volatility of data features. By dividing the feature deviation by the product of the volatility factor and the standard deviation, the detection indicator is more adaptable to the inherent volatility of the data and reduces false positives. This improvement is particularly suitable for highly volatile financial data, improving the specificity and reliability of conflict detection. In implementation, the historical volatility factor for each data feature is first calculated and then incorporated into the conflict detection indicator calculation.

[0099] Preferably, the conflict resolution unit for dynamic policy generation considers the relationship between network bandwidth and data volume when generating the data synchronization policy, and the formula is:

[0100]

[0101] Among them, S sync represents the data synchronization priority; Q is the amount of data to be synchronized; B is the current network bandwidth; λ is the bandwidth impact coefficient; Urgency(x) is the urgency of data x, which is determined according to financial business needs. This formula is used to calculate the synchronization priority of each conflicting data and reasonably arrange the data synchronization order.

[0102] Specifically, when generating a data synchronization strategy, the relationship between network bandwidth and data volume is considered. The synchronization priority is determined by calculating the ratio of the amount of data to be synchronized to the current bandwidth, combining the bandwidth impact coefficient with the data urgency. This method prioritizes the synchronization of urgent and moderately sized data when network resources are limited, avoiding network congestion and improving overall synchronization efficiency. During implementation, network bandwidth is monitored in real time, and the synchronization priority of each data item is calculated using a formula to ensure orderly synchronization.

[0103] like Figure 2 The method for automatically managing data consistency in a distributed system in the financial industry includes the following steps:

[0104] Step S1: Establish communication connections with various heterogeneous data sources in the financial industry distributed system through the heterogeneous data access adapter unit, dynamically adjust the data receiving format and transmission protocol based on the metadata characteristics of the data source, and perform standardized packaging of financial multi-dimensional heterogeneous data;

[0105] Step S2: The standardized and encapsulated data is transferred to the distributed data storage unit. According to the partition and sharding storage strategy, the data is stored in the distributed storage cluster in combination with the business attributes and time series characteristics of the financial data, and a data storage index structure combining hash index and range index is constructed.

[0106] Step S3: The abnormal data feature extraction unit constructs the stored data into graph structure data, uses an optimized deep graph neural network to perform multi-layer convolution operations on the graph structure data, and extracts abnormal feature vectors in the data through a node feature propagation and aggregation mechanism;

[0107] Step S4: The heterogeneous data association relationship modeling unit constructs an association relationship model between data based on the extracted abnormal feature vectors, combined with financial data business rules and historical data, and uses a directed acyclic graph to represent the dependency and influence path between data entities, and quantifies the association weights between each data node;

[0108] Step S5: The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold;

[0109] Step S6: The conflict resolution unit for dynamic policy generation dynamically generates conflict resolution solutions from a preset conflict resolution policy library based on the detected data consistency conflicts, combined with financial business scenarios and data priorities, including data update policies, data rollback policies, and data synchronization policies.

[0110] Step S7: The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects status information and conflict resolution results of each data node, and feeds back the monitoring data to each related unit to perform closed-loop control of data consistency management.

[0111] The automated data consistency management platform and method for distributed systems in the financial industry, through innovative architectural design and the application of advanced technologies, comprehensively overcomes the bottlenecks of existing technologies and demonstrates significant advantages. Regarding heterogeneous data processing, the platform incorporates a heterogeneous data access adaptation unit, employing a multi-protocol conversion interface and metadata-driven adaptive mechanism. This unit automatically identifies and adapts to the formats and protocols of various financial data sources, efficiently integrating multidimensional, heterogeneous data scattered across different systems and achieving standardized encapsulation. This design breaks through the traditional approach's reliance on a single data format, significantly improving data access efficiency and quality and laying a solid foundation for subsequent consistency management.

[0112] In the areas of anomaly detection and conflict resolution, the platform leverages an optimized deep graph neural network and an anomaly screening model for multi-dimensional, heterogeneous financial data to build an intelligent processing system. The abnormal data feature extraction unit maps financial data into a graph structure. Through multi-layer convolution operations and an attention mechanism, it accurately captures potential correlations and abnormal features between data, breaking through the limitations of traditional rule-driven detection methods. The heterogeneous data association relationship modeling unit combines business rules with a time sliding window mechanism to dynamically quantify the association weights between data nodes, enabling the model to adapt to the dynamic changes in financial data. The consistency conflict detection unit incorporates a historical volatility factor to significantly improve the accuracy of conflict detection and avoid false positives and missed detections.

[0113] The platform's dynamic policy generation and closed-loop control mechanisms further enhance data consistency management capabilities. The conflict resolution unit, generated by dynamic policies, automatically generates optimal solutions based on business scenarios and data priorities. Through an intelligent combination of data update, rollback, and synchronization policies, it efficiently resolves data conflicts. The data consistency status monitoring and feedback unit collects system status information in real time, forming a closed-loop control loop to ensure continuous optimization of the entire management process. These innovative designs enable the platform to achieve automated and intelligent management of data consistency in high-concurrency, low-latency financial business environments, comprehensively improving the stability and reliability of financial distributed systems and effectively addressing the shortcomings of existing technologies in heterogeneous data processing, anomaly detection, and conflict resolution.

[0114] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0115] While embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The data consistency automation management platform for distributed systems in the financial industry is characterized by: include: Heterogeneous data access adaptation unit, distributed data storage unit, abnormal data feature extraction unit, heterogeneous data association relationship modeling unit, consistency conflict detection unit, conflict resolution unit for dynamic strategy generation, data consistency status monitoring and feedback unit; The heterogeneous data access adapter unit communicates with various heterogeneous data sources in the financial industry distributed system through a multi-protocol conversion interface, dynamically adjusts the data receiving format and transmission protocol based on the metadata characteristics of the data source, and standardizes the packaging of financial multi-dimensional heterogeneous data with different structures and storage methods; The distributed data storage unit adopts a partitioned and sharded storage strategy, combines the business attributes and time series characteristics of financial data, stores standardized and encapsulated data in a distributed storage cluster, and constructs a data storage index structure by combining hash indexes with range indexes to achieve fast data storage and retrieval; The abnormal data feature extraction unit constructs the financial multi-dimensional heterogeneous data stored in the distributed storage unit into graph-structured data, uses an optimized deep graph neural network to perform multi-layer convolution operations on the graph-structured data, extracts abnormal feature vectors in the data, and mines potential correlation features between the data through a node feature propagation and aggregation mechanism; The heterogeneous data association relationship modeling unit constructs an association relationship model between data based on the extracted abnormal feature vectors, combined with the business rules and historical data of financial data, represents the dependency and influence path between data entities through a directed acyclic graph, and quantifies the association weights between each data node.

2. The data consistency automation management platform for distributed systems in the financial industry according to claim 1 is characterized in that: The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold; The conflict resolution unit for dynamic policy generation, which dynamically generates conflict resolution solutions based on detected data consistency conflicts, combined with financial business scenarios and data priorities, based on a preset conflict resolution policy library, including data update policies, data rollback policies, and data synchronization policies; The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects status information and conflict resolution results of each data node, and performs closed-loop control of data consistency management with other units; The abnormal data feature extraction unit uses an optimized deep graph neural network to extract abnormal features. The model formula is: Among them, H (l) Represents the node feature matrix of the lth layer. The nodes correspond to financial data entities, and the features include the numerical attributes and timestamp parameters of the financial data. To add the self-connected adjacency matrix, A represents the relationship between financial data entities, I N is the identity matrix; for The diagonal node degree matrix of W (l) is the trainable weight matrix of the lth layer; σ is the activation function; through multi-layer iterative calculation, the output abnormal feature vector H (L) , L is the total number of network layers.

3. The data consistency automation management platform for distributed financial systems according to claim 1 is characterized in that: The heterogeneous data association relationship modeling unit constructs a data association relationship model using the following formula: E ij =α·Sim(x i ,x j )+β·Weight(x i ,x j ) Among them, E ij Represents financial data node x i with x j The association weight between i , x j ) is the node feature similarity calculated based on cosine similarity, and the features include financial transaction amount, transaction time, and customer credit rating parameters; Weight(x i , x j ) is the node association weight preset according to financial business rules; α and β are weight adjustment coefficients, which are determined through historical data training and are used to balance the impact of feature similarity and preset weights on the association weight.

4. The data consistency automation management platform for distributed systems in the financial industry according to claim 1 is characterized in that: The consistency conflict detection unit uses a financial multi-dimensional heterogeneous data anomaly screening model to perform conflict detection. The formula is: Among them, C k represents the conflict detection index of the kth data node; f i (x k ) is the data node x k The i-th financial data feature value, including account balance and transaction frequency; is the mean value of the feature within the normal data range; is the standard deviation of the feature; ω i is the weight coefficient of each feature, which is determined according to the importance of financial business; when C k When the conflict threshold τ is exceeded, it is determined that the data node has a consistency conflict.

5. The data consistency automation management platform for distributed systems in the financial industry according to claim 1 is characterized in that: The conflict resolution unit generated by the dynamic strategy takes into account the timeliness and business priority of financial data when generating a data update strategy. The formula is: Among them, P u Indicates data update priority; T cur is the current time, T last is the last update time of the data; ΔT is the data update cycle threshold; γ is the time decay coefficient; Priority(x) is the business priority of data x, which is determined according to the financial business type and the amount involved. The update priority of each conflicting data is calculated by this formula to determine the data update order.

6. The data consistency automation management platform for distributed systems in the financial industry according to claim 1 is characterized in that: The abnormal data feature extraction unit further improves the optimized deep graph neural network and introduces the attention mechanism. The formula is: Among them, Att i is the attention weight of the i-th node, which is used to measure the importance of the node in abnormal feature extraction; W a is the weight matrix of the attention mechanism; Represents the feature vector of the i-th node in the l-th layer.

7. The financial industry distributed system data consistency automation management platform according to claim 1 is characterized in that: The heterogeneous data association relationship modeling unit considers the dynamic change characteristics of financial data when constructing the association relationship model and adopts a time sliding window mechanism. The formula is: in, Represents the financial data node x at time t i with x j The association weight between and They are node x at time s i with x j The characteristic vector of ; m is the sliding window size; through the time sliding window mechanism, the association weights between data nodes are dynamically updated to adapt to the dynamic changes of financial data.

8. The data consistency automation management platform for distributed systems in the financial industry according to claim 1 is characterized in that: The consistency conflict detection unit introduces the fluctuation range of historical data when calculating the conflict detection index. The formula is: Among them, C' k represents the conflict detection index of the kth data node after improvement; Fluct(x k ) is based on the data node x k The volatility factor calculated based on the historical data fluctuations reflects the degree of fluctuation of the data characteristics.

9. The financial industry distributed system data consistency automation management platform according to claim 1 is characterized in that: The conflict resolution unit generated by the dynamic strategy considers the relationship between network bandwidth and data volume when generating the data synchronization strategy. The formula is: Among them, S sync represents the data synchronization priority; Q is the amount of data to be synchronized; B is the current network bandwidth; λ is the bandwidth impact coefficient; Urgency(x) is the urgency of data x, which is determined according to financial business needs.

10. A method for automatically managing data consistency in a distributed system in the financial industry, characterized in that: The following steps are involved: Step S1: Establish communication connections with various heterogeneous data sources in the financial industry distributed system through the heterogeneous data access adapter unit, dynamically adjust the data receiving format and transmission protocol based on the metadata characteristics of the data source, and perform standardized packaging of financial multi-dimensional heterogeneous data; Step S2: The standardized and encapsulated data is transferred to the distributed data storage unit. According to the partition and sharding storage strategy, the data is stored in the distributed storage cluster in combination with the business attributes and time series characteristics of the financial data, and a data storage index structure combining hash index and range index is constructed. Step S3: The abnormal data feature extraction unit constructs the stored data into graph structure data, uses an optimized deep graph neural network to perform multi-layer convolution operations on the graph structure data, and extracts abnormal feature vectors in the data through a node feature propagation and aggregation mechanism; Step S4: The heterogeneous data association relationship modeling unit constructs an association relationship model between data based on the extracted abnormal feature vectors, combined with financial data business rules and historical data, and uses a directed acyclic graph to represent the dependency and influence path between data entities, and quantifies the association weights between each data node; Step S5: The consistency conflict detection unit inputs the constructed association relationship model into the financial multi-dimensional heterogeneous data anomaly screening model, detects data consistency conflicts by calculating the feature similarity and association weight deviation between data nodes, and determines whether there is a data anomaly based on the conflict threshold; Step S6: The conflict resolution unit for dynamic policy generation dynamically generates conflict resolution solutions from a preset conflict resolution policy library based on the detected data consistency conflicts, combined with financial business scenarios and data priorities, including data update policies, data rollback policies, and data synchronization policies. Step S7: The data consistency status monitoring and feedback unit monitors the consistency status of data in the distributed system in real time, collects status information and conflict resolution results of each data node, and feeds back the monitoring data to each related unit to perform closed-loop control of data consistency management.

Images