Data fusion evaluation system and method based on privacy calculation

Abstract

The invention discloses a data fusion evaluation system and method based on privacy computing, and relates to the technical field of big data analysis, and the method comprises the steps: constructing a node response baseline, precisely analyzing the processing duration of each node, and sensitively capturing a potential transaction rollback risk; screening out abnormal transactions of the node by means of a business sliding window, a neural network model and threshold setting based on a historical transaction rollback rate; and carrying out snapshot storage on the abnormal transaction, and intelligently managing the cache according to the business processing process. Under the privacy calculation complex operation environment, the risk is early warned in advance, invalid consumption of resources is reduced, multi-stage encrypted data tracing is avoided, and resource waste of the whole process from calculation to storage is reduced; the accuracy of data fusion evaluation is improved, it is ensured that the evaluation result is real and reliable, it is ensured that data fusion evaluation can be carried out in a stable and clean environment, and efficient and stable operation of businesses is comprehensively promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and specifically to a data fusion evaluation system and method based on privacy computing. Background Art

[0002] In today's digital wave, data has become the core asset that drives the development of various industries. In this context, the importance of privacy computing has become increasingly prominent, like a mainstay. It can achieve cross-domain circulation and deep integration analysis of data on the basis of ensuring that data privacy is not leaked, breaking the data island. From building accurate risk control models in the financial field to promoting cutting-edge disease research in the medical industry, privacy computing can show its prowess and effectively promote the implementation of innovative applications. It cleverly balances the relationship between efficient use of data and strict privacy protection, laying a solid foundation for the steady development of the digital economy and becoming an indispensable key technology in the digital age.

[0003] However, despite the rich and diverse application scenarios of privacy computing, the emergence of transaction rollbacks will cause a series of thorny problems. Since privacy computing involves complex multi-party calculations and encryption processing, when a user rolls back a transaction, it is necessary to trace back multiple processing stages of the encrypted data, which undoubtedly greatly increases resource consumption from computing to storage. In terms of data consistency, during the rollback process, due to synchronization delays and other reasons, the data status of each participant may be inconsistent. For data fusion evaluation, transaction rollback disrupts the original data processing path, and the fused data needs to be reorganized and integrated, which damages its accuracy and integrity, making it difficult for the evaluation results to reflect the real situation. This seriously hinders the effective implementation of data fusion evaluation and delays the subsequent decision-making process based on the evaluation results. Summary of the invention

[0004] The purpose of the present invention is to provide a data fusion evaluation system and method based on privacy computing to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a data fusion evaluation method based on privacy computing, the data fusion evaluation method comprising the following steps: Step S1: Read the processing time of the user's business data on each node, and build a node response baseline according to the processing time of n groups of historical business data of the user on each node; Step S1-1, analyzing the log records, extracting the timestamps included in the log records to obtain the processing time of the user's business data on each node, wherein the nodes are represented as functional entities interconnected by preset rules or network topology, and are used to perform specific business data processing, storage and transaction coordination tasks; Step S1-2: distinguish the user identity by encrypting the unique identifier, obtain the user's historical n groups of business data, and build a node response baseline based on the processing time of each group of business data on each node and the node correspondence. The node response baseline is represented as a curve graph of the node and the node processing time.

[0006] Extracting the timestamp from the log to obtain the processing time can accurately grasp the time consumption of business processing. Using encrypted identifiers to distinguish user identities and combining the node response baseline built with historical data can intuitively present the regularity of node processing time, efficiently monitor node performance, and quickly discover abnormal processing time, which helps to ensure the smooth operation of business processes.

[0007] Step S2: collect business traffic data of real-time users, set a business sliding window to extract business traffic data, set a baseline deviation threshold for the node response baseline in combination with the neural network model, and monitor the processing time of each node. Filter out nodes with abnormal processing time through the node response baseline combined with the baseline deviation threshold to build a transaction rollback exception set; Step S2-1, real-time collection of user service flow data by service flow monitoring technology, wherein the service flow monitoring technology includes non-intrusive probe or flow mirroring technology to capture encrypted service flow data in a bypass manner; Step S2-2, setting a service sliding window by presetting a time window length and a sliding step length, wherein the service sliding window is represented as a data collection interval defined by a preset time length on the time axis of the service flow data, and the interval moves on the time axis with a fixed sliding step length to collect the service flow data of the user at different times; Step S2-3, construct a training set based on the data extracted from the service sliding window, input the data of the training set into the neural network model for training to obtain the relationship function between the user's service flow and the processing time of each node, and bring the latest data of the service sliding window into the relationship function to obtain the first baseline deviation threshold; Step S2-4, obtaining the transaction rollback times and corresponding business numbers of each transaction in the user's historical transactions, wherein the transaction rollback means that when a transaction is interrupted during the execution of a business due to a local node exception, data consistency conflict, resource competition or external failure, part of the executed operations are undone, so that the business is rolled back to the initial state before the transaction is executed; the business number represents the number of each business operation in the user's historical transactions; the rollback rate of each transaction is obtained by dividing the number of rollbacks of each transaction by the number of businesses, and the value range of the preset coefficient is determined according to the three sigma principle, and the preset coefficient is a value for adjusting the degree of deviation determined based on the three sigma principle and the tolerance of the business to abnormalities; the second baseline deviation threshold is obtained by adding the product of the transaction rollback rate mean and the preset coefficient and the standard deviation; The relationship function between business traffic and the processing time of each node is as follows: ; Where x∈R d is an input vector, which represents the user's business traffic characteristic data; d is determined by the number of business traffic indicators, which include x1, x2, x3 and x4; x1 is the request volume (the number of business requests per unit time); x2 is the data packet size (the amount of data transmitted per unit request); x3 is the number of concurrent connections (the number of business sessions processed simultaneously); x4 is the encryption protocol type (such as TLS version quantization encoding); y is the predicted value of the node processing time; Represented as the L-th layer weight matrix; Represented as the L-th layer bias vector; Represented as activation function; Expressed as model parameters, the specific calculation uses the following formula: ; In the formula, Represented as the k-th layer weight matrix, is represented by the bias vector of the kth layer; L is represented by the total number of neural network layers; the model parameters It is used to define the nonlinear mapping relationship from input features to output prediction values, extract the combination pattern of input features or hidden layer features through linear transformation, determine the contribution weight of different features to the prediction results, and adjust the activation threshold of neurons to enhance the robustness of the model to feature distribution deviation; Step S2-5, obtaining a baseline deviation threshold of the node response baseline according to the first baseline deviation threshold and the second baseline deviation threshold combined with weighted data fusion, and the sum of the first baseline deviation threshold weight and the second baseline deviation threshold weight is always equal to 1; The baseline deviation threshold is calculated using the following formula: ; In the formula, y max is represented by the baseline deviation threshold of the node response baseline; Y is represented by the second baseline deviation threshold; q1 is represented by the weight coefficient of the first baseline deviation threshold; q2 is represented by the weight coefficient of the second baseline deviation threshold; and satisfies ; Step S2-6, filter out nodes whose processing time exceeds the baseline deviation threshold, analyze according to the log records corresponding to the nodes, extract the transactions processed by the nodes to construct a transaction rollback exception set, and the transaction rollback exception set is used to store the transaction set determined to have transaction rollback.

[0008] By using the business sliding window to extract data, combining the neural network model with the historical transaction rollback rate to determine the baseline deviation threshold, accurately screening out nodes with abnormal processing time, and building a transaction rollback exception set, it can detect potential transaction rollback risks in advance, ensure stable business operation, optimize system performance, and ensure the accuracy of data fusion evaluation.

[0009] Step S3, filtering out transactions with abnormal transaction processing responses through transaction rollback exception set combined with logical analysis of transaction processing to construct an abnormal transaction set; Step S3-1, traverse and read the transactions in the transaction rollback exception set, and mark a single transaction as a target exception transaction when searching for associated transactions; Step S3-2: By parsing the log data generated during the transaction execution process, extract the transaction global unique identifier, parent transaction identifier, business entity identifier and operation node information; combine distributed tracing technology to filter out transactions that have a strong logical association with the target abnormal transaction based on the log timestamp to build an abnormal transaction set.

[0010] Traverse the transaction rollback exception collection to mark the target abnormal transaction, making the search for related transactions more targeted. Analyze log data and combine distributed tracing technology to accurately filter out transactions with strong logical associations with target abnormal transactions. It can clearly locate transactions with abnormal transaction processing responses to ensure stable and efficient business operations.

[0011] Step S4: Create a snapshot storage cache, traverse the abnormal transaction set, generate snapshots of the abnormal transactions, and store them in the snapshot storage cache; Step S4-1, traverse and read the transactions in the abnormal transaction set, and extract the complete status data of the transaction execution in real time, the complete status data includes the transaction identifier, associated service node, operation parameters, log serial number and uncommitted data changes; generate a transaction snapshot including a timestamp for the complete status data; Step S4-2: Create a snapshot storage cache through memory partitioning and metadata indexing mechanism, generate an index based on the transaction identifier of the transaction snapshot combined with the timestamp, establish a snapshot index and version mapping table, and store the transaction snapshot in the snapshot storage cache; Traversing the abnormal transaction set to generate transaction snapshots can timely retain the complete status data of the abnormal transaction at the time of execution. This data covers key information such as transaction identification. The snapshot storage cache is created through memory segmentation and metadata indexing mechanism, and index and version mapping tables are established to store transaction snapshots, which greatly improves the orderliness and retrieval efficiency of data storage. When it is necessary to trace the status of abnormal transactions, the corresponding snapshot can be quickly located based on the index, and the actual situation of the abnormal transaction at that time can be accurately obtained, which helps technical personnel to quickly troubleshoot problems and analyze causes, thereby efficiently solving abnormal conditions in business processes and ensuring the stable operation of the business and the integrity of data.

[0012] Step S5: monitor the business data processing process, manage the snapshot storage cache according to the business processing process, and send a signal for data fusion evaluation when the data in the snapshot storage cache is empty; Obtain the processing progress of business data through event monitoring. When business data processing is completed, release the transaction snapshot cache in the snapshot storage cache through the snapshot index and version mapping table. When the data in the snapshot storage cache is empty, send a signal for data fusion evaluation. Analyzing the processing time of nodes to determine transaction rollback can detect potential business risks in advance and buy time for timely intervention. Analyzing related transactions and storing their snapshots in the cache can completely retain information related to abnormal transactions, facilitate subsequent review and tracing, help quickly locate the root cause of the problem, optimize business processes, and improve system stability.

[0013] The cache storage snapshot can retain the complete status data of the abnormal transaction execution in real time, including key information such as transaction identifier, associated service node, operation parameters, log serial number and uncommitted data changes. When the business encounters an abnormal situation and needs to be reviewed, with the help of these snapshots, the technicians can accurately trace back to the moment when the abnormal transaction occurred and clearly understand the actual operation status of the transaction at that time; By continuously monitoring the business processing process, you can grasp the progress of the business in real time and ensure that the snapshot storage cache is operated at the right time. When the business data processing is completed, the transaction snapshot cache is released with the help of the snapshot index and version mapping table, which can effectively recycle memory resources, avoid unnecessary occupation of resources, and improve system operation efficiency. When the snapshot storage cache is empty, the data fusion evaluation signal is issued to ensure that the evaluation is carried out when all abnormal transactions are properly handled and the relevant caches are cleared, so that the start of the data fusion evaluation is based on a clean and stable environment, and the evaluation results are more accurate and reliable.

[0014] Furthermore, a data fusion evaluation system based on privacy computing includes a node response baseline construction module, an abnormal node screening module, an abnormal transaction determination module, an abnormal transaction snapshot storage module and a data fusion evaluation trigger module; The node response baseline construction module is used to read the processing time of user business data at each node and construct a node response baseline; the abnormal node screening module is used to collect data using business traffic monitoring technology, and after analysis and setting thresholds, screen out nodes with abnormal processing time to construct a transaction rollback abnormality set; the abnormal transaction determination module is used to traverse the transaction rollback abnormality set, and combine transaction log analysis and distributed tracking technology to screen out transaction processing response abnormal transactions to construct an abnormal transaction set; the abnormal transaction snapshot storage module is used to extract complete state data from transactions in the abnormal transaction set to generate snapshots, and use a specific mechanism to create cache storage snapshots and establish index mappings; the data fusion evaluation trigger module is used to monitor the business data processing process, and send a data fusion evaluation signal when the business processing is completed and the snapshot storage cache is empty; The output end of the node response baseline construction module is electrically connected to the input end of the abnormal node screening module; the output end of the abnormal node screening module is electrically connected to the input end of the abnormal transaction determination module; the output end of the abnormal transaction determination module is electrically connected to the input end of the abnormal transaction snapshot storage module; the output end of the abnormal transaction snapshot storage module is electrically connected to the input end of the data fusion evaluation trigger module; The node response baseline construction module includes a data reading unit and a baseline construction unit; the data reading unit is used to read the user business data processing time at each node from the log record; the baseline construction unit is used to construct a curve chart of the node and the processing time, i.e., the node response baseline, according to the historical business data at each node processing time; The abnormal node screening module includes a traffic collection unit and a threshold setting monitoring unit; the traffic collection unit is used to collect user service traffic data in real time using service traffic monitoring technology; the threshold setting monitoring unit is used to set the baseline deviation threshold through a neural network model and transaction rollback analysis, and monitor the node processing time to screen abnormal nodes; The abnormal transaction determination module includes a transaction traversal unit and a logic analysis unit; the transaction traversal unit is used to traverse transactions in the transaction rollback exception set; the logic analysis unit is used to parse transaction log data and filter abnormal transactions based on logical association in combination with distributed tracing technology; The abnormal transaction snapshot storage module includes a snapshot generation unit and a cache creation and storage unit; the snapshot generation unit is used to traverse the abnormal transaction set and extract the complete state data of the transaction execution in real time to generate a transaction snapshot with a timestamp; the cache creation and storage unit is used to create a snapshot storage cache using a memory block and metadata index mechanism, store transaction snapshots and establish an index and version mapping table; The data fusion evaluation trigger module includes a process monitoring unit and a cache management trigger unit; the process monitoring unit is used to obtain the business data processing process through event monitoring and monitor the business processing status; the cache management trigger unit is used to release the transaction snapshot in the snapshot storage cache when the business data processing is completed, and send an evaluation signal when the cache is empty.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention can keenly capture potential transaction rollback risks by building a node response baseline and accurately analyzing the processing time of each node. In the complex computing environment of privacy computing, early warning of risks can greatly reduce the ineffective consumption of resources, avoid multi-stage encrypted data tracing caused by transaction rollback, effectively reduce resource waste from computing to storage, and significantly improve system operation efficiency.

[0016] 2. This invention uses business sliding windows, neural network models, and threshold settings based on historical transaction rollback rates to screen out abnormal nodes and transactions, greatly improving the accuracy of data fusion evaluation. In privacy computing scenarios where data consistency is easily disturbed, this precise screening ensures that the evaluation results are true and reliable, effectively guarantees the scientific nature of decision-making based on the evaluation results, and drives the business in the right direction.

[0017] 3. The present invention stores snapshots of abnormal transactions and intelligently manages caches according to the business processing process. In the case where transaction rollbacks are likely to destroy the data processing path, snapshot storage provides a detailed basis for replay, and cache management optimizes resource utilization, ultimately allowing data fusion evaluation to be carried out in a stable and clean environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a data fusion evaluation method based on privacy computing according to the present invention; Figure 2 This is a structural schematic diagram of a data fusion evaluation system based on privacy computing in the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1: Figure 1 As shown, the present invention provides a technical solution, a data fusion evaluation method based on privacy computing, and the data fusion evaluation method comprises the following steps: Step S1: Read the processing time of the user's business data on each node, and build a node response baseline according to the processing time of n groups of historical business data of the user on each node; Step S1-1, analyzing the log records, extracting the timestamps included in the log records to obtain the processing time of the user's business data on each node, wherein the nodes are represented as functional entities interconnected by preset rules or network topology, and are used to perform specific business data processing, storage and transaction coordination tasks; Step S1-2: distinguish the user identity by encrypting the unique identifier, obtain the user's historical n groups of business data, and build a node response baseline based on the processing time of each group of business data on each node and the node correspondence. The node response baseline is represented as a curve graph of the node and the node processing time.

[0021] In the specific implementation, the log collection agent (such as Filebeat or Fluentd) is used to collect the logs of each node in real time, parse the timestamp fields (such as start_time and end_time), and calculate the processing time of a single business on the node tprocess=end_time−start_time; classify the historical n groups of business data by node, count the mean and standard deviation of the processing time of each node, and draw the node response baseline curve: the horizontal axis is the node number, and the vertical axis is the time it takes for the node to process the transaction.

[0022] Step S2: collect business traffic data of real-time users, set a business sliding window to extract business traffic data, set a baseline deviation threshold for the node response baseline in combination with the neural network model, and monitor the processing time of each node. Filter out nodes with abnormal processing time through the node response baseline combined with the baseline deviation threshold to build a transaction rollback exception set; Step S2-1, real-time collection of user service flow data by service flow monitoring technology, wherein the service flow monitoring technology includes non-intrusive probe or flow mirroring technology to capture encrypted service flow data in a bypass manner; Step S2-2, setting a service sliding window by presetting a time window length and a sliding step length, wherein the service sliding window is represented as a data collection interval defined by a preset time length on the time axis of the service flow data, and the interval moves on the time axis with a fixed sliding step length to collect the service flow data of the user at different times; Step S2-3, construct a training set based on the data extracted from the service sliding window, input the data of the training set into the neural network model for training to obtain the relationship function between the user's service flow and the processing time of each node, and bring the latest data of the service sliding window into the relationship function to obtain the first baseline deviation threshold; Step S2-4, obtaining the transaction rollback times and corresponding business numbers of each transaction in the user's historical transactions, wherein the transaction rollback means that when a transaction is interrupted during the execution of a business due to a local node exception, data consistency conflict, resource competition or external failure, part of the executed operations are undone, so that the business is rolled back to the initial state before the transaction is executed; the business number represents the number of each business operation in the user's historical transactions; the rollback rate of each transaction is obtained by dividing the number of rollbacks of each transaction by the number of businesses, and the value range of the preset coefficient is determined according to the three sigma principle, and the preset coefficient is a value for adjusting the degree of deviation determined based on the three sigma principle and the tolerance of the business to abnormalities; the second baseline deviation threshold is obtained by adding the product of the transaction rollback rate mean and the preset coefficient and the standard deviation; Step S2-5, obtaining a baseline deviation threshold of the node response baseline according to the first baseline deviation threshold and the second baseline deviation threshold combined with weighted data fusion, and the sum of the first baseline deviation threshold weight and the second baseline deviation threshold weight is always equal to 1; Step S2-6, filter out nodes whose processing time exceeds the baseline deviation threshold, analyze according to the log records corresponding to the nodes, extract the transactions processed by the nodes to construct a transaction rollback exception set, and the transaction rollback exception set is used to store the transaction set determined to have transaction rollback.

[0023] In specific implementation, encrypted traffic is captured in a bypass manner at the operating system kernel layer through a non-intrusive probe (such as eBPF technology), the traffic sequence is divided according to the business sliding window (such as a window length of 60 milliseconds and a step length of 10 milliseconds), the user's business data is extracted in pieces, and the neural network model is input into the neural network model for training based on the extracted multiple groups of business data to obtain a relationship function between the number of businesses and the processing time of transactions processed by the node, the data newly added in the business sliding window is input into the relationship function to obtain a first baseline deviation threshold, the second baseline deviation threshold is calculated based on the proportional relationship between the number of transaction rollbacks that occurred when the user historically executed the business and the total number of businesses, and weights are assigned to the first baseline deviation threshold and the second baseline deviation threshold to obtain the baseline deviation threshold of the node response baseline.

[0024] Step S3, filtering out transactions with abnormal transaction processing responses through transaction rollback exception set combined with logical analysis of transaction processing to construct an abnormal transaction set; Step S3-1, traverse and read the transactions in the transaction rollback exception set, and mark a single transaction as a target exception transaction when searching for associated transactions; Step S3-2: By parsing the log data generated during the transaction execution process, extract the transaction global unique identifier, parent transaction identifier, business entity identifier and operation node information; combine distributed tracing technology to filter out transactions that have a strong logical association with the target abnormal transaction based on the log timestamp to build an abnormal transaction set.

[0025] In the specific implementation, a distributed tracing system (such as Jaeger) is used to extract the transaction call chain and build a parent-child transaction graph. For example, if transaction A calls service X and triggers transaction B, A is marked as the parent transaction of B, and the logical constraints between transactions (such as "payment transactions must be associated with order creation transactions") are matched through business rule engines (such as Drools); for transactions in the transaction rollback exception set, the logs of the associated service nodes are traced back by the global transaction ID to identify associated abnormal transactions caused by the same data competition (such as oversold inventory) or service timeout.

[0026] Step S4: Create a snapshot storage cache, traverse the abnormal transaction set, generate snapshots of the abnormal transactions, and store them in the snapshot storage cache; Step S4-1, traverse and read the transactions in the abnormal transaction set, and extract the complete status data of the transaction execution in real time, the complete status data includes the transaction identifier, associated service node, operation parameters, log serial number and uncommitted data changes; generate a transaction snapshot including a timestamp for the complete status data; Step S4-2: Create a snapshot storage cache through memory partitioning and metadata indexing mechanism, generate an index based on the transaction identifier of the transaction snapshot combined with the timestamp, establish a snapshot index and version mapping table, and store the transaction snapshot in the snapshot storage cache.

[0027] In the specific implementation, when a transaction rollback is triggered, the transaction context is captured through a memory snapshot tool (such as CRIU), including uncommitted database dirty pages, thread stack information and lock status; timestamps and node topology labels are attached to generate transaction snapshot files; a sharding storage strategy (such as consistent hashing) is used to distribute snapshots to different shards of the Redis cluster according to the transaction ID hash value, and the snapshot index table is maintained through ZooKeeper.

[0028] Step S5: monitor the business data processing process, manage the snapshot storage cache according to the business processing process, and send a signal for data fusion evaluation when the data in the snapshot storage cache is empty; The processing progress of business data is obtained through event monitoring. When the business data processing is completed, the transaction snapshot cache in the snapshot storage cache is released through the snapshot index and version mapping table. When the data in the snapshot storage cache is empty, a signal for data fusion evaluation is issued.

[0029] In the specific implementation, the business data processing completion event is subscribed through the message queue (such as Kafka), triggering the callback function to clear the corresponding transaction snapshot in the snapshot storage cache; when the data in the snapshot storage cache is read as null, it is determined that there is no data in the snapshot storage cache, that is, there is no transaction snapshot, and a signal is issued that data fusion evaluation can be performed.

[0030] Embodiment 2, as Figure 2 As shown, the present invention provides a data fusion evaluation system based on privacy computing, which includes a node response baseline construction module, an abnormal node screening module, an abnormal transaction determination module, an abnormal transaction snapshot storage module and a data fusion evaluation trigger module; The node response baseline construction module is used to read the processing time of user business data at each node and construct a node response baseline; the abnormal node screening module is used to collect data using business traffic monitoring technology, and after analysis and setting thresholds, screen out nodes with abnormal processing time to construct a transaction rollback abnormality set; the abnormal transaction determination module is used to traverse the transaction rollback abnormality set, and combine transaction log analysis and distributed tracking technology to screen out transaction processing response abnormal transactions to construct an abnormal transaction set; the abnormal transaction snapshot storage module is used to extract complete state data from transactions in the abnormal transaction set to generate snapshots, and use a specific mechanism to create cache storage snapshots and establish index mappings; the data fusion evaluation trigger module is used to monitor the business data processing process, and send a data fusion evaluation signal when the business processing is completed and the snapshot storage cache is empty; The output end of the node response baseline construction module is electrically connected to the input end of the abnormal node screening module; the output end of the abnormal node screening module is electrically connected to the input end of the abnormal transaction determination module; the output end of the abnormal transaction determination module is electrically connected to the input end of the abnormal transaction snapshot storage module; the output end of the abnormal transaction snapshot storage module is electrically connected to the input end of the data fusion evaluation trigger module; The node response baseline construction module includes a data reading unit and a baseline construction unit; the data reading unit is used to read the user business data processing time at each node from the log record; the baseline construction unit is used to construct a curve chart of the node and the processing time, i.e., the node response baseline, according to the historical business data at each node processing time; The abnormal node screening module includes a traffic collection unit and a threshold setting monitoring unit; the traffic collection unit is used to collect user service traffic data in real time using service traffic monitoring technology; the threshold setting monitoring unit is used to set the baseline deviation threshold through a neural network model and transaction rollback analysis, and monitor the node processing time to screen abnormal nodes; The abnormal transaction determination module includes a transaction traversal unit and a logic analysis unit; the transaction traversal unit is used to traverse transactions in the transaction rollback exception set; the logic analysis unit is used to parse transaction log data and filter abnormal transactions based on logical association in combination with distributed tracing technology; The abnormal transaction snapshot storage module includes a snapshot generation unit and a cache creation and storage unit; the snapshot generation unit is used to traverse the abnormal transaction set and extract the complete state data of the transaction execution in real time to generate a transaction snapshot with a timestamp; the cache creation and storage unit is used to create a snapshot storage cache using a memory block and metadata index mechanism, store transaction snapshots and establish an index and version mapping table; The data fusion evaluation trigger module includes a process monitoring unit and a cache management trigger unit; the process monitoring unit is used to obtain the business data processing process through event monitoring and monitor the business processing status; the cache management trigger unit is used to release the transaction snapshot in the snapshot storage cache when the business data processing is completed, and send an evaluation signal when the cache is empty.

[0031] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A data fusion evaluation method based on privacy computing, characterized by: The data fusion evaluation method comprises the following steps: Step S1: Read the processing time of the user's business data on each node, and build a node response baseline according to the processing time of n groups of historical business data of the user on each node; Step S2: collect business traffic data of real-time users, set a business sliding window to extract business traffic data, set a baseline deviation threshold for the node response baseline in combination with the neural network model, and monitor the processing time of each node. Filter out nodes with abnormal processing time through the node response baseline combined with the baseline deviation threshold to build a transaction rollback exception set; Step S3, filtering out transactions with abnormal transaction processing responses through transaction rollback exception set combined with logical analysis of transaction processing to construct an abnormal transaction set; Step S4: Create a snapshot storage cache, traverse the abnormal transaction set, generate snapshots of the abnormal transactions, and store them in the snapshot storage cache; Step S5: monitor the business data processing process, manage the snapshot storage cache according to the business processing process, and send a signal for data fusion evaluation when the data in the snapshot storage cache is empty.

2. According to the data fusion evaluation method based on privacy computing according to claim 1, it is characterized by: The specific steps of step S1 are as follows: Step S1-1, analyzing the log records, extracting the timestamps included in the log records to obtain the processing time of the user's business data on each node, wherein the nodes are represented as functional entities interconnected by preset rules or network topology, and are used to perform specific business data processing, storage and transaction coordination tasks; Step S1-2: distinguish the user identity by encrypting the unique identifier, obtain the user's historical n groups of business data, and build a node response baseline based on the processing time of each group of business data on each node and the node correspondence. The node response baseline is represented as a curve graph of the node and the node processing time.

3. The data fusion evaluation method based on privacy computing according to claim 2 is characterized in that: The specific steps of step S2 are as follows: Step S2-1, real-time collection of user service flow data by service flow monitoring technology, wherein the service flow monitoring technology includes non-intrusive probe or flow mirroring technology to capture encrypted service flow data in a bypass manner; Step S2-2, set the service sliding window by presetting the time window length and sliding step. The service sliding window is represented as a data collection interval defined by a preset time length on the time axis of the service flow data. The interval will move on the time axis with a fixed sliding step to collect the service flow data of the user at different times.

4. The data fusion evaluation method based on privacy computing according to claim 3 is characterized in that: In step S2, it also includes: Step S2-3, construct a training set based on the data extracted from the service sliding window, input the data of the training set into the neural network model for training to obtain the relationship function between the user's service flow and the processing time of each node, and bring the latest data of the service sliding window into the relationship function to obtain the first baseline deviation threshold; Step S2-4, obtaining the transaction rollback times and corresponding business numbers of each transaction in the user's historical transactions, wherein the transaction rollback means that when a transaction is interrupted during the execution of a business due to a local node exception, data consistency conflict, resource competition or external failure, part of the executed operations are undone, so that the business is rolled back to the initial state before the transaction is executed; the business number represents the number of each business operation in the user's historical transactions; the rollback rate of each transaction is obtained by dividing the number of rollbacks of each transaction by the number of businesses, and the value range of the preset coefficient is determined according to the three sigma principle, and the preset coefficient is a value for adjusting the degree of deviation determined based on the three sigma principle and the tolerance of the business to abnormalities; the second baseline deviation threshold is obtained by adding the product of the transaction rollback rate mean and the preset coefficient and the standard deviation; Step S2-5, obtaining a baseline deviation threshold of the node response baseline according to the first baseline deviation threshold and the second baseline deviation threshold combined with weighted data fusion, and the sum of the first baseline deviation threshold weight and the second baseline deviation threshold weight is always equal to 1; Step S2-6, filter out nodes whose processing time exceeds the baseline deviation threshold, analyze according to the log records corresponding to the nodes, extract the transactions processed by the nodes to construct a transaction rollback exception set, and the transaction rollback exception set is used to store the transaction set determined to have transaction rollback.

5. According to claim 4, a data fusion evaluation method based on privacy computing is characterized in that: The specific steps of step S3 are as follows: Step S3-1, traverse and read the transactions in the transaction rollback exception set, and mark a single transaction as a target exception transaction when searching for associated transactions; Step S3-2: By parsing the log data generated during the transaction execution process, extract the transaction global unique identifier, parent transaction identifier, business entity identifier and operation node information; combine distributed tracing technology to filter out transactions that have a strong logical association with the target abnormal transaction based on the log timestamp to build an abnormal transaction set.

6. The data fusion evaluation method based on privacy computing according to claim 5 is characterized in that: The specific steps of step S4 are as follows: Step S4-1, traverse and read the transactions in the abnormal transaction set, and extract the complete status data of the transaction execution in real time, the complete status data includes the transaction identifier, associated service node, operation parameters, log serial number and uncommitted data changes; Generate a timestamped transaction snapshot of the complete state data; Step S4-2: Create a snapshot storage cache through memory partitioning and metadata indexing mechanism, generate an index based on the transaction identifier of the transaction snapshot combined with the timestamp, establish a snapshot index and version mapping table, and store the transaction snapshot in the snapshot storage cache.

7. The data fusion evaluation method based on privacy computing according to claim 6 is characterized in that: In step S5, the processing progress of the business data is obtained according to event monitoring. When the business data processing is completed, the transaction snapshot cache in the snapshot storage cache is released through the snapshot index and version mapping table. When the data in the snapshot storage cache is empty, a signal for data fusion evaluation is issued.

8. A data fusion evaluation system based on privacy computing, which is applied to a data fusion evaluation method based on privacy computing according to any one of claims 1 to 7, characterized in that: The data fusion evaluation system includes a node response baseline construction module, an abnormal node screening module, an abnormal transaction determination module, an abnormal transaction snapshot storage module and a data fusion evaluation trigger module; The node response baseline building module is used to read the processing time of user business data at each node and build a node response baseline; The abnormal node screening module is used to collect data using business traffic monitoring technology, and after analysis and setting thresholds, it screens abnormal nodes with processing time to build a transaction rollback abnormal collection; the abnormal transaction determination module is used to traverse the transaction rollback abnormal collection, and combines transaction log analysis and distributed tracking technology to screen transaction processing response abnormal transactions to build an abnormal transaction collection; The abnormal transaction snapshot storage module is used to extract complete state data of transactions in the abnormal transaction set to generate snapshots, create cache storage snapshots and establish index mapping using a specific mechanism; the data fusion evaluation trigger module is used to monitor the business data processing process and send a data fusion evaluation signal when the business processing is completed and the snapshot storage cache is empty; The output end of the node response baseline construction module is electrically connected to the input end of the abnormal node screening module; the output end of the abnormal node screening module is electrically connected to the input end of the abnormal transaction determination module; the output end of the abnormal transaction determination module is electrically connected to the input end of the abnormal transaction snapshot storage module; the output end of the abnormal transaction snapshot storage module is electrically connected to the input end of the data fusion evaluation trigger module.

9. The data fusion evaluation system based on privacy computing according to claim 8, characterized in that: The node response baseline construction module includes a data reading unit and a baseline construction unit; the data reading unit is used to read the user business data processing time at each node from the log record; the baseline construction unit is used to construct a curve chart of the node and the processing time, i.e., the node response baseline, according to the historical business data at each node processing time; The abnormal node screening module includes a traffic collection unit and a threshold setting monitoring unit; the traffic collection unit is used to collect user service traffic data in real time using service traffic monitoring technology; the threshold setting monitoring unit is used to set the baseline deviation threshold through a neural network model and transaction rollback analysis, and monitor the node processing time to screen abnormal nodes; The abnormal transaction determination module includes a transaction traversal unit and a logic analysis unit; the transaction traversal unit is used to traverse transactions in a transaction rollback exception set; the logic analysis unit is used to parse transaction log data and filter abnormal transactions based on logical associations in combination with distributed tracing technology.

10. The data fusion evaluation system based on privacy computing according to claim 8, characterized in that: The abnormal transaction snapshot storage module includes a snapshot generation unit and a cache creation and storage unit; the snapshot generation unit is used to traverse the abnormal transaction set and extract the complete state data of the transaction execution in real time to generate a transaction snapshot with a timestamp; the cache creation and storage unit is used to create a snapshot storage cache using a memory block and metadata index mechanism, store transaction snapshots and establish an index and version mapping table; The data fusion evaluation trigger module includes a process monitoring unit and a cache management trigger unit; the process monitoring unit is used to obtain the business data processing process through event monitoring and monitor the business processing status; the cache management trigger unit is used to release the transaction snapshot in the snapshot storage cache when the business data processing is completed, and send an evaluation signal when the cache is empty.

Images