A method, device and equipment for detecting abnormality of enterprise order data processing link based on consistency check and a storage medium

Abstract

The application discloses a kind of enterprise order data processing link exception detection method, device and equipment based on consistency check, storage medium, and related to computer technical field, comprising: by constructing link containing several data processing nodes, node reads record containing source number and accumulable value from source database;After determining the certainty mapping of source number, the data source check identifier including value and source check information is generated by being associated with accumulable value, when node executes data processing, the check identifier of actual input record is accumulated to obtain input check result;When executing output, based on transaction submission, write success, target storage read-back confirmation result is determined actual output record, and output check result is accumulated. According to input and output check result, internal consistency check and handover consistency check are carried out on each node, and the abnormal node is located and recovered when first check fails, to improve the efficiency of detecting the exception of enterprise order data processing link.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for detecting anomalies in enterprise order data processing links based on consistency verification. Background Technology

[0002] Currently, after a batch of enterprise order data has been cleaned, linked, split, and summarized, the number, primary key, and data granularity of upstream and downstream records may change. Existing total verification can detect under- or over-counting, but it cannot detect "source A is 30 less, source B is 30 more"; whole row hash can identify whether a row has changed, but the row structure before and after splitting and summarizing is different, so it cannot be directly compared.

[0003] Taking a typical enterprise order data processing chain as an example, the original enterprise order data is first cleaned in terms of format, then linked with other tables to supplement fields, then filtered, split, or added records according to rules, and finally summarized and written to the target table. After these steps, one input may become multiple details, and multiple details may only have one summary record left, and there is no longer a unified primary key that can be directly linked.

[0004] In actual troubleshooting, we usually first check if there are any errors in the task, and then compare the number of records and the total values ​​at both ends. After finding discrepancies, we write structured query statements to check layer by layer from the last table backwards. This can be handled when the chain is short, but as the number of tables and processing steps increases, the troubleshooting scripts become longer and longer. Furthermore, these scripts do not actively issue alerts when the total amount remains unchanged but the source is entered incorrectly.

[0005] Currently, the main approaches are as follows: 1. Expand two adjacent tables to the same granularity, and then compare them row by row according to the business primary key.

[0006] 2. Compare the number of records, total values, or full table summary at the start and end points of the link.

[0007] 3. Check the task log to confirm whether each task started and ended normally.

[0008] 4. Check the data lineage to confirm which tables and processing tasks the target field has passed through.

[0009] However, the above approach has the following problems: Question 1: The results are inconsistent, but I don't know where I started going wrong.

[0010] Question 2: Even if the total value is the same, it can still pass if the source of the enterprise order data is written incorrectly.

[0011] Question 3: After the granularity of enterprise order data changes, ordinary summaries can no longer be compared.

[0012] Question 4: The sharding verification of the primary and backup databases cannot be directly applied to the processing chain.

[0013] As can be seen from the above, improving the efficiency of anomaly detection in the enterprise order data processing chain is an urgent problem to be solved. Summary of the Invention

[0014] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for detecting anomalies in enterprise order data processing links based on consistency verification, which can improve the efficiency of anomaly detection in the enterprise order data processing links based on consistency verification. The specific solution is as follows: Firstly, this application provides a method for detecting anomalies in enterprise order data processing links based on consistency verification, including: An enterprise order data processing link is constructed based on enterprise order data processing tasks, including several data processing nodes. The data processing nodes are used to read data records to be processed from the source database. The data records to be processed include source numbers and corresponding cumulative values. Using a preset verification identifier generation tool, an accumulative data source verification identifier is generated corresponding to the data record to be processed; the data source verification identifier includes numerical verification information and source verification information. When each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate the corresponding input verification results. When the data processing node performs a data output operation, it writes the output confirmation result into the target database, determines the actual output data record based on the obtained database readback confirmation result, and accumulates the data source verification identifier of the actual output data record to generate an output verification result. Based on the input verification results and the output verification results, each data processing node is subjected to internal consistency verification and handover consistency verification. When the verification failure is detected for the first time, the corresponding abnormal data processing node is identified so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

[0015] Optionally, the step of using a preset verification identifier generation tool to generate an accumulative data source verification identifier corresponding to the data record to be processed includes: Perform deterministic mapping processing on the source number according to the preset encoding rule to obtain the source mapping value; The source mapping value is associated with the cumulative value to obtain source verification information, and numerical verification information is generated based on the cumulative value. The numerical verification information and the source verification information are combined using a preset verification identifier generation tool to obtain a data source verification identifier of preset length, and the data source verification identifier is associated with and stored with the corresponding data record to be processed. The numerical verification information is used to reflect the changes in the accumulable numerical value, and the source verification information is used to reflect the correspondence between the source number and the accumulable numerical value.

[0016] Optionally, when each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate corresponding input verification results, including: When the data splitting node in the data processing node performs a data splitting operation, it obtains each sub-data record obtained from the splitting, and regenerates the corresponding data source verification identifier based on the source number and cumulative value of each sub-data record. The data source verification identifier is then accumulated based on the regenerated data source verification identifier to obtain the verification result after splitting. When the data aggregation node in the data processing node performs a data aggregation operation, it determines each aggregated data record participating in the aggregation operation and regenerates a corresponding data source verification identifier for each aggregated data record. The aggregated verification result is obtained by accumulating the regenerated data source verification identifiers. The input verification result includes the split verification result and the aggregated verification result.

[0017] Optionally, when each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate corresponding input verification results, including: When the data addition node in the data processing node performs a data addition operation, it acquires the newly added data record and accumulates the data source verification identifiers corresponding to each newly added data record to obtain the addition verification result; When the data deletion node in the data processing node performs a data deletion operation, it acquires the filtered data record and accumulates the data source verification identifier corresponding to the filtered data record to obtain the filtering verification result; the input verification result includes the new verification result and the filtering verification result.

[0018] Optionally, before performing intra-node consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result, the method further includes: Determine the batch identifier and calculation version corresponding to the enterprise order data processing task; The batch identifier is used to limit the range of data participating in the consistency verification; the calculation version is used to limit the source number encoding rules, cumulative value conversion rules, source verification information generation rules, and data source verification identifier calculation rules; when the batch identifier and the calculation version are the same, the corresponding data source verification identifier is used to perform node internal consistency verification and handover consistency verification.

[0019] Optionally, the step of performing intra-node consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result includes: Based on the batch identifier, the node number of the data processing node, the input verification result, the output verification result, the current calculation version, and the output confirmation method, determine the node verification record corresponding to each data processing node; Based on the node verification record, perform an internal consistency check on the data processing node to obtain the internal consistency check result; the internal consistency check is to determine whether the sum of the input verification result and the newly added verification result corresponding to the data processing node is equal to the sum of the output verification result and the filter verification result. Based on the node verification record, the output verification result and the input verification result corresponding to each of the adjacent data processing nodes, a handover consistency verification is performed on each of the adjacent data processing nodes to obtain a handover consistency verification result; the handover consistency verification result is used to determine whether the output verification result of the previous data processing node is equal to the input verification result of the next data processing node.

[0020] Optionally, the step of identifying the corresponding abnormal data processing node upon first detection of a verification failure, and performing abnormal recovery on the abnormal data processing node in the enterprise order data processing chain, includes: When the first failure of the internal consistency check or the handover consistency check is detected, the node position of the corresponding data processing node or the handover position between adjacent data processing nodes is determined as the abnormal location result. Based on the anomaly location result, obtain the node data of the corresponding data processing node, and perform corresponding data processing operations on the node data corresponding to the abnormal data processing node in the enterprise order data processing link to complete the anomaly recovery; the node data includes data processing logs, data transmission records and original order data.

[0021] Secondly, this application provides an anomaly detection device for enterprise order data processing links based on consistency verification, comprising: The data record reading module is used to construct an enterprise order data processing link including several data processing nodes based on the enterprise order data processing task, and to read the data records to be processed from the source database using the data processing nodes; the data records to be processed include the source number and the corresponding cumulative value; The data source verification identifier generation module is used to generate an accumulative data source verification identifier corresponding to the data record to be processed using a preset verification identifier generation tool; the data source verification identifier includes numerical verification information and source verification information. The input verification result generation module is used to obtain the actual input data records that actually participate in the processing in the data processing node when each of the data processing nodes performs data processing operations, and to accumulate and calculate the data source verification identifier corresponding to the actual input data records to generate the corresponding input verification result. The output verification result generation module is used to write the output confirmation result into the target database when the data processing node performs a data output operation, determine the actual output data record based on the obtained database readback confirmation result, and perform cumulative calculation on the data source verification identifier of the actual output data record to generate the output verification result. The node recovery module is used to perform internal consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result, and to identify the corresponding abnormal data processing node when the verification failure is detected for the first time, so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

[0022] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned method for detecting anomalies in the enterprise order data processing chain based on consistency verification.

[0023] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned method for detecting anomalies in the enterprise order data processing link based on consistency verification.

[0024] As can be seen from the above, before performing anomaly detection in the enterprise order data processing chain based on consistency verification, this application needs to bind the source number and the value to the same verification value. This invention generates a fixed-length source verification status at each processing node. This status is calculated jointly by the source number and the symbolic value undertaken by that source, and is transmitted according to unified rules during the splitting and summarizing processes. The system checks whether the source contribution within a node is conserved and whether the output and input between adjacent nodes are consistent, and then determines the first mismatch node or handover boundary according to the processing order.

[0025] Therefore, this application first requires accumulating verification status for the actual reading location, and then synchronously accumulating or filtering verification status according to the actual hit rules, avoiding the blind spot of simply judging "whether the total amount is equal". In this way, the accuracy and efficiency of anomaly detection are improved in the process of detecting anomalies in the enterprise order data processing link, the scope of investigation is significantly narrowed, and the user experience is enhanced. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This application discloses a flowchart of an anomaly detection method for enterprise order data processing links based on consistency verification. Figure 2 This application discloses a specific method for detecting anomalies in enterprise order data processing links based on consistency verification. Figure 3 This application discloses a specific flowchart of node internal verification and adjacent node handover verification. Figure 4 This application discloses a specific flowchart of an additive calculation process that binds sources and values. Figure 5 This is a schematic diagram illustrating the determination of the earliest mismatch node in a specific source relationship disclosed in this application; Figure 6 This is a schematic diagram of a specific program access method for a fixed-length verification record disclosed in this application; Figure 7 This is a schematic diagram of the structure of an anomaly detection device for enterprise order data processing links based on consistency verification disclosed in this application; Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] After a batch of data undergoes cleaning, association, splitting, and aggregation, the number, primary key, and data granularity of upstream and downstream records may change. Existing aggregate checks can detect under- or over-counting, but cannot detect the corresponding data from each source. Furthermore, while whole-row hashing can identify whether a row has changed, the row structure before and after splitting and aggregation is inherently different, making direct comparison impossible. To address this, this application provides a method for detecting anomalies in the enterprise order data processing chain based on consistency checks. This method can capture mismatches of equal sources through source verification status and accurately locate the first observable mismatch position according to the processing sequence for anomaly recovery.

[0030] See Figure 1 As shown in the figure, this invention discloses a method for detecting anomalies in enterprise order data processing links based on consistency verification, including: Step S11: Construct an enterprise order data processing link based on the enterprise order data processing task, which includes several data processing nodes, and use the data processing nodes to read the data records to be processed from the source database; the data records to be processed include the source number and the corresponding cumulative value.

[0031] In this embodiment, the first step is to determine the batch boundary and calculation version V for this verification. Calculation version V is a set of version information describing the calculation caliber, including source number encoding, numerical precision, rounding rules, mapping parameters, and modulus.

[0032] First, this embodiment requires obtaining the source identifier (s) and addable value (v) of the record to be verified, and converting the addable value (v) into a signed integer of uniform precision. In this embodiment, the source identifier uses unambiguous encoding. Second, the constituent fields of the composite source identifier are sequentially written as field identifier, null flag, byte length, and field content; character fields are encoded using UTF-8 (Unicode Transformation Format 8-bit), with null values ​​and empty strings represented separately, leading zeros preserved, and no implicit conversion of case or leading and trailing spaces. The addable value is converted into a signed integer according to the minimum unit specified by the calculation version.

[0033] Step S12: Perform deterministic mapping processing on the source number to obtain the source mapping value, and use a preset verification identifier generation tool to associate the source mapping value with the corresponding accumulative value to generate an accumulative data source verification identifier corresponding to the data record to be processed; the data source verification identifier includes numerical verification information and source verification information.

[0034] In this embodiment, the source number that remains stable during processing needs to be selected first. In one specific implementation, the order number is used. For any record r, the record source number is s, and the signed integer value is v. The amount is converted to the smallest unit of measurement with uniform precision, and negative numbers are used for cancellation. After deterministic mapping, the source number is multiplied by the actual value borne by the source, so that records with the same amount but different sources get different source check codes; after the same source is split into multiple records, the check results of each record can still be accumulated by component.

[0035] This embodiment uses pseudocode logic to represent the calculation process: Let F(r) = (M, C1, C2). Where M = v; C1 = norm(v) H1(s),P1); C2=norm(v H2(s),P2); where norm(x,P) represents normalizing the modulo operation result of x to 0 to P-1. H1 and H2 are two deterministic mappings constructed using different domain separation parameters.

[0036] In this embodiment, the source number is first encoded into a byte sequence E(s), and then the secure hash algorithm SHA-256 (Secure Hash Algorithm 256-bit) and different prefixes are used for field separation: H1(s)=1+(int(SHA-256(0x01||E(s)))mod(P1-1)); H2(s)=1+(int(SHA-256(0x02||E(s)))mod(P2-1)); Where || represents byte sequence concatenation, and the digest is interpreted as an unsigned big-endian integer; in one specific implementation, P1 is 2305843009213693951, and P2 is 2305843009213693921. The above values ​​are used to illustrate a directly implementable calculation method; the mapping function and modulus can also be replaced according to the calculation version.

[0037] The verification status of a batch of records S is: F(S) = Sum(F(r)).

[0038] Specifically, ordinary signed integer addition is performed on M; modulo P1 addition is performed on C1; and modulo P2 addition is performed on C2. The node checksum record only stores M, C1, C2, and the computation version V; it does not store the complete source list, and the number of checksum fields does not increase with the number of source records. The two sets of source checksums are used to reduce the possibility of different source combinations producing the same checksum result, but do not guarantee absolute collision-free operation.

[0039] Furthermore, the source number uses unambiguous encoding. The source number encoding rules, numerical precision, tail difference rules, algorithms and parameters of H1 and H2, and P1 and P2 together constitute the calculation version V. Multiplication uses 128-bit integers, high-precision integers, or equivalent modular multiplication algorithms to avoid intermediate result overflow. Only verification states with the same calculation version can be directly compared; if the versions are different, "incomparable" is returned, and it is not considered as a mismatch in business data.

[0040] Step S13: When each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate the corresponding input verification results.

[0041] In this embodiment, the input verification status Ik (the actual input verification status of the kth node) is accumulated at the actual reading position of each processing node; when a new rule or filtering rule is actually hit, the newly added verification status Ak (the actual newly added verification status) or the filtering verification status Rk (the actual filtering verification status) is accumulated simultaneously. The specific operation is as follows: When the data splitting node in the data processing node performs a data splitting operation, it obtains each sub-data record obtained from the splitting, and accumulates them based on the source number and accumulative value corresponding to each sub-data record to obtain the splitting verification result. During splitting, each actually generated sub-record is recalculated according to its own source number and signed value. For example, if order 1001's 100 yuan is split into 70 yuan and 30 yuan, the 70 yuan should be calculated separately. H (Order 1001) and 30 For H (Order 1001), the checksum of the parent record cannot be copied to any child record. When splitting and generating a tail difference, the tail difference is added to the specified child record in a preset fixed order, and it is guaranteed that the sum of the integer values ​​of all child records is equal to the integer value of the parent record.

[0042] When the data aggregation node in the data processing node performs a data aggregation operation, it determines each aggregation data record participating in the aggregation operation and accumulates each verification component in the data source verification identifier corresponding to each aggregation data record to obtain the corresponding aggregation verification result. During aggregation, the total amount M is calculated using ordinary addition, and the first source verification code C1 and the second source verification code C2 are calculated using modulo P1 and modulo P2, respectively.

[0043] It's worth noting that when a summary record corresponds to multiple sources, its source verification status is obtained by merging the contribution statuses of each source participating in the summary according to their components, and is then associated with and saved as a business field or bypass verification record. When the next node reads this summary record, it reads the associated M, C1, C2, and V, and does not recalculate the source checksum using the new primary key of the summary record.

[0044] When the data addition node in the data processing node performs a data addition operation, it acquires the newly added data record and accumulates the data source verification identifiers corresponding to each newly added data record to obtain the addition verification result Ak. When the data deletion node in the data processing node performs a data deletion operation, it acquires the filtered data record and accumulates the data source verification identifiers corresponding to the filtered data record to obtain the filtering verification result Rk.

[0045] It's worth noting that when a new or filtered event occurs, the system needs to synchronously generate a verification event. Each event must record at least the following: batch_id (Batch Identifier), node_no (Node Number), event_id (Event Identifier), rule_id, source number, signed value, calculation version, and execution status. The event_id serves as the idempotent key; only successfully confirmed events participate in the accumulation, and the corresponding event is invalidated when a node is rolled back. Ak and Rk cannot be recalculated after a difference is discovered. The same batch_id, verification metric, data range, and calculation version are used for the same comparison. When multiple additive metrics such as amount, quantity, and points are verified simultaneously in the same batch, each verification metric forms a separate set of M, C1, C2, and V states; states of different metrics are not mixed, accumulated, or compared.

[0046] Step S14: When the data processing node performs a data output operation, it determines the actual output data record based on the output confirmation result, and accumulates the data source verification identifier of the actual output data record to generate an output verification result; the output confirmation result includes a transaction commit confirmation result, a write success confirmation result, and a target storage readback confirmation result.

[0047] In this embodiment, for the k-th node, the actual output verification status is recorded as Ok. Write nodes must not directly use data in memory to be written as the final Ok. For overall transaction writes, the record is only counted as Ok after the transaction is successfully committed; it is not counted during transaction rollback. For writes that allow partial success, Ok is formed based on the records confirmed as successful in the write result. When the write interface cannot return success details, the target storage is read back based on the success record identifier, and Ok is recalculated only based on the data confirmed to exist during the readback. The confirmation method used by each node is recorded in the node configuration. Actual reads, actual additions, actual filtering, and actual outputs are accumulated as Ik, Ak, Rk, and Ok, respectively, each containing three components: M, C1, and C2.

[0048] Step S15: Based on the input verification result and the output verification result, perform internal consistency verification and handover consistency verification on each data processing node, and determine the corresponding abnormal data processing node when the verification failure is detected for the first time, so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

[0049] In one specific implementation, this application embodiment also needs to determine the batch identifier and calculation version corresponding to the enterprise order data processing task. The batch identifier is used to limit the range of data participating in consistency verification (represented by snapshot number, time range, or data processing level); the calculation version is used to limit the source number encoding rules, cumulative value conversion rules, source verification information generation rules, and data source verification identifier calculation rules; when the batch identifier and the calculation version are the same, the corresponding data source verification identifier is used to perform intra-node consistency verification and handover consistency verification. All nodes under the same batch_id use the same range; completed batches remain unchanged, and data exceeding the batch boundary enters the subsequent correction batch.

[0050] Specifically, node verification records are determined for each data processing node based on the batch identifier, the node number of the data processing node, the input verification result, the output verification result, the current calculation version, and the output confirmation method. Node verification records can be stored separately or appended to existing task execution records. Each record stores at least the batch_id, node_no, operator_type, partition_id, attempt_no (Attempt Number), Ik, Ok, Ak, Rk, calculation version V, output confirmation method, and verification status; Ik, Ok, Ak, and Rk are expanded to store three components: M, C1, and C2. Comparisons are performed by querying by batch_id and sorting by node_no.

[0051] Based on the node verification record, an internal consistency check is performed on the data processing node: It is determined whether the sum of the input verification result and the newly added verification result corresponding to the data processing node is equal to the sum of the output verification result and the filtered verification result. That is, the internal node difference Dk = Ik + Ak - Ok - Rk. Dk is calculated for the three components M, C1, and C2 respectively. M is calculated using signed integer operations, and C1 and C2 are normalized according to their respective modulo values. Passing occurs when all three components are zero, and failure occurs when any component is non-zero.

[0052] Based on the node verification records, the output verification results and input verification results corresponding to each of the adjacent data processing nodes, a handover consistency check is performed on each of the adjacent data processing nodes: it is determined whether the output verification result of the previous data processing node is equal to the input verification result of the next data processing node. That is, the handover difference between adjacent nodes Bk = Ok - I(k+1). When there is no addition or filtering, Ak and Rk are zero; data that has not actually passed through the corresponding rule branch shall not be included in Ak or Rk.

[0053] When the first failure of internal consistency verification or handover consistency verification is detected, the node position of the corresponding data processing node, or the handover position between adjacent data processing nodes, is determined as the anomaly location result. The verification program checks in ascending order of node_no and stops when the first discrepancy occurs. Subsequent investigation combines the original details and business logs. Based on the anomaly location result, the node data of the corresponding data processing node (including data processing logs, data transmission records, and original order data) is obtained, and the corresponding data processing operations are performed on the node data corresponding to the anomaly data processing node in the enterprise order data processing chain to complete the anomaly recovery.

[0054] To further illustrate the effects of the embodiments of this application, this embodiment takes order 1001 with an amount of 100 yuan and order 1002 with an amount of 50 yuan as examples. Order 1001 is split into 70 yuan and 30 yuan according to the rules. If 30 yuan is incorrectly associated with order 1002, the total amount for the billing period is still 150 yuan, and the normal total check will pass; however, the source verification status will change.

[0055] Assume H1(order 1001) = 3, H1(order 1002) = 7, and in this example, the values ​​are less than the modulus P1. The first checksum of the original input is 100. 3+50 7 = 650. After splitting correctly, the first checksum is 70. 3+30 3+50 7 = 650, the same as the input. After splitting in the incorrect way, the 30 yuan used the source number of order 1002, and the first check digit became 70. 3+30 7+50 7 = 770. The total input and output amounts are both 150 yuan, but the check codes 650 and 770 are different, so this node's verification fails.

[0056] In this embodiment, the processing sequence is: read order, clean fields, split rules, summarize by payment period, and write results. After each node completes, the input and output validation status is saved, and the validation program checks nodes sequentially. First, it checks if (Ik+Ak) within the node equals (Ok+Rk). After the node passes internal checks, it checks if Ok of the previous node equals I(k+1) of the next node. If they are not equal, it indicates that data was missing, duplicated, or its source changed during writing, transmission, or rereading. The program checks in ascending order of node_no and stops when the first discrepancy occurs. In this example, both the read and clean nodes pass; the total amount in the split node remains 150 yuan, but the source checksum changes from 650 to 770, making the split node the first observable mismatch node. Even if subsequent summary nodes complete the summary normally according to the error details, the system still returns the previously discovered split node.

[0057] It's worth noting that the node handover point can also be located independently. For example, if the split node processes correctly internally and generates a first source checksum of 650, but during data handover, the 30 yuan record for order 1001 is lost and the 30 yuan record for order 1002 is duplicated for the same amount, the next node will still read a total amount of 150 yuan, but the first source checksum will be 770. In this case, the internal verification of the split node passes, but O3 and I4 are not equal, and the verification result points to the handover boundary between the split node and the next node. The first observable mismatch location is the earliest location where the difference can be found based on the recorded verification status. It is used to narrow down the scope of investigation, but it does not mean that the true root cause at the program code level has been found.

[0058] In one specific implementation, this application embodiment further includes a parallel processing and retry mechanism. During parallel processing, each partition accumulates its local M, C1, and C2 values ​​separately. After a partition is successfully processed, the values ​​are merged into a node verification status based on the components. The batch_id, node_no, and partition_id (Partition Identifier) ​​are used as unique keys for the partition verification record. Only partition records with a status of "confirmed" participate in node aggregation. When a retry occurs for the same partition, the original batch_id, node_no, and partition_id are retained, and attempt_no is incremented. Through unique key or conditional updates, only one successful execution result is allowed to be marked as "confirmed"; other execution results remain unconfirmed or invalid and do not participate in node aggregation, thereby avoiding duplicate accumulation of the same partition.

[0059] In one specific embodiment, this application also includes a corresponding data processing chain verification system, comprising a data acquisition module, an encoding calculation module, a node accumulation module, an output confirmation module, a verification record storage module, and a comparison and positioning module.

[0060] Each module can be deployed within the same data processing service or distributed across multiple processing tasks. The data acquisition module obtains input records, new events, filtered events, and candidate output records before writing at the actual execution location of the node, along with batch_id, node_no, partition_id, and calculation version. The encoding and calculation module completes source number encoding, numerical integerization, and single-source verification status calculation. Data processing nodes can be for cleaning, association, filtering, splitting, summarizing, transmitting, or writing. When a single processing unit cannot form a unified input / output boundary, it is split into multiple basic nodes according to the actual execution order. The node accumulation module accumulates Ik, Ok, Ak, and Rk respectively, and merges the parallel partitioning results. The output confirmation module determines the final Ok based on transaction commit, write-by-write results, or target storage readback results. The verification record storage module saves batch, node, partition, number of attempts, calculation version, and each verification status. Splitting, summarizing, reversing, and parallel merging all use the same set of calculation rules.

[0061] Subsequently, the comparison and positioning module checks whether (Ik+Ak) equals (Ok+Rk) and whether Ok equals I(k+1) in node order. If the check fails, the module outputs batch_id, node_no, failed component, previous and subsequent states, and the corresponding node or handover boundary for later investigation in conjunction with the original details and business logs.

[0062] In one specific implementation, the above method can also perform electronic wallet transaction reconciliation. First, the transaction number and wallet transaction number are converted into a unified reconciliation key agreed upon by both parties, and then this key is used as the source code. When two transactions of the same amount are mismatched, the number of matched entries and the total amount may both be correct, but the source verification codes will be different.

[0063] It is worth mentioning that the key point of this invention is: for each record, a stable source number s and the signed value v actually undertaken by that source are obtained, and the two are bound together to construct a fixed-length source verification state. The source number directly participates in the source check code calculation, rather than being stored separately as a record number. Furthermore, the source verification state of a single record includes a numerical component M and at least one source check code component. This embodiment adopts: F(r) = (M, C1, C2); Where M=v, C1=norm(v) H1(s),P1),C2=norm(v) H2(s),P2).

[0064] During the splitting process, the source number and signed value used for each generated sub-record are recalculated. During the summarization process, ordinary addition is performed on M, and corresponding modular addition is performed on C1 and C2 respectively. Subsequently, for the k-th processing node, actual input Ik, actual output Ok, actual addition Ak, and actual filtering Rk are generated respectively. The components (Ik+Ak) are checked to see if they are equal to (Ok+Rk), and the adjacent nodes are checked to see if Ok is equal to I(k+1). The input verification status is taken from the data actually read by the node; for nodes containing write operations, only data that has been successfully confirmed by transaction or confirmed to exist by reading back from the target storage is counted. Data that has failed to write or has been rolled back by transaction is not included in the final output.

[0065] Furthermore, when the total value M is the same but C1 or C2 is different, it is determined that the correspondence between the source number and the value has changed, which is used to identify equal source mismatches that cannot be detected by ordinary total verification. In addition, the internal verification of the node and the handover verification of adjacent nodes are performed according to the node processing order, and the first node or handover boundary that does not meet the verification conditions is returned as the first observable mismatch position of the batch.

[0066] In this way, the aforementioned technical features work together to form a complete processing procedure from calculating source contribution, cross-granularity transfer, actual boundary data retrieval to anomaly location determination. Simply adding the checksums of multiple records cannot complete the above verification and location.

[0067] In other words, the embodiments of this application do not simply address whether the total amount is equal, but rather whether the correspondence between the source and the value remains correct after the data granularity changes. After the source number and the value are bound, even if A is 30 less and B is 30 more, although the total amount remains the same, the two sets of source check codes will still change. This set of check values ​​can be recalculated as the data is split or directly added as it is summarized. It does not require the upstream and downstream tables to have the same number of records, primary keys, or data granularity, nor does it need to store a complete list of sources that grows with the number of sources in the summary record, making it suitable for placement between consecutive data processing nodes.

[0068] Furthermore, each node calculates a checksum using the actual input and output, and then returns the first inconsistency in the processing order. This narrows down the problem from "a problem with the entire link" to "a problem within a specific node" or "a problem at the junction of two nodes." Even after a checksum fails, the original details still apply. This embodiment of the application is responsible for narrowing the scope of investigation, but does not replace the original data.

[0069] Without changing the source contribution verification relationship, the following alternative solutions can also be used: The source ID can be the original record number, order number, serial number, or other identifiers that remain stable and unambiguously encoded throughout the link. A single source checksum can be used, or two or more source checksums with different field separation parameters and moduli can be used; the node checksum relationship remains unchanged as the number of checksums increases.

[0070] Node verification records can be stored separately or attached to existing task execution records, but batch identifiers, node order, and calculation versions should be retained. When a node simultaneously includes filtering, adding, and splitting, it can be broken down into multiple basic processing steps according to the actual execution order and recorded separately. Write nodes can generate the final output based on successful transaction results, or recalculate the final output after reading back from the target storage based on the successful record identifier. After identifying abnormal nodes, the local verification status can be further calculated by partition key or source number range to further narrow down the detailed query scope.

[0071] When business rules allow for legitimate changes to the source number, this step is treated as an explicit source transfer node: the equivalent contribution from the original source is recorded in Rk, and the equivalent contribution from the new source is recorded in Ak. During actual rule execution, a transfer event is generated containing event_id, rule_id, the original source number, the new source number, a signed value, and the calculated version. If the transferred-out and transferred-in values ​​are not equal, this node fails validation; the transfer event must not be supplemented based on the validation difference.

[0072] Further, see Figure 2 , Figure 2 This is a specific verification flowchart. In this embodiment, the processing flow is as follows: (1) Determine the batch boundary and calculation version of this verification, obtain the source number and addable value of the record to be verified, and convert the addable value into a signed integer with uniform precision.

[0073] (2) Accumulate the input verification status at the actual reading position of each processing node; when a new rule or filtering rule is actually hit, the new verification status or filtering verification status is accumulated synchronously.

[0074] (3) When a node is split, it is recalculated according to the source number and value actually used by each sub-record; when a node is summarized, each verification component is accumulated separately.

[0075] (4) When a node contains a write operation, the final output verification status is formed by the data of successful transaction confirmation or target storage readback confirmation.

[0076] (5) Check the input, addition, output and filtering relationships within the node respectively, and check whether the output and input of adjacent nodes are consistent.

[0077] (6) Check in sequence according to node number. When the verification condition is not met for the first time, output the corresponding node or node junction boundary.

[0078] Furthermore, this application embodiment requires defining the internal conservation of nodes and node handover verification. For the k-th node, the verification states of actual input, actual output, actual addition, and actual filtering are denoted as Ik, Ok, Ak, and Rk, respectively, each containing three components: M, C1, and C2. Internal node differences: Dk = Ik + Ak - Ok - Rk; Intersection difference between adjacent nodes: Bk = Ok - I(k+1).

[0079] The same batch_id, verification index, data range, and calculation version are used for each comparison. When multiple additive indices such as amount, quantity, and points are verified simultaneously in the same batch, each verification index forms a separate set of M, C1, C2, and V states. States of different indices are not mixed, accumulated, or compared. When there are no additions or filters, Ak and Rk are zero; data that has not actually passed through the corresponding rule branch must not be included in Ak or Rk. Dk and Bk are calculated separately for the three components M, C1, and C2. M is calculated using signed integer operations, and C1 and C2 are normalized according to their respective modulo values. A pass is achieved when all three components are zero, and a failure is achieved when any component is non-zero.

[0080] Meanwhile, to ensure data processing accuracy, write nodes must not directly use data in memory ready to be written as the final OK. For full transaction writes, the record is only counted as OK after the transaction is successfully committed; it is not counted during transaction rollback. For writes that allow partial success, OK is formed based on the records confirmed as successful in the write result. If the write interface cannot return success details, the target storage is read back based on the success record identifier, and OK is recalculated only based on the data confirmed to exist during the readback. The confirmation method used by each node is recorded in the node configuration.

[0081] Furthermore, this embodiment introduces an explicit source transfer node. Specifically, when the business rules allow for a legal change to the source number, this step is treated as an explicit source transfer node: the equivalent contribution of the original source is recorded in Rk, and the equivalent contribution of the new source is recorded in Ak. During the actual execution of the rule, a transfer event is generated containing event_id (event identifier), rule_id (rule identifier), the original source number, the new source number, a signed value, and the calculated version. If the transferred-out and transferred-in values ​​are not equal, this node fails verification; the transfer event cannot be supplemented based on the verification difference.

[0082] For internal node verification, see [link / reference]. Figure 3 As shown, Figure 3This is a flowchart illustrating the internal verification of a node and the verification process at the junction with adjacent nodes. For internal node verification, the judgment formula is: I+A=O+R; If the formulas are not equal, it is determined that "there is a problem with the calculation of this node"; for the verification between adjacent nodes, the judgment formula is: previous node O = next node I. If they are not equal, it is determined that "there is a problem with the node handover".

[0083] To further illustrate the verification effect corresponding to the embodiments of this application, see [link to relevant documentation]. Figure 4 As shown, Figure 4 The flowchart shows the cumulative calculation process after the source and value are bound, mainly illustrating the process of "equal source swapping".

[0084] For ease of explanation, assume H1 (order 1001) = 3, H1 (order 1002) = 7, and that the values ​​in this example are less than the modulus P1. The first checksum of the original input is 100×3 + 50×7 = 650. After splitting correctly, the first checksum is 70×3 + 30×3 + 50×7 = 650, the same as the input. After splitting incorrectly, the 30 yuan uses the source number of order 1002, and the first checksum becomes 70×3 + 30×7 + 50×7 = 770. The total input and output amounts are both 150 yuan, but the checksums 650 and 770 are different, so this node fails to verify. The formal implementation will then calculate C1 and C2 simultaneously.

[0085] Furthermore, embodiments of this application require a method for determining the first observable mismatch location, see [link to relevant documentation]. Figure 5 As shown, Figure 5 This diagram illustrates the determination of the earliest mismatch node in the source relationship. Although the nodes after splitting process erroneous data, the data within each node may still pass. Therefore, the system returns the initial "650→770" instead of the subsequent summary node. The processing sequence in this example is: read the order, clean the fields, split the rule, summarize by payment period, and write the result. After each node is completed, the input and output validation status is saved, and the validation program checks the data in node order. The validation program first checks if Ik+Ak=Ok+Rk within the node.

[0086] After the node passes internal checks, the system checks if the Ok value of the previous node equals the I(k+1) value of the next node. If they are not equal, it indicates that data was missing, duplicated, or its source changed during writing, transmission, or rereading. The program checks node_no from smallest to largest and stops when the first discrepancy occurs. In this example, both the read and clean nodes pass; the total amount of the split nodes remains 150 yuan, but the source checksum changes from 650 to 770, making the split node the first observable mismatch. Even if subsequent summary nodes complete the summary normally according to the error details, the system can still return to the previously discovered split node.

[0087] Furthermore, the node handover point can also be located independently. For example, if the split node processes correctly internally and generates a first source checksum of 650, but loses the 30 yuan record for order 1001 during data handover and duplicates the 30 yuan record for order 1002 by the same amount, the next node will still read a total amount of 150 yuan, but the first source checksum will be 770. In this case, the internal verification of the split node passes, but O3 and I4 are not equal, and the verification result points to the handover boundary between the split node and the next node. The first observable mismatch location is the earliest location where the difference can be found based on the already recorded verification status.

[0088] For details on implementation in existing procedures, see [link / reference]. Figure 6 As shown, Figure 6 This diagram illustrates the program access method for fixed-length verification records. The stored content consists of a fixed number of fields, rather than a complete source list that grows with the number of sources. Node verification records can be stored separately or appended to existing task execution records. Each record stores at least batch_id, node_no, operator_type, partition_id, attempt_no, Ik, Ok, Ak, Rk, calculation version V, output confirmation method, and verification status. Ik, Ok, Ak, and Rk are expanded to store three components: M, C1, and C2. Comparisons are performed by querying by batch_id and sorting by node_no. Verification status is accumulated synchronously during program data reading, splitting, summarizing, and writing. Each record undergoes only a fixed number of source mappings, multiplications, and additions; node verification status only retains fixed fields such as M, C1, C2, and calculation version, and the number of fields does not increase with the number of sources participating in the summary.

[0089] Furthermore, this application details the batch, parallel partitioning, and retry processing schemes. Each verification uses a unique batch_id, and the data range is determined at the start of the batch. The data range is represented by a snapshot number, a time range, or a data processing watermark. All nodes under the same batch_id use the same range; completed batches remain unchanged, and data arriving beyond the batch boundary enters subsequent correction batches.

[0090] During parallel processing, each partition accumulates its local M, C1, and C2 values ​​separately. After a partition is successfully processed, the values ​​are merged into a node verification status based on the components. `batch_id`, `node_no`, and `partition_id` serve as unique keys for partition verification records. Only partition records with a status of "confirmed" participate in node aggregation. When a retry occurs within the same partition, the original `batch_id`, `node_no`, and `partition_id` are retained, and `attempt_no` is incremented. Through unique key or conditional updates, only one successful execution result is allowed to be marked as "confirmed"; other execution results remain unconfirmed or invalidated and do not participate in node aggregation, thus avoiding duplicate accumulation of the same partition.

[0091] This application also includes a data acquisition module, an encoding calculation module, a node accumulation module, an output confirmation module, a verification record storage module, and a comparison and positioning module.

[0092] The data acquisition module obtains input records, new events, filtered events, and candidate output records before writing at the actual execution location of the node, along with batch_id, node_no, partition_id, and calculation version. The encoding and calculation module completes source number encoding, numerical integerization, and single-source verification status calculation. Data processing nodes can be cleaning, association, filtering, splitting, summarizing, transmitting, or writing. When a single processing unit cannot form a unified input-output boundary, it is split into multiple basic nodes according to the actual execution order. The node accumulation module accumulates Ik, Ok, Ak, and Rk respectively, and merges the parallel partitioning results. The output confirmation module determines the final Ok based on transaction commit, write-by-write results, or target storage readback results. The verification record storage module saves batch, node, partition, number of attempts, calculation version, and each verification status. Splitting, summarizing, reversing, and parallel merging all use the same set of calculation rules. The comparison and positioning module checks Ik+Ak=Ok+Rk and Ok=I(k+1) according to the node order. When the check fails, the module outputs batch_id, node_no, failed component, previous and subsequent states, and the corresponding node or handover boundary, for subsequent troubleshooting by combining the original details and business logs.

[0093] Furthermore, the embodiments of this application also need to explain the application's scope. This invention checks whether the additable value of the source attribution is conserved, but does not verify all business fields. If within the same source, 70 yuan and 30 yuan are incorrectly split into 60 yuan and 40 yuan, but the total amount of the source is still 100 yuan, this verification may pass, but it still needs to be checked in conjunction with business rules such as the splitting ratio. A fixed-length source verification status cannot reveal the complete source details.

[0094] As can be seen from the above, this application first requires binding the source number and the numerical value to the same check value. This invention generates a fixed-length source check status at each processing node. This status is calculated jointly by the source number and the symbolic value undertaken by that source, and is transmitted according to unified rules during the splitting and summarizing processes. The system checks whether the source contribution within a node is conserved and whether the output and input between adjacent nodes are consistent, and then determines the first mismatch node or handover boundary according to the processing order, thereby improving the efficiency of anomaly detection in the enterprise order data processing chain.

[0095] Accordingly, see Figure 7 As shown, this application also provides an anomaly detection device for enterprise order data processing links based on consistency verification, including: The data record reading module 11 is used to construct an enterprise order data processing link including several data processing nodes based on the enterprise order data processing task, and to read the data records to be processed from the source database using the data processing nodes; the data records to be processed include the source number and the corresponding cumulative value; The data source verification identifier generation module 12 is used to perform deterministic mapping processing on the source number to obtain a source mapping value, and use a preset verification identifier generation tool to associate the source mapping value with the corresponding accumulative value to generate an accumulative data source verification identifier corresponding to the data record to be processed; the data source verification identifier includes numerical verification information and source verification information; The input verification result generation module 13 is used to obtain the actual input data records that actually participate in the processing in the data processing node when each of the data processing nodes performs data processing operations, and to accumulate and calculate the data source verification identifier corresponding to the actual input data records to generate the corresponding input verification result. The output verification result generation module 14 is used to determine the actual output data record based on the output confirmation result when the data processing node performs a data output operation, and to accumulate and calculate the data source verification identifier of the actual output data record to generate an output verification result; the output confirmation result includes a transaction commit confirmation result, a write success confirmation result, and a target storage readback confirmation result; The node recovery module 15 is used to perform internal consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result, and to identify the corresponding abnormal data processing node when the verification failure is detected for the first time, so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

[0096] In some specific embodiments, the data source verification identifier generation module 12 may specifically include: The numbering and encoding unit is used to perform deterministic mapping processing on the source number according to a preset encoding rule to obtain the source mapping value; A binding calculation unit is used to associate the source mapping value with the accumulable value to obtain source verification information, and generate numerical verification information based on the accumulable value. The combination generation unit is used to combine the numerical verification information and the source verification information using a preset verification identifier generation tool to obtain a data source verification identifier of preset length, and to associate and store the data source verification identifier with the corresponding data record to be processed; wherein, the numerical verification information is used to reflect the change of the accumulable numerical value, and the source verification information is used to reflect the correspondence between the source number and the accumulable numerical value.

[0097] In some specific embodiments, the input verification result generation module 13 may specifically include: The split verification unit is used to obtain each sub-data record obtained by splitting when the data splitting node in the data processing node performs a data splitting operation, and to accumulate the sub-data record based on the source number and the cumulative value corresponding to each sub-data record to obtain the verification result after splitting. The summary verification unit is used to determine each summary data record participating in the summary operation when the data summary node in the data processing node performs a data summary operation, and to accumulate each verification component in the data source verification identifier corresponding to each summary data record to obtain the summary verification result corresponding to each summary data record; the input verification result includes the split verification result and the summary verification result.

[0098] In some specific embodiments, the input verification result generation module 13 may specifically include: An additional verification unit is added, which is used to obtain newly added data records when the data addition node in the data processing node performs a data addition operation, and to accumulate the data source verification identifiers corresponding to each newly added data record to obtain the new verification result; The deletion verification unit is used to obtain the filtered data record when the data deletion node in the data processing node performs a data deletion operation, and to accumulate the data source verification identifier corresponding to the filtered data record to obtain the filtering verification result; the input verification result includes the new verification result and the filtering verification result.

[0099] In some specific embodiments, the enterprise order data processing link anomaly detection device based on consistency verification may further include: The parameter determination unit is used to determine the batch identifier and calculation version corresponding to the enterprise order data processing task; wherein, the batch identifier is used to limit the data range participating in consistency verification; the calculation version is used to limit the source number encoding rules, cumulative value conversion rules, source verification information generation rules, and data source verification identifier calculation rules; when the batch identifier and the calculation version are the same, the corresponding data source verification identifier is used to perform node internal consistency verification and handover consistency verification.

[0100] In some specific embodiments, the node recovery module 15 may specifically include: The record generation unit is used to determine the node verification record corresponding to each data processing node based on the batch identifier, the node number of the data processing node, the input verification result, the output verification result, the current calculation version, and the output confirmation method. An internal verification unit is used to perform an internal consistency verification on the data processing node based on the node verification record, and obtain an internal consistency verification result; the internal consistency verification is to determine whether the sum of the input verification result and the newly added verification result corresponding to the data processing node is equal to the sum of the output verification result and the filter verification result; The handover verification unit is used to perform handover consistency verification on each of the adjacent data processing nodes based on the node verification record, the output verification result and the input verification result corresponding to each of the adjacent data processing nodes, and to obtain a handover consistency verification result; the handover consistency verification result is to determine whether the output verification result of the previous data processing node is equal to the input verification result of the next data processing node.

[0101] In some specific embodiments, the node recovery module 15 may specifically include: The anomaly location result determination unit is used to determine the node position of the corresponding data processing node or the handover position between adjacent data processing nodes as an anomaly location result when the first failure of the internal consistency check or the handover consistency check is detected. An anomaly location and recovery unit is used to obtain the node data of the corresponding data processing node based on the anomaly location result, and to perform corresponding data processing operations on the node data corresponding to the abnormal data processing node in the enterprise order data processing link to complete the anomaly recovery; the node data includes data processing logs, data transmission records and original order data.

[0102] Furthermore, embodiments of this application also disclose an electronic device, Figure 8This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the enterprise order data processing link anomaly detection method based on consistency verification disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0103] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output confirmation results to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0104] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0105] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the enterprise order data processing link anomaly detection method based on consistency verification executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0106] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for detecting anomalies in the enterprise order data processing chain based on consistency verification. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0108] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0110] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting anomalies in enterprise order data processing links based on consistency verification, characterized in that, include: An enterprise order data processing link is constructed based on enterprise order data processing tasks, including several data processing nodes. The data processing nodes are used to read data records to be processed from the source database. The data records to be processed include source numbers and corresponding cumulative values. The source number is subjected to deterministic mapping processing to obtain a source mapping value. Then, a preset verification identifier generation tool is used to associate the source mapping value with the corresponding accumulative value to generate an accumulative data source verification identifier corresponding to the data record to be processed. The data source verification identifier includes numerical verification information and source verification information. When each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate the corresponding input verification results. When the data processing node performs a data output operation, it determines the actual output data record based on the output confirmation result, and accumulates the data source verification identifier of the actual output data record to generate an output verification result. The output confirmation results include transaction commit confirmation results, write success confirmation results, and target storage readback confirmation results; Based on the input verification results and the output verification results, each data processing node is subjected to internal consistency verification and handover consistency verification. When the verification failure is detected for the first time, the corresponding abnormal data processing node is identified so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

2. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to claim 1, characterized in that, The process of performing deterministic mapping on the source number to obtain a source mapping value, and then using a preset verification identifier generation tool to associate the source mapping value with the corresponding accumulative value to generate an accumulative data source verification identifier corresponding to the data record to be processed, includes: Perform deterministic mapping processing on the source number according to the preset encoding rule to obtain the source mapping value; The source mapping value is associated with the cumulative value to obtain source verification information, and numerical verification information is generated based on the cumulative value. The numerical verification information and the source verification information are combined using a preset verification identifier generation tool to obtain a data source verification identifier of preset length, and the data source verification identifier is associated with and stored with the corresponding data record to be processed. The numerical verification information is used to reflect the changes in the accumulable numerical value, and the source verification information is used to reflect the correspondence between the source number and the accumulable numerical value.

3. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to claim 1, characterized in that, When each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate corresponding input verification results, including: When the data splitting node in the data processing node performs a data splitting operation, it obtains each sub-data record obtained from the splitting, and accumulates them based on the source number and cumulative value corresponding to each sub-data record to obtain the splitting verification result; When the data aggregation node in the data processing node performs a data aggregation operation, it determines each aggregated data record participating in the aggregation operation, and accumulates each verification component in the data source verification identifier corresponding to each aggregated data record to obtain the aggregation verification result corresponding to each aggregated data record; the input verification result includes the split verification result and the aggregation verification result.

4. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to claim 1, characterized in that, When each of the data processing nodes performs a data processing operation, the actual input data records that actually participate in the processing in the data processing node are obtained, and the data source verification identifiers corresponding to the actual input data records are cumulatively calculated to generate corresponding input verification results, including: When the data addition node in the data processing node performs a data addition operation, it acquires the newly added data record and accumulates the data source verification identifiers corresponding to each newly added data record to obtain the addition verification result; When the data deletion node in the data processing node performs a data deletion operation, it acquires the filtered data record and accumulates the data source verification identifier corresponding to the filtered data record to obtain the filtering verification result; the input verification result includes the new verification result and the filtering verification result.

5. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to claim 4, characterized in that, Before performing intra-node consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result, the method further includes: Determine the batch identifier and calculation version corresponding to the enterprise order data processing task; The batch identifier is used to limit the range of data participating in the consistency verification; the calculation version is used to limit the source number encoding rules, cumulative value conversion rules, source verification information generation rules, and data source verification identifier calculation rules; when the batch identifier and the calculation version are the same, the corresponding data source verification identifier is used to perform node internal consistency verification and handover consistency verification.

6. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to claim 5, characterized in that, The step of performing intra-node consistency verification and handover consistency verification on each data processing node based on the input verification result and the output verification result includes: Based on the batch identifier, the node number of the data processing node, the input verification result, the output verification result, the current calculation version, and the output confirmation method, determine the node verification record corresponding to each data processing node; Based on the node verification record, perform an internal consistency check on the data processing node to obtain the internal consistency check result; the internal consistency check is to determine whether the sum of the input verification result and the newly added verification result corresponding to the data processing node is equal to the sum of the output verification result and the filter verification result. Based on the node verification record, the output verification result and the input verification result corresponding to each of the adjacent data processing nodes, a handover consistency verification is performed on each of the adjacent data processing nodes to obtain a handover consistency verification result; the handover consistency verification result is used to determine whether the output verification result of the previous data processing node is equal to the input verification result of the next data processing node.

7. The method for detecting anomalies in enterprise order data processing links based on consistency verification according to any one of claims 1 to 6, characterized in that, The step of identifying the corresponding abnormal data processing node upon first detection of a verification failure, and performing abnormal recovery on the abnormal data processing node in the enterprise order data processing chain, includes: When the first failure of the internal consistency check or the handover consistency check is detected, the node position of the corresponding data processing node or the handover position between adjacent data processing nodes is determined as the abnormal location result. Based on the anomaly location result, obtain the node data of the corresponding data processing node, and perform corresponding data processing operations on the node data corresponding to the abnormal data processing node in the enterprise order data processing link to complete the anomaly recovery; the node data includes data processing logs, data transmission records and original order data.

8. A device for detecting anomalies in enterprise order data processing links based on consistency verification, characterized in that, include: The data record reading module is used to construct an enterprise order data processing link including several data processing nodes based on the enterprise order data processing task, and to read the data records to be processed from the source database using the data processing nodes; the data records to be processed include the source number and the corresponding cumulative value; The data source verification identifier generation module is used to perform deterministic mapping processing on the source number to obtain a source mapping value, and use a preset verification identifier generation tool to associate the source mapping value with the corresponding accumulative value to generate an accumulative data source verification identifier corresponding to the data record to be processed; the data source verification identifier includes numerical verification information and source verification information; The input verification result generation module is used to obtain the actual input data records that actually participate in the processing in the data processing node when each of the data processing nodes performs data processing operations, and to accumulate and calculate the data source verification identifier corresponding to the actual input data records to generate the corresponding input verification result. The output verification result generation module is used to determine the actual output data record based on the output confirmation result when the data processing node performs a data output operation, and to accumulate and calculate the data source verification identifier of the actual output data record to generate an output verification result. The output confirmation results include transaction commit confirmation results, write success confirmation results, and target storage readback confirmation results; The node recovery module is used to perform internal consistency verification and handover consistency verification on each of the data processing nodes based on the input verification result and the output verification result, and to identify the corresponding abnormal data processing node when the verification failure is detected for the first time, so as to perform abnormal recovery on the abnormal data processing node in the enterprise order data processing link.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the enterprise order data processing link anomaly detection method based on consistency verification as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the enterprise order data processing link anomaly detection method based on consistency verification as described in any one of claims 1 to 7.

Images