Data processing method and device, equipment, medium and product

Abstract

The invention discloses a data processing method, device and equipment, a medium and a product, and relates to the technical field of data processing and intelligent information management. The method comprises the following steps: performing difference analysis on target data to obtain a difference atlas; the difference map comprises a change field, a change field type and change field associated information; determining at least one piece of conflict information according to the difference atlas; the conflict information comprises a conflict field, a conflict field type and a field value corresponding to the conflict field; determining a field influence chain according to a semantic dependency directed acyclic graph and the change field; determining a change topology sequence according to the field influence chain; determining the dependency influence degree of the conflict information according to the conflict field and the change topology sequence; and processing the at least one piece of conflict information according to the dependency influence degree to obtain a conflict processing result. Through the technical scheme, the processing effect of the target conflict data can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing and intelligent information management technology, and in particular to a data processing method, apparatus, equipment, medium and product. Background Technology

[0002] In scenarios such as e-commerce, Customer Relationship Management (CRM), ticketing, logistics, and IoT configuration, target data (JSON or semi-structured data) is concurrently edited and synchronized by multiple parties, which can easily lead to problems such as mutual overwriting, disordered order, and semantic conflicts. The mainstream approach is usually based on two-way or three-way difference comparison and semantic merging and conflict resolution. However, due to the lack of systematic modeling of data semantics and business constraints, situations often occur where "merging passes but semantic inconsistency exists," leading to subsequent verification failures or requiring a large amount of manual intervention. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, device, medium, and product to improve the processing efficiency and accuracy of concurrent change conflict data.

[0004] According to one aspect of the present invention, a data processing method is provided, the method comprising:

[0005] Perform difference analysis on the target data to obtain a difference map; the difference map includes the changed fields, the types of the changed fields, and the association information of the changed fields.

[0006] At least one conflict information is determined based on the difference map; the conflict information includes a conflict field, a conflict field type, and a field value corresponding to the conflict field;

[0007] Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain;

[0008] Determine the order of topology changes based on the impact chain of the aforementioned fields;

[0009] The dependency impact of the conflict information is determined based on the conflict field and the changed topology order;

[0010] Based on the degree of dependency influence, the at least one conflict information is processed to obtain a conflict resolution result.

[0011] According to another aspect of the present invention, a data processing apparatus is provided, the apparatus comprising:

[0012] The difference map determination module is used to perform difference analysis on target data and obtain a difference map; the difference map includes changed fields, changed field types, and changed field association information.

[0013] The conflict information determination module is used to determine at least one conflict information based on the difference map; the conflict information includes a conflict field, a conflict field type, and a field value corresponding to the conflict field;

[0014] The field influence chain determination module is used to determine the field influence chain based on the semantic dependency directed acyclic graph and the changed field;

[0015] The topology order determination module is used to determine the order of topology changes based on the influence chain of the field.

[0016] The dependency impact determination module is used to determine the dependency impact of the conflict information based on the conflict field and the changed topology order.

[0017] The conflict resolution result determination module is used to process the at least one conflict information according to the dependency impact degree to obtain the conflict resolution result.

[0018] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the data processing method according to any embodiment of the present invention.

[0024] The technical solution of this invention involves performing difference analysis on target data to obtain a difference map; identifying at least one conflicting piece of information based on the difference map; determining field influence chains based on a semantic dependency directed acyclic graph and changed fields; determining the change topology order based on the field influence chains; determining the dependency influence degree of the conflicting information based on the conflicting fields and the change topology order; and processing at least one conflicting piece of information based on the dependency influence degree to obtain a conflict resolution result. This technical solution, combining a difference map and a semantic dependency directed acyclic graph to identify and process conflicting data in the target data, can improve conflict resolution efficiency.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0030] Figure 4 This is a flowchart of a data processing method provided according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of an object-level alignment process based on optimal matching according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of a heuristic minimum cost repair path search process provided by an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the data processing method of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of target data and related data involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] Existing technological approaches mainly include:

[0039] Text / line level differences and rule merging: suitable for flat text, insensitive to nested structures, field renaming or hierarchical migration, arrays are usually aligned by index, and are prone to mismatching object elements.

[0040] Structured merging based on Schema / AST relies on stable schemas or abstract syntax trees, which are not adaptable to dynamically evolving semi-structured data and are difficult to balance semantic dependencies across fields / objects with business constraints.

[0041] Collaborative editing algorithms (such as OT / CRDT) emphasize usability and eventual consistency, but typically do not encode domain constraints and may converge to semantically invalid states (such as reference breaks, uniqueness conflicts, and illegal state transitions).

[0042] Log- or patch-based replication focuses on replica convergence and conflict accounting, but lacks constraint-oriented automatic verification and repair strategies.

[0043] Under the above approach, several typical challenges of semi-structured data have long lacked effective solutions:

[0044] Field renaming, hierarchical migration, and aggregation splitting: Traditional difference analysis struggles to reliably identify semantic equivalence relationships and is prone to misinterpreting renaming as "deletion + addition".

[0045] Object / array element alignment: Stable matching is difficult when there is no explicit identifier, and index alignment will amplify mismatches; a combination of similarity and optimal matching strategy is needed.

[0046] Cross-object reference and derivation computation: There are constraints such as integrity constraints, uniqueness constraints, scope constraints, and functional dependencies of the referenced objects; the merged derived fields need to be recalculated according to the dependency order.

[0047] Business and state machine constraints: such as states can only be transferred according to legal migration edges, time fields must be monotonic, and amounts must be conserved, simple priority rules or "last written is the standard" may cause implicit data loss or illegal states.

[0048] Global consistency and propagation impact: Changes will propagate along dependencies. The update order and scope of impact need to be topologically reasoned and quantitatively evaluated. Otherwise, local correctness may lead to global failure.

[0049] In summary, existing technologies generally lack explicit modeling and reasoning capabilities for semantically dependent directed acyclic graphs (DAGs), lack guidance on topological ordering and propagation impact measurement based on DAGs, and lack multi-stage constraint verification (before / during / after merging and before / after repair) and automatic repair mechanisms. When faced with complex structures and highly coupled constraints, systems typically degenerate into fixed priorities or "last write as the criterion," requiring extensive manual review and rollback, making it difficult to meet production requirements in terms of efficiency and reliability.

[0050] Therefore, the industry urgently needs a technical approach for semantic merging and conflict resolution that can:

[0051] Semantic dependency directed acyclic graph (DAG) is used to express the functional dependencies, references and constraint propagation relationships between target fields;

[0052] Conflicts are handled progressively at the field level, object level, semantic constraint level, and global consistency level, quantifying dependency impact and determining processing priorities;

[0053] If the verification fails, a minimum cost fix is ​​generated based on the cost function and heuristic search, and audit and rollback capabilities are provided.

[0054] By adaptively updating weights, thresholds, and label confidence based on historical target data and operational feedback, automation and accuracy are improved.

[0055] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention. This embodiment is applicable to situations where conflicting data in a target scenario, such as an e-commerce scenario, needs to be processed. The method can be executed by a data processing device, which can be implemented in hardware and / or software. This device can be configured in an electronic device that carries data processing functions, such as a server. Figure 1 As shown, the method includes:

[0056] S110. Perform differential analysis on the target data to obtain a differential map.

[0057] In this embodiment, the difference map refers to a map composed of target change data of the same objective, i.e., a target change comparison table. Optionally, the difference map includes changed fields, changed field types, and changed field association information. The changed fields refer to the fields in the target data that have been modified; the field change type refers to the type of the changed field; and the changed field association information refers to the information associated with the changed field. Further, the difference map also includes field paths, difference operation types, timestamps, and source identifiers; the difference operation types include, but are not limited to, adding, deleting, modifying, and moving.

[0058] Specifically, a three-way difference analysis is performed on the base version (Base), the first modified version (Left), and the second modified version (Right) of the target data to obtain a difference map. A concrete example is an order scenario where two customer service representatives simultaneously modify the same order. The original order's status at 10:00 was:

[0059] Order number: ORD001;

[0060] Order status: Pending confirmation;

[0061] Item 1: Quantity 2 pieces, unit price 100 yuan, subtotal 200 yuan;

[0062] Total order price: 200 yuan.

[0063] Customer Service A's modification operation (modification time 11:00):

[0064] The order status has been changed from "Pending Confirmation" to "Confirmed".

[0065] The quantity of goods has been changed: from 2 items to 5 items;

[0066] The total order price has been changed from 200 yuan to 500 yuan.

[0067] Customer Service B's edits (edited at 11:15):

[0068] A new product has been added: Product 2, quantity 1 piece, unit price 80 yuan;

[0069] The total order price has been changed from 200 yuan to 280 yuan.

[0070] Then, these order modification data are analyzed to generate a difference map, which includes the change list for Customer Service A and the change list for Customer Service B, as shown in the table below:

[0071] Customer Service A's Change List:

[0072]

[0073] Customer Service B's Change List:

[0074]

[0075] S120. Determine at least one conflict information based on the difference map.

[0076] In this embodiment, conflict information refers to data that conflicts with modifications made to the target. Optionally, conflict information includes conflict fields, conflict field types, and field values ​​corresponding to the conflict fields. Conflict fields refer to fields that cause conflicts with different modifications made to the target; conflict field types refer to the types of conflict fields. The target can be a target order in an order scenario.

[0077] Specifically, by comparing the change lists in the difference map, at least one conflict is identified. For example, the change lists are checked line by line, and if the same field appears in different lists, it is marked as a "potential conflict," and detailed conflict information is recorded.

[0078] Furthermore, conflict evidence vectors are constructed based on the difference graph. Specifically, at least one conflict is identified based on the difference graph, and a conflict evidence vector is generated for each conflict. These conflict evidence vectors may include, but are not limited to: field paths (JSON Pointer or dot-matrix paths), Base / Left / Right values, conflict type, dependency chain summary, context fragments, business weights, historical success rate / rollback rate, conflict frequency, and timestamps, etc., to support automated decision-making, auditability, and replay.

[0079] S130. Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain.

[0080] In this embodiment, the semantic dependency directed acyclic graph (DAG) refers to a relational map composed of related fields of the target data. It can be a directed acyclic graph. The edges of the semantic dependency DAG contain dependency type, constraint expression and strength weight, which are used for consistency propagation and reasoning, and the influence chain summary is backfilled into the conflict evidence vector. A specific example is the semantic dependency DAG of e-commerce orders in an e-commerce scenario, which includes bottom-level fields (such as product quantity, product unit price, discount amount, tax rate, etc.), second-level fields (dependent on bottom-level fields, including product amount = quantity × unit price, tax = order subtotal × tax rate), third-level fields (dependent on second-level fields, including order subtotal = the sum of all product amounts), and top-level fields (dependent on all lower-level fields, including order total price = order subtotal - discount + tax).

[0081] The so-called field influence chain refers to the influence chain consisting of the changed field and its affected fields.

[0082] Specifically, based on the changed field, the associated fields are determined from the semantic dependency directed acyclic graph, that is, the fields affected by the changed field. The changed field and its associated fields are combined according to the chain relationship in the field relationship graph to obtain the field influence chain.

[0083] S140. Determine the order of topology changes based on the field influence chain.

[0084] In this embodiment, changing the topology sorting refers to affecting the topology sorting of fields in the chain; whereby the topology order is used to guide the recalculation order and the determination of conflict escalation conditions.

[0085] Specifically, this is based on the topological order of fields in the field influence chain. For example, in an e-commerce order, if the quantity and discount fields in the difference graph change, then by searching the semantic dependency directed acyclic graph, we get: Quantity changes → Amount needs to be recalculated → Subtotal needs to be recalculated → Total price needs to be recalculated. That is, the field influence chain is: Quantity - Amount - Subtotal - Total Price; and Discount changes → Total price needs to be recalculated. That is, the field influence chain is: Discount - Total Price. Therefore, the topological order of change is: Quantity (bottom layer) - Amount (dependent on quantity) - Subtotal (dependent on amount) - Discount (independent field) - Total Price (dependent on subtotal and discount).

[0086] S150. Determine the dependency impact of conflict information based on the conflict fields and the order of topology changes.

[0087] In this embodiment, the dependency impact is used to assess the extent to which a conflicting field affects other fields; the higher the dependency impact, the higher the priority it needs to be addressed; the processing priority can be determined based on the dependency impact.

[0088] An optional approach is to determine the dependency impact of conflicting information based on conflicting fields and the order of topology changes, including: determining the impact dimension based on the field impact chain; the impact dimension includes the number of impacting fields, chain depth, and field weighting; determining the dimension impact based on the impact dimension and dimension coefficient; determining the conflicting field type of the conflicting field and the conflict coefficient corresponding to the conflicting field type; and determining the dependency impact based on the dimension impact and the conflict coefficient.

[0089] The influence dimension refers to the dimension used to assess the degree of influence of the dependency. Optionally, the influence dimension includes the number of influencing fields, chain depth, and field weighting. The number of influencing fields is used to assess the scope of influence of conflicting fields; chain depth is used to assess the depth of influence of conflicting fields; and field weighting is used to assess the importance of conflicting fields. Optionally, the field weight can be determined based on the field label. For example, the weight of a monetary field (such as total price, amount, etc.) is 3, the weight of a status field (such as target status, payment status, etc.) is 2.5, the weight of a quantity field (such as product quantity, inventory, etc.) is 2, the weight of an identifier field (such as ID, number, etc.) is 1.5, and the weight of a display field (such as name, description, etc.) is 1. It should be noted that the higher the weight, the more important the identifier field.

[0090] The so-called dimensional coefficient is used to assess the importance of the dimension; it can be configured or obtained through learning.

[0091] The so-called conflict coefficient is used to assess the severity of conflict. It can be configured or learned. Different conflict field types correspond to different conflict coefficients. For example, simple numerical difference: conflict coefficient = 1.0 (normal); structural change: conflict coefficient = 1.5 (relatively serious); modification and deletion at the same time: conflict coefficient = 2.0 (serious); external reference involved: conflict coefficient = 1.8 (relatively serious); order / payment status involved: conflict coefficient = 2.5 (very serious).

[0092] Specifically, based on the field influence chain, the number of influencing fields, chain depth, and field weight bonus are determined. Then, the number of influencing fields, chain depth, and field weight bonus are multiplied by their respective dimension coefficients and summed to obtain the dimension influence. Next, the conflicting field type and its corresponding conflict coefficient are determined. Finally, the dimension influence and conflict coefficient are multiplied to obtain the dependency influence. For example, in a scenario where modifying the quantity of goods causes a conflict (i.e., the conflicting field is quantity), specifically: Customer Service A changes the quantity to 5 items, and Customer Service B changes the quantity to 3 items. The dependency influence of the conflicting field is determined as follows: First, the field influence chain is quantity-amount-subtotal-total; the number of influencing fields is 3; the chain depth is 3 levels; the field weight bonus consists of 3 amount fields, with each amount field having a weight of 3, i.e., 3×3=9; the conflicting field type is determined to be a simple difference in values, with a corresponding conflict coefficient of 1; then the dimension influence is determined as: 3×0.4+3×0.3+9×0.3=4.8; finally, the dependency influence is 4.8×1=4.8 points. For example, consider a scenario where an order has been paid but a conflict arises from a change in the amount. Customer service A marks the order as "paid 200 yuan," while customer service B changes the quantity of goods, resulting in a total price of 500 yuan. The specific impact factors of the conflicting fields are determined as follows: First, customer service B's modification chain: quantity → amount → subtotal → total price. Customer service A's modification involves payment status (global status). The impact dimensions are: number of affected fields: 5 (amount, subtotal, total price, payment status, actual payment amount); chain depth: 3 levels; field weighting: 3 amount fields × 3 + 1 status field × 2.5 = 11.5 points; the conflict field type is a global status conflict (involving payment), with a corresponding conflict coefficient of 2.5; the dimension impact is 5 × 0.4 + 3 × 0.3 + 11.5 × 0.3 = 6.35; the dependency impact is 6.35 × 2.5 = 15.875 points.

[0093] Understandably, determining the degree of dependence and influence of conflict information from different dimensions such as breadth, depth, and weight can quantify the severity of the conflict, thereby facilitating subsequent conflict handling.

[0094] S160. Based on the degree of dependence, process at least one piece of conflicting information to obtain the conflict resolution result.

[0095] One alternative approach is to determine the processing priority of conflict information based on dependency impact. Conflict information is then processed sequentially according to this priority, starting with those with higher dependency impact and proceeding to those with lower impact, thus obtaining the conflict resolution result. Constraint verification and consistency checks are performed during the processing. For example, if five conflicts are detected simultaneously, their processing order is shown in the table below:

[0096]

[0097] Another option is to process at least one conflicting piece of information based on the dependency impact score and the impact score threshold to obtain a conflict resolution result. The impact score threshold can be determined through strategy configuration or adaptive learning of the model. For example, dependency impact score < 5 points: automatic processing (the system can handle it completely); 5 points ≤ dependency impact score < 10 points: advanced engine processing (requires Level 3-4); dependency impact score ≥ 10 points: mandatory manual intervention (too complex or too important). A specific example: product quantity conflict (4.8 points) → automatic processing; payment status conflict (15.875 points) → manual processing.

[0098] Another option is to generate a conflict report based on the degree of dependency impact. For example, the conflict field could be: quantity of goods; impact score: 4.8 / 10.0 (moderately severe); scope of impact: 3 fields (amount, subtotal, total price); depth of impact: 3 levels; business importance: high (involves monetary calculations); recommended strategy: Level 2 object-level alignment; expected success rate: 92%.

[0099] The technical solution of this invention involves performing difference analysis on target data to obtain a difference map; identifying at least one conflicting piece of information based on the difference map; determining field influence chains based on a semantic dependency directed acyclic graph and changed fields; determining the change topology order based on the field influence chains; determining the dependency influence degree of the conflicting information based on the conflicting fields and the change topology order; and processing at least one conflicting piece of information based on the dependency influence degree to obtain a conflict resolution result. This technical solution, combining a difference map and a semantic dependency directed acyclic graph to identify and process conflicting data in the target data, can improve conflict resolution efficiency.

[0100] Figure 2 This is a flowchart of a data processing method provided by an embodiment of the present invention. Based on the above embodiments, this embodiment further elaborates on the specific processing of conflict information, especially optimizing the step of "processing at least one piece of conflict information according to the degree of dependency influence to obtain a conflict processing result," providing an optional implementation scheme. Figure 2 As shown, the method includes:

[0101] S210. Perform differential analysis on the target data to obtain a differential map.

[0102] The difference map includes the changed fields, the types of changed fields, and the related information of the changed fields;

[0103] S220. Determine at least one conflicting piece of information based on the difference map.

[0104] Conflict information includes the conflicting field, the type of the conflicting field, and the field value corresponding to the conflicting field;

[0105] S230. Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain.

[0106] S240. Determine the order of topology changes based on the field influence chain.

[0107] S250. Determine the dependency impact of conflict information based on the conflict field and the order of topology changes.

[0108] S260. Determine the conflict type of the conflict information.

[0109] S270. Process the conflict information according to the processing method corresponding to the conflict type to obtain the conflict processing result.

[0110] In this embodiment, the conflict type refers to the type of conflict handling, including no conflict, simple conflict, complex conflict, special conflict, and global consistency conflict.

[0111] An alternative approach is to merge at least one change list if the conflict type is non-conflictual and execute the change information, for example: original target: status = "pending confirmation", quantity = 2 items; customer service A changed: status = "confirmed"; customer service B changed: quantity = 5 items; result: both are accepted (status = "confirmed", quantity = 5 items).

[0112] An alternative approach is to directly select the target change information from at least two change information options based on the modification time, and execute the target change information. For example, the original target is: price = 100 yuan; Customer service A changed it to: price = 120 yuan; Customer service B changed it to: price = 150 yuan; The decision engine determines: select Customer service B's modification (because it is later); Result: price = 150 yuan.

[0113] An optional approach is to process the conflict information according to the processing method corresponding to the conflict type to obtain the conflict processing result, including: if the conflict type is a complex conflict, then process the conflict information according to the field influence chain and the change topology order to obtain the conflict processing result.

[0114] Specifically, if the conflict type is complex, a chain reaction is triggered. This means that based on the data field's influence chain, the conflict information is processed according to the change topology order to obtain the conflict resolution result, and the relevant fields on the influence chain are recalculated for consistency. A specific example is as follows: Original order: Quantity = 2 items, Amount = 100 yuan, Subtotal 100, Total Price = 100 yuan; Customer service A changed: Quantity = 5 items; The processing flow for Quantity → Amount → Subtotal → Total Price is: Accept Quantity = 5 items, Automatically recalculate Amount = 5 × 50 = 250 yuan, Automatically recalculate Total Price = 250 + 25 (tax) = 275 yuan.

[0115] It is understandable that modifying conflict information by changing the topological order can improve conflict resolution efficiency.

[0116] An optional approach is to process the conflict information according to the processing method corresponding to the conflict type if the conflict information includes arrays and / or objects, to obtain a conflict processing result. This includes: if the conflict type includes special conflicts, intelligent merging processing of the conflict information is performed based on array merging strategies and / or nested object merging strategies to obtain a conflict processing result. Where there is no explicit identifier, alignment is based on comprehensive similarity and optimal matching.

[0117] Among them, the array merging strategy refers to an intelligent merging strategy based on product IDs, used for merging product lists. The nested object merging strategy refers to a deep recursive merging strategy, used for merging customer addresses, etc.

[0118] Specifically, in object / array merging scenarios, element alignment is prioritized based on explicit IDs. When explicit IDs are missing, a comprehensive similarity matrix is ​​constructed (integrating key field equality, text similarity, structural similarity, etc.), and the Hungarian algorithm is used to obtain the optimal match. The similarity threshold can be configured or obtained through training. After alignment is completed, the matched nested objects are recursively merged, and consistency checks and constraint verifications are performed on the merged results. Unmatched elements are added, retained, or marked for review according to a strategy.

[0119] For cases where the conflict information includes arrays, the following example illustrates the situation: Original order: Product 1: ID=1, Name = "Product A", Product 2: ID=2, Name = "Product B"; Customer Service A modifies: Product 2: Renamed to "Product B Renamed"; Customer Service B modifies: Add Product 3: ID=3, Name = "Product C"; Merging strategy: Identify the same product through the "ID" field: ID=1: All three versions are the same → Keep unchanged; ID=2: Customer Service A changed the name → Accept "Product B Renamed"; ID=3: Customer Service B added a product → Accept the addition. Final conflict resolution result: Product 1: ID=1, Name = "Product A"; Product 2: ID=2, Name = "Product B Renamed"; Product 3: ID=3, Name = "Product C".

[0120] For cases where conflicting information includes objects, consider the following example: Original order: Customer Name: Zhang San, Address City: Beijing, Address District: Chaoyang District; Customer Service A modifies: Address City changes to: Shanghai; Customer Service B modifies: Address District changes to: Pudong New Area; Merging strategy: Recursively process customer objects, recursively process address objects, City field: only modified in Customer Service A → accept "Shanghai", District field: only modified in Customer Service B → accept "Pudong New Area". Final conflict resolution result (intelligent deep merging): Customer Name: Zhang San, Address City: Shanghai, Address District: Pudong New Area.

[0121] It is understandable that intelligent processing of conflict information of different conflict types can improve conflict resolution efficiency.

[0122] It should be noted that conflicts are handled progressively at the field level → object level → semantic constraint level → global consistency level, with upgrades or rollbacks implemented when necessary, and auditable operation logs and rollback points are generated.

[0123] The technical solution provided by this invention involves performing difference analysis on target data to obtain a difference map; determining at least one conflict information based on the difference map; the conflict information includes a conflict field, a conflict field type, and the field value corresponding to the conflict field; determining the field influence chain based on a semantic dependency directed acyclic graph and the changed field; determining the change topology order based on the field influence chain; determining the dependency influence degree of the conflict information based on the conflict field and the change topology order; determining the conflict type of the conflict information; and processing the conflict information according to the processing method corresponding to the conflict type to obtain the conflict processing result. This technical solution improves conflict processing efficiency by using different processing methods for different conflict types.

[0124] Figure 3 This is a flowchart of a data processing method provided by an embodiment of the present invention; based on the above embodiments, this embodiment further verifies and repairs the conflict handling results, providing an optional implementation scheme. Figure 3 As shown, the method includes:

[0125] S310. Perform differential analysis on the target data to obtain a differential map.

[0126] The difference map includes the changed fields, the types of changed fields, and the related information of the changed fields;

[0127] S320. Determine at least one conflicting piece of information based on the difference map.

[0128] Conflict information includes the conflicting field, the type of the conflicting field, and the field value corresponding to the conflicting field;

[0129] S330. Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain.

[0130] S340. Determine the order of topology changes based on the field influence chain.

[0131] S350. Determine the dependency impact of conflict information based on the conflict field and the order of topology changes.

[0132] S360. Based on the degree of dependency influence, process at least one conflicting piece of information to obtain the conflict resolution result.

[0133] S370. Perform semantic integrity verification on the conflict resolution results.

[0134] S380. If verification fails, the conflict resolution result shall be corrected.

[0135] Semantic integrity verification includes constraint verification, which checks whether all predefined constraint rules are satisfied; dependency verification, which verifies whether the dependency relationships between fields are consistent; type verification, which ensures the consistency and compatibility of data types; and business logic verification, which checks the integrity of business rules.

[0136] Specifically, based on a semantically dependent directed acyclic graph (DAG), the semantic integrity of conflict resolution results is verified. If the verification succeeds, a success result is returned; if the verification fails, the conflict resolution result is located and analyzed for repair, and a pre-verification result and rollback point are output, i.e., a repair patch is generated. The repair patch uses the standard JSON Patch format to ensure the atomicity and reversibility of the operation, merges related operations, and reduces patch complexity. It also supports pre-verification and rollback mechanisms for patches, and records all repair operations for subsequent auditing and analysis. For example, using a semantically dependent DAG for semantic verification, the system checks all constraints:

[0137] Product Amount = Quantity × Unit Price (Approved);

[0138] Order subtotal = Sum of all item amounts (passed);

[0139] Total order price = Subtotal + Tax (Failed!);

[0140] Actual value: total = 200;

[0141] Expected value: total = 250 + 25 = 275.

[0142] An optional approach to repair conflict resolution results includes: identifying failed nodes from the conflict resolution results; backtracking dependency paths based on the failed nodes and the directed acyclic graph of semantic dependencies to construct a failed dependency graph; searching for repair paths based on the failed dependency graph to determine at least one candidate repair path; determining the repair cost of the candidate repair path; determining the target repair path from the at least one candidate repair path based on the repair cost; and repairing the conflict resolution results based on the target repair path.

[0143] In this context, a failure node refers to the node corresponding to a field that failed verification in the conflict resolution result. A failure dependency graph is a field map built based on failure nodes and their affected fields; it is a subgraph of the relational field map. A candidate repair path refers to possible field modification paths. Repair cost refers to the cost incurred when modifying a conflict based on a candidate repair path, including but not limited to the number of modifications, operational costs, dependency impact costs, constraint violation costs, and business risk costs. The target repair path refers to the final determined conflict repair path.

[0144] Specifically, the failure node is first identified from the conflict resolution results. Based on the failure node, the failure dependency graph is constructed by tracing upwards from the semantic dependency directed acyclic graph (DAG) to identify which fields affect the failure node, and downwards from the DAG to identify which fields the failure node affects. Then, a heuristic minimum cost search algorithm is used to intelligently search for repair paths based on the failure dependency graph, obtaining at least one candidate repair path and determining the repair cost of each candidate repair path. The candidate repair path with the lowest repair cost among the at least one candidate repair path is selected as the target repair path. The conflict resolution results are then repaired using the target repair path and applied in a standard patch format. For example, a semantic dependency directed acyclic graph (containing 15 nodes): product ID, product name, quantity, unit price, amount, subtotal, discount, tax, total price, customer information, address, etc.; the extracted invalid dependency graph (5 nodes): invalid node: total order price; tracing upwards (who affects it): subtotal → total order price; tax → total order price; tracing downwards (who it affects): total order price → payment amount;

[0145] Complete failure dependency graph: subtotal(250)→total(200)→payment Amount(200); tax(25)↗; Note: the value of total(200) is not equal to subtotal+tax(275).

[0146] Using a dependency graph, a repair path was searched. Possible repair paths were: Path 1: Repair the `total` field (lowest cost). Operation: Change `total` from 200 to 275. Cost: 1 modification. Impact: Requires simultaneous modification of `payment Amount`. Path 2: Repair the `subtotal` and `tax` fields (high cost). Operation: Change `subtotal` from 250 to 175, and `tax` from 25 to 0. Cost: 2 modifications + violation of business logic (tax cannot be 0). Path 3: Repair all dependent fields (highest cost). Operation: Recalculate all product amounts, subtotals, taxes, and total prices. Cost: 5 modifications. Impact: Too large a scope of modification, high risk. The final system selection was: Path 1 (lowest cost, logically sound) as the target repair path.

[0147] The detailed steps for constructing the failure dependency graph are as follows:

[0148] Step 1: Identify the failed node. Verification revealed that the value of the total field was incorrect, and it was marked as a failed node.

[0149] Step 2: Backtrack the dependencies upwards, starting from the total node, and trace upwards along the semantic dependency DAG: What does total depend on? → subtotal (order target subtotal) → tax (tax); What does subtotal depend on? → items[0].amount → items[1].amount → items[2].amount Continue tracing back to the bottom layer...;

[0150] Step 3: Trace the impact downwards, starting from the total node, and find out who depends on it: Who depends on total? → payment Amount → refund Amount (if any);

[0151] Step 4: Extract subgraphs from the complete DAG, i.e., the semantically dependent directed acyclic graph:

[0152] Failed node: total;

[0153] Upstream nodes: subtotal, tax (affecting total);

[0154] Downstream node: payment amount (affected by total);

[0155] Related dependency edges;

[0156] Form a failure dependency graph (with only 5 nodes instead of the full 15).

[0157] Step 5: Mark failure information on the failure dependency graph:

[0158] Which field is invalid (total)?

[0159] What is the expected value (275)?

[0160] What is the actual value (200)?

[0161] Which constraint (total = subtotal + tax) was violated?

[0162] Understandably, constructing a failure dependency graph for repair can narrow the search scope, increase the search speed, reduce repair risks, make it easier to find the optimal solution, and increase the success rate of repair. At the same time, the failure dependency graph only contains the parts related to the problem, which can avoid interference, focus on repair, repair only the necessary fields, avoid unnecessary chain reactions, and reduce the cost of repair.

[0163] For example, determining the repair cost of a candidate repair path includes: determining the operation cost based on the operation cost of repairing fields in the candidate repair path; determining the dependency impact cost based on the number of fields affected by the field repair in the candidate repair path; determining the constraint violation cost based on whether the candidate repair path violates business constraints; determining the business risk cost based on the business risk level of the repair operation corresponding to the candidate repair path; and determining the repair cost based on the operation cost, dependency impact cost, constraint violation cost, and business risk cost.

[0164] Specifically, the operational costs of fields in the candidate repair path are determined. Based on the correspondence between cost and price, the corresponding operational price is determined according to the operational cost. For example, the cost of each repair operation itself is as follows: modifying a simple field (such as quantity): price = 1.0; modifying an amount field: price = 3.0 (business-critical); deleting a field: price = 5.0 (high-risk); adding a field: price = 2.0. Then, based on the correspondence between quantity and dependency price, the dependency impact price is determined according to the number of fields affected by the field repair in the candidate repair path. For example, how many other fields will be affected by modifying this field: affecting 1 field: price + 1.0; affecting 3 fields: price + 3.0; affecting 5 fields: price + 5.0. Next, based on the correspondence between constraints and prices, the constraint violation price is determined according to whether the candidate repair path violates business constraints. For example, whether other constraints are violated during the repair process: no constraints violated: price + 0; temporarily violating 1 constraint: price + 5.0; violating core business constraints: price + 20.0. Next, based on the correspondence between business risks and costs, the business risk cost is determined according to the business risk level of the corresponding repair operation based on the candidate repair path. For example, the business risks of the repair operation are as follows: modifying the display field (name, description): low risk, cost +0; modifying the quantity / amount field: medium risk, cost +10.0; modifying the payment status field: high risk, cost +50.0.

[0165] Finally, the cost of remediation is obtained by adding up the operational costs, the costs of dependency impact, the costs of constraint violation, and the costs of business risks.

[0166] Furthermore, operational costs, dependency impact costs, constraint violation costs, and business risk costs can be linearly or non-linearly combined. For example, the repair cost J can be determined using the following formula:

[0167] J=α·C_op+β·C_dep+γ·C_con+δ·C_risk;

[0168] Where C_op represents the operation cost, C_dep represents the dependency impact cost, C_con represents the constraint violation cost, and C_risk represents the business risk cost; α, β, γ, and δ represent the weights of each cost, which can be configured offline and can be adaptively updated online based on historical verification results (e.g., weighted updates based on feedback or Bayesian estimation) to reduce the repair failure rate or rollback rate.

[0169] Understandably, determining the cost of repair from different dimensions facilitates the rapid identification of the target repair path.

[0170] For example, business constraints include at least one of the following: field value computation function dependency constraint, referenced object integrity constraint, data range constraint, business logic constraint, field consistency constraint, and field value uniqueness constraint. Among them, business logic constraints include state machine constraints.

[0171] The meaning of the field value calculation function dependency constraint is that the value of a certain field is calculated from other fields. Some examples: Constraint 1: Formula for calculating the amount of goods: items[*].amount=items[*].quantity×items[*].unitPrice, meaning: the amount of each item must equal the quantity multiplied by the unit price; Constraint 2: Formula for calculating the order subtotal: subtotal=sum(items[*].amount), meaning: the order subtotal must equal the sum of the amounts of all goods; Constraint 3: Formula for calculating the total price of the order: total=subtotal-discount+tax, meaning: the total price = subtotal - discount + tax; Constraint 4: Formula for calculating the tax: tax=subtotal×taxRate, meaning: the tax = subtotal × tax rate. Incorrect data: quantity=5, unitPrice=50.00, amount=100.00 ← Incorrect! It should be 250.00; System check: Expected value = 5 × 50.00 = 250.00, actual value = 100.00, conclusion: functional dependency constraint violated.

[0172] The so-called order reference integrity constraint means that the referenced object must exist. Some examples: Constraint 1: Product reference integrity, rule: items[*].productId∈productCatalog.keys, meaning: each product ID in the order must exist in the product catalog; Constraint 2: Customer reference integrity, rule: order.customerId∈customerDatabase.keys, meaning: the customer ID of the order must exist in the customer database; Constraint 3: Coupon reference integrity, rule: order.couponId∈couponList.keys, meaning: the coupon used must exist in the coupon list. Example of violation: Incorrect data: in the order: productId="P003", product catalog: only "P001", "P002"; system detection: query if "P003" exists in productCatalog, result: not found, conclusion: violation of reference integrity constraint (dangling reference).

[0173] Data range constraints mean that field values ​​must be within a reasonable range. Example constraints: Constraint 1: Quantity range, rule: quantity > 0 AND quantity <= 9999, explanation: The quantity of goods must be greater than 0 and not exceed 9999; Constraint 2: Price range, rule: price >= 0 AND price <= 999999.99, explanation: The price cannot be negative and cannot exceed a reasonable upper limit; Constraint 3: Discount range, rule: discount >= 0 AND discount <= subtotal, explanation: The discount cannot be negative and cannot exceed the order subtotal; Constraint 4: Tax rate range, rule: taxRate >= 0 AND taxRate <= 1.0, explanation: The tax rate must be between 0 and 100%. Example of violation: Incorrect data: quantity = -5 ← Error! Quantity cannot be negative; System check: Check quantity > 0, result: -5 is not greater than 0, conclusion: Data range constraint violated.

[0174] Business logic constraints are constraints that conform to actual business rules. Examples of constraints include: Constraint 1: Order Status Constraint, rule: If paymentStatus = "PAID", the order amount cannot be modified. Explanation: The amount of a paid order is locked and cannot be changed arbitrarily. Constraint 2: Inventory Constraint, rule: order.items[*].quantity <= product.stock. Explanation: The order quantity cannot exceed the product inventory. Constraint 3: Payment Consistency Constraint, rule: If paymentStatus = "PAID", then paidAmount = total. Explanation: The actual payment amount of a paid order must equal the total order price. Constraint 4: State Machine Constraint, rule: orderStatus can only flow in the order PENDING → CONFIRMED → PAID → SHIPPED → COMPLETED. Explanation: The order status cannot be reversed or jumped. Example of violation, error scenario: Order status: paymentStatus="PAID" (paid), actual amount paid: paidAmount=200.00, modification operation: change total to 500.00, system detection, check: if paid, the amount cannot be modified, result: attempted to modify the amount of a paid order, conclusion: violation of business logic constraint (operation prohibited).

[0175] Field consistency constraints mean that related fields must be consistent. Examples of constraints: Constraint 1: Currency consistency, rule: all amount fields must use the same currency. Explanation: the amount, tax, and discount in an order must use the same currency. Constraint 2: Precision consistency, rule: all amount fields must have two decimal places. Explanation: prices must maintain a precision of 0.01. Constraint 3: Time zone consistency, rule: all timestamp fields must use the UTC time zone. Explanation: to avoid time zone inconsistencies leading to time errors. Constraint 4: Unit consistency, rule: the weight field and weightUnit field must match. Explanation: the weight value and unit (kg / g) must correspond. Example of violation: Incorrect data: subtotal:{value:100.00,currency:"CNY"}, tax:{value:10.00,currency:"USD"} ← Error! Inconsistent currency. System detection: Checking the currency of all amount fields, it finds: CNY ≠ USD. Conclusion: Violation of consistency constraint (currency mismatch).

[0176] The so-called field value uniqueness constraint means that the values ​​of certain fields must be unique and cannot be repeated. Constraint examples: Constraint 1: Order ID uniqueness, rule: order.id must be unique in the system, explanation: each order must have a unique order number; Constraint 2: Product ID uniqueness, rule: items[*].id must be unique within the same order, explanation: product row IDs within an order cannot be repeated; Constraint 3: Email uniqueness, rule: customer.email must be unique in the customer database, explanation: one email address can only correspond to one customer account; Violation example: Error data: Order product list: [{"id":1,"productId":"P001"},{"id":1,"productId":"P002"} ← Error! id is duplicated] System detection: Checking if items[*].id is unique, it was found that id=1 appeared twice, conclusion: uniqueness constraint violated.

[0177] It should be noted that the timing of business constraint checks is shown in the table below:

[0178]

[0179] In addition, it should be noted that business constraints have priorities. When multiple constraints conflict, they are processed according to their priority. The constraint priorities are shown in the table below:

[0180]

[0181] It is understandable that by setting various constraints, data processing can be made more regulated, thereby improving data processing efficiency.

[0182] Furthermore, it should be noted that after the repair is completed, a repair report is generated, which includes repair statistics, repair details, and risk assessment results. For example, repair statistics: - Total number of repair operations: 2, - Total cost: 18.0 points, - Repair path length: 3 steps, - Expected success rate: 95%, - Manual confirmation required: No. Repair details: Operation 1: Repair items[0].amount, - Before modification: 100.00, - After modification: 250.00, - Reason: Function dependency constraint violation, - Cost: 15.0, - Scope of impact: subtotal, total; Operation 2: Repair total, - Before modification: 200.00, - After modification: 275.00, - Reason: Computation constraint violation, - Cost: 3.0, - Scope of impact: None. Risk assessment: - Risk level: Low, - Amount change: 200.00 → 275.00 (increase of 37.5%), - Audit required: Yes, - Recommended measures: Notify the administrator after automatically applying the patch. The information summary table is then output, as shown in the table below:

[0183]

[0184] The technical solution provided by this invention involves performing difference analysis on target data to obtain a difference map; identifying at least one conflicting piece of information based on the difference map; the conflicting information includes a conflicting field, a conflicting field type, and the corresponding field value; determining a field influence chain based on a semantically dependent directed acyclic graph and the changed field; determining the change topology order based on the field influence chain; determining the dependency influence degree of the conflicting information based on the conflicting field and the change topology order; processing at least one conflicting piece of information based on the dependency influence degree to obtain a conflict processing result; verifying the semantic integrity of the conflict processing result; and repairing the conflict processing result if the verification fails. This technical solution, along with the semantic integrity verification and repair of the conflict processing result, forms a complete closed-loop processing mechanism, ensuring the accuracy of conflict processing.

[0185] Figure 4 This is a flowchart of a data processing method provided by an embodiment of the present invention; based on the above embodiments, this embodiment elaborates on the method for determining semantically dependent directed acyclic graphs and provides an optional implementation scheme. For example... Figure 4 As shown, the method includes:

[0186] S410. Perform differential analysis on the target data to obtain a differential map.

[0187] The difference map includes the changed fields, the types of changed fields, and the related information of the changed fields;

[0188] S420. Determine at least one conflicting piece of information based on the difference map.

[0189] Conflict information includes the conflicting field, the type of the conflicting field, and the field value corresponding to the conflicting field;

[0190] S430. Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain.

[0191] S440. Determine the order of topology changes based on the field influence chain.

[0192] S450. Determine the dependency impact of conflict information based on the conflict field and the order of topology changes.

[0193] S460. Based on the degree of dependency, process at least one conflicting piece of information to obtain the conflict resolution result.

[0194] In one alternative approach, the semantic dependency directed acyclic graph is determined as follows: field labels are assigned to target fields; field labels have label confidence; historical target data is obtained; historical target data includes at least one of historical merge processing records, historical constraint verification results, and repair operation feedback data; the historical target data is dynamically analyzed to determine field relationships and update field labels; and a semantic dependency directed acyclic graph is constructed based on field relationships, target fields, field labels, and label confidence.

[0195] In this context, "target field" refers to the fields involved in the target. "Field label" refers to the functional label of the target field, used to describe its purpose. "Label confidence" refers to the degree of confidence of the field label. "Field relationship" refers to the dependency relationship between target fields. "Historical merged processing record" refers to the merged record of historical processing results of the target. "Historical constraint verification result" refers to the relevant results of constraint verification in the historical target. "Remediation operation feedback data" refers to the relevant remediation data of historical conflict handling results.

[0196] Specifically, firstly, in the initial case, field labels can be assigned to the target field. For example, field labels can be assigned to the target field based on a field name pattern library, where the field name pattern library includes the correspondence between fields and field labels. An example of a field name pattern library is:

[0197] Amount tags containing "price", "amount", "cost", and "fee";

[0198] Quantity labels containing "quantity", "count", or "num";

[0199] Status labels containing "status" or "state";

[0200] → Identifier tags containing "id" and "key";

[0201] Time tags containing "date", "time", or "at".

[0202] For specific examples, when "unitPrice" appears in an order and "price" is detected, an amount label is automatically assigned; when "productId" appears in an order and "id" is detected, an identifier label is automatically assigned, and so on.

[0203] For example, the field label of a target field can be inferred based on its data type. The data type inference rules are as follows:

[0204] Data type = Number + value with decimal point → could be an amount;

[0205] Data type = Number + value is an integer → could be quantity or ID;

[0206] Data type = String + value is an enumeration (PENDING / PAID, etc.) → could be a state;

[0207] Data type = String + value conforming to date format → possibly time.

[0208] For example, if the order contains price: 100.50, and the data type is detected as Number with a decimal point, then the price field is assigned an amount label with a confidence level of 80%. If the order contains quantity: 5, and the data type is detected as Number with an integer, then the quantity field is assigned a quantity label with a confidence level of 60%.

[0209] For example, field labels can be assigned to order fields based on business rules, and the field labels can be obtained in response to the administrator's label configuration operation for order fields.

[0210] During system operation, historical target data is acquired and dynamically analyzed to determine field relationships and update field labels. For example, historical merge processing records show which fields are frequently modified together (indicating a relationship), which fields need to be recalculated after modifying a certain field (indicating a dependency), and which fields have a high conflict rate (indicating business importance). For example, the system finds that in the past 1000 merges, in 90% of cases, modifying "quantity" also changes "amount" and "total". → The system infers that "amount" may depend on "quantity", automatically adds a calculation label to "amount", and establishes a dependency relationship: quantity → amount. For example, historical constraint validation results record which fields frequently fail validation (indicating rule complexity), which constraints are frequently triggered (indicating core constraints), and which fields are frequently modified during repairs (indicating key fields). An example is as follows: The system discovered that in the past 500 validations, the "total" field failed 50 times, with the reason being "total ≠ subtotal + tax". → The system infers that "total" is a calculated field, dependent on "subtotal" and "tax", and automatically adds a calculation label and an amount label to "total", increasing the business importance weight of the "total" field. For example, the feedback data on repair operations statistically analyzes which repair operations had high success rates (indicating the strategy was correct), which fields had high repair costs (indicating the wide impact of the modification), and which solution was chosen during manual intervention (indicating human judgment tendencies). Learning example: The system discovered that in the past 200 "price" field conflicts, manual intervention occurred 100 times, and the "Right" version was chosen 85 times, with the reason being "latest price priority." The system inferred that the "price" field should adopt a "latest priority" strategy, updating the decision rule to "amount tag + timestamp comparison." The next time a price conflict occurs, the later version will be automatically selected.

[0211] Finally, based on the field relationships, target field, field labels, and label confidence, a semantic dependency directed acyclic graph is constructed.

[0212] It is understandable that by establishing a semantic dependency directed acyclic graph to represent the relationships between target fields, the handling of target conflicts can be made more efficient and reasonable.

[0213] For example, historical target data can be dynamically analyzed and field labels updated through at least one of the following methods:

[0214] Analyze the frequency of field usage in historical target data and adjust the label confidence of field labels accordingly;

[0215] If a new usage pattern of the target field is identified in the historical target data, the new field label is determined and the new field label is automatically added to the target field;

[0216] If a change in the field structure of a target field in historical target data is detected, the field label migration is performed based on the new field structure.

[0217] Specifically, the frequency of field usage in historical target data is analyzed, and the confidence level of field labels is adjusted accordingly. It's important to note that label confidence is configurable and learnable. For example, initially assigned labels have confidence scores: based on field name: 60% confidence; based on data type: 50% confidence; based on business configuration: 100% confidence. As usage increases, the confidence is dynamically adjusted: if the prediction is correct (e.g., the field is indeed an amount), confidence increases by 5%; if the prediction is incorrect (e.g., the field is actually a quantity), confidence decreases by 10%. Confidence > 90%: the label is stable and no longer frequently adjusted; confidence < 30%: the label is removed and relearned. Example: Field: “orderAmount”, initial label: Amount label (60% confidence); after 100 merges: 90 involved amount calculations → confidence + 45% = 105% (locked at 100%), final label: Amount label (100% confidence) → stable.

[0218] If the new usage pattern of the order field in the historical order data is identified, the new field label is determined and automatically added to the order field. The discovery process is as follows: The system notices the field "items[*].productId" and finds that the value of this field must exist in "productCatalog". Statistical analysis shows that 100% of the validation failures are due to the absence of productId, leading to the inference that it is a reference field. Label expansion is performed: Initially, an identifier label is used; after evolution, an identifier label, a reference label, and a required label are used. Dependency relationships are established: items[*].productId → referential integrity constraint → productCatalog.

[0219] If a change in the field structure of the order field in historical order data is detected, the field label migration will be performed based on the new field structure. Scenario: Order structure reconstruction, old structure:

[0220] {"price":100.00,

[0221] "currency":"CNY"}.

[0222] Tag: price → [Amount Tag];

[0223] New structure:

[0224] {"pricing":{

[0225] "amount":100.00,

[0226] "currency":"CNY"}}.

[0227] Tag migration: The system detected the disappearance of the "price" field and detected a new field "pricing.amount". Comparing the value and type (both are 100.00), it is determined that this is a field renaming / moving. Migrate tag: pricing.amount ← [Amount tag] (inherited from price), inheritance confidence: 100% (from the original field).

[0228] It is understandable that automatically updating field labels can make them more accurate, thus making conflict resolution more reasonable; at the same time, it can also enhance the adaptability of field labels.

[0229] Based on the above embodiments, combined with Figure 5 This invention provides an object-level alignment process based on optimal matching, as detailed below:

[0230] S510, Construct a three-way difference map.

[0231] Specifically, a three-way difference analysis is performed on the baseline version (Base), the first modified version (Left), and the second modified version (Right) of the target data to obtain a difference map, i.e., a three-way difference map. For example, analyzing Base yields the difference on the left (Left), and analyzing Base yields the difference on the right (Right).

[0232] S520. Obtain the topological sort of the DAG based on the semantic dependency DAG.

[0233] Specifically, obtain the semantic dependency DAG, calculate the topological sorting sequence, and ensure the dependency processing order, i.e., change the topological order.

[0234] S530. Perform conflict detection according to topological sorting to obtain the conflict detection results.

[0235] Specifically, each field path is traversed. If a conflict is found, semantic conflict detection is performed, the dependency propagation impact (i.e., the dependency impact of the conflicting information) is calculated, and the conflicting information is classified and its features are extracted before proceeding to the next field. If there is no conflict, the next field is then detected.

[0236] S540. Perform four-level decision processing on the conflict detection results to obtain the decision processing results.

[0237] Specifically, after the conflict traversal detection is completed, the conflict detection results are processed in four levels: first, field-level matching is performed; then, object-level alignment is performed; next, constraint reasoning is performed; and finally, global consistency is performed to obtain the decision processing result.

[0238] S550: Intelligently merge the decision processing results to obtain preliminary merge results.

[0239] Specifically, the decision processing results are intelligently merged using a merging strategy to obtain the merged processing result.

[0240] S560. Based on the preliminary merging results, perform closed-loop verification and repair.

[0241] Based on the above embodiments, combined with Figure 6 This invention also provides a heuristic minimum-cost repair path search process. The specific procedures for closed-loop verification and repair based on preliminary merging results are described in detail. For example... Figure 6 As shown, the path search process is as follows:

[0242] S601, Initialize the number of repair attempts.

[0243] Specifically, the initial number of repetitions is set to 0, and the maximum number of repetitions is 5.

[0244] S602. Perform semantic integrity verification on the initial merged result and determine whether the verification passes. If the verification passes, proceed to S603; if the verification fails, proceed to S604.

[0245] S603. Returns a successful semantic verification result.

[0246] S604. Locate and analyze the failure nodes, and construct a failure dependency graph.

[0247] Specifically, the failure nodes are identified from the conflict resolution results; the dependency path is backtracked based on the failure nodes and the directed acyclic graph of semantic dependencies to construct the failure dependency graph.

[0248] S605. Use the A* algorithm to search for the minimum repair path based on the failure dependency graph.

[0249] Specifically, a heuristic search or an equivalent minimum cost path algorithm (such as A*) is used to search for the repair path.

[0250] S606. Determine if the target repair path has been found. If yes, proceed to S607; otherwise, proceed to S610.

[0251] S607, Generate a repair patch.

[0252] Specifically, it calculates the repair operations and generates repair patches in JSON Patch format.

[0253] S608, apply the repair patch to fix it.

[0254] Specifically, a patch is applied to fix the initial merge results.

[0255] S609, Increase the number of repetitions by 1, and execute S611.

[0256] S610, upgrade to manual intervention and return the manual intervention result.

[0257] S611. Determine if the number of repetitions is less than the maximum number of repetitions. If yes, return to execute S602; otherwise, execute S612.

[0258] S612. The repair failed, and an error message was returned.

[0259] Figure 7 This is a schematic diagram of a data processing device according to an embodiment of the present invention. This embodiment is applicable to situations involving the processing of conflicting data in a target scenario, such as an e-commerce scenario. The data processing device can be implemented in hardware and / or software, and can be configured in an electronic device that carries data processing functions, such as a server. Figure 7 As shown, the device includes:

[0260] The difference map determination module 710 is used to perform difference analysis on the target data and obtain a difference map; the difference map includes the changed fields, the types of changed fields, and the association information of the changed fields.

[0261] The conflict information determination module 720 is used to determine at least one conflict information based on the difference map; the conflict information includes a conflict field, a conflict field type, and a field value corresponding to the conflict field;

[0262] The Field Influence Chain Determination Module 730 is used to determine the field influence chain based on the semantic dependency directed acyclic graph and the changed field.

[0263] The topology order determination module 740 is used to determine the order of topology changes based on the field influence chain;

[0264] The dependency impact determination module 750 is used to determine the dependency impact of conflict information based on the conflict field and the order of topology changes.

[0265] The conflict resolution result determination module 760 is used to process at least one conflict information according to the degree of dependency impact to obtain the conflict resolution result.

[0266] The technical solution of this invention involves performing difference analysis on target data to obtain a difference map; identifying at least one conflicting piece of information based on the difference map; determining field influence chains based on a semantic dependency directed acyclic graph and changed fields; determining the change topology order based on the field influence chains; determining the dependency influence degree of the conflicting information based on the conflicting fields and the change topology order; and processing at least one conflicting piece of information based on the dependency influence degree to obtain a conflict resolution result. This technical solution, combining a difference map and a semantic dependency directed acyclic graph to identify and process conflicting data in the target data, can improve conflict resolution efficiency.

[0267] Optional, the dependency impact determination module 750 is used for:

[0268] Based on the field's influence chain, determine the influence dimension; the influence dimension includes the number of fields affected, chain depth, and field weight bonus;

[0269] The degree of influence of a dimension is determined based on the dimension of influence and the dimension coefficient.

[0270] Determine the conflict field type and the corresponding conflict coefficient for the conflict field type;

[0271] Determine the degree of dependence based on the dimensional influence and the conflict coefficient.

[0272] Optionally, the conflict resolution result determination module 760 is used for:

[0273] Based on the degree of dependence and impact, determine the processing priority of at least one piece of conflict information;

[0274] Based on the processing priority, at least one conflicting piece of information is processed to obtain the conflict resolution result.

[0275] Optionally, the conflict resolution result determination module 760 is used for:

[0276] Determine the type of conflict in the conflict information;

[0277] The conflict information is processed according to the handling method corresponding to the conflict type to obtain the conflict resolution result.

[0278] Optionally, the conflict resolution result determination module 760 is specifically used for:

[0279] If the conflict type is a complex conflict, the conflict information is processed according to the field influence chain and the order of topology changes to obtain the conflict resolution result.

[0280] Optionally, if the conflict information includes arrays and / or objects, then the conflict resolution result determination module 760 is specifically used for:

[0281] If the conflict type includes special conflicts, the conflict information is intelligently merged based on array merging strategy and / or nested object merging strategy to obtain the conflict resolution result.

[0282] Optionally, the device also includes a verification and repair module for:

[0283] Process at least one conflicting piece of information, obtain the conflict processing result, and then perform semantic integrity verification on the conflict processing result;

[0284] If verification fails, the conflict resolution result will be corrected.

[0285] Optional, a verification and repair module is used for:

[0286] Identify the failed nodes from the conflict resolution results;

[0287] Dependency path backtracking is performed based on the failed nodes and the directed acyclic graph of semantic dependencies to construct a failed dependency graph;

[0288] Search for repair paths based on the failure dependency graph to determine at least one candidate repair path;

[0289] Determine the repair cost of candidate repair paths;

[0290] The target repair path is determined from at least one candidate repair path based on the repair cost;

[0291] The conflict resolution results are repaired according to the target repair path.

[0292] Optional, a verification and repair module is used for:

[0293] The operation cost is determined based on the operation cost of repairing fields in the candidate repair paths;

[0294] The cost of dependency impact is determined based on the number of fields affected by the field repair in the candidate repair path;

[0295] Determine the cost of constraint violation based on whether the candidate repair path violates business constraints;

[0296] The business risk cost is determined based on the degree of business risk of the corresponding repair operation for the candidate repair path;

[0297] The cost of remediation is determined based on operational costs, dependency impact costs, constraint violation costs, and business risk costs.

[0298] Optionally, business constraints may include at least one of the following: field value calculation function dependency constraint, target reference object integrity constraint, data range constraint, business logic constraint, field consistency constraint, and field value uniqueness constraint.

[0299] Optionally, the apparatus further includes a semantically dependent directed acyclic graph determination module for:

[0300] Assign field labels to the target field; field labels have label confidence levels;

[0301] Acquire historical target data; historical target data includes at least one of the following: historical merge processing records, historical constraint verification results, and repair operation feedback data.

[0302] Perform dynamic analysis on historical target data to determine field relationships and update field labels;

[0303] Construct a semantic dependency directed acyclic graph based on field relationships, target field, field labels, and label confidence.

[0304] Optionally, the apparatus further includes a semantically dependent directed acyclic graph determination module for:

[0305] Dynamically analyze historical target data and update field labels using at least one of the following methods:

[0306] Analyze the frequency of field usage in historical target data and adjust the label confidence of field labels accordingly;

[0307] If a new usage pattern of the target field is identified in the historical target data, the new field label is determined and the new field label is automatically added to the target field;

[0308] If a change in the field structure of a target field in historical target data is detected, the field label migration is performed based on the new field structure.

[0309] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0310] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0311] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the data processing method of the present invention. Figure 8A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0312] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0313] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0314] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.

[0315] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0316] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0317] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0318] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0319] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0320] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0321] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0322] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0323] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Perform difference analysis on the target data to obtain a difference map; the difference map includes the changed fields, the types of the changed fields, and the association information of the changed fields. At least one conflict information is determined based on the difference map; The conflict information includes the conflict field, the conflict field type, and the field value corresponding to the conflict field; Based on the semantic dependency directed acyclic graph and the changed fields, determine the field influence chain; Determine the order of topology changes based on the impact chain of the aforementioned fields; The dependency impact of the conflict information is determined based on the conflict field and the changed topology order; Based on the degree of dependency influence, the at least one conflict information is processed to obtain a conflict resolution result.

2. The method according to claim 1, characterized in that, Determining the dependency impact of the conflict information based on the conflict field and the changed topology order includes: The influence dimension is determined based on the field influence chain; the influence dimension includes the number of influencing fields, chain depth, and field weight bonus. The degree of influence of a dimension is determined based on the aforementioned influence dimensions and dimension coefficients; Determine the conflict field type and the conflict coefficient corresponding to the conflict field type; Determine the dependency influence degree based on the aforementioned dimensional influence degree and conflict coefficient.

3. The method according to claim 1, characterized in that, Based on the dependency impact degree, the at least one conflict information is processed to obtain a conflict resolution result, including: Based on the degree of dependency impact, the processing priority of the at least one conflict information is determined respectively; According to the processing priority, the at least one conflict information is processed to obtain a conflict resolution result.

4. The method according to claim 1 or 3, characterized in that, Processing the at least one conflicting piece of information to obtain a conflict resolution result includes: Determine the conflict type of the conflict information; The conflict information is processed according to the processing method corresponding to the conflict type to obtain the conflict processing result.

5. The method according to claim 4, characterized in that, The conflict information is processed according to the processing method corresponding to the conflict type to obtain the conflict processing result, including: If the conflict type is a complex conflict, then the conflict information is processed according to the change topology order based on the field influence chain to obtain the conflict processing result.

6. The method according to claim 4, characterized in that, If the conflict information includes arrays and / or objects, then the conflict information is processed according to the processing method corresponding to the conflict type to obtain the conflict processing result, including: If the conflict type includes a special conflict, the conflict information is intelligently merged based on an array merging strategy and / or a nested object merging strategy to obtain the conflict resolution result.

7. The method according to claim 1, characterized in that, After processing the at least one conflict information to obtain the conflict resolution result, the process further includes: The semantic integrity of the conflict resolution result is verified. If the verification fails, the conflict resolution result will be repaired.

8. The method according to claim 7, characterized in that, Repairing the conflict resolution result includes: The failed node is determined from the conflict resolution results; Dependency path backtracking is performed based on the failed nodes and the directed acyclic graph of semantic dependencies to construct a failed dependency graph; Based on the failure dependency graph, a repair path search is performed to determine at least one candidate repair path; Determine the repair cost of the candidate repair paths; The target repair path is determined from at least one candidate repair path based on the repair cost; The conflict resolution results are repaired according to the target repair path.

9. The method according to claim 8, characterized in that, Determining the repair cost of the candidate repair path includes: The operation cost is determined based on the operation cost of repairing the fields in the candidate repair paths; The dependency impact cost is determined based on the number of fields affected by the field repair in the candidate repair path; The cost of violating the constraints is determined based on whether the candidate repair paths violate business constraints. The business risk cost is determined based on the business risk level of the repair operation corresponding to the candidate repair path; The repair cost is determined based on the operational cost, the dependency impact cost, the constraint violation cost, and the business risk cost.

10. The method according to claim 9, characterized in that, The business constraints include at least one of the following: field value calculation function dependency constraint, reference object integrity constraint, data range constraint, business logic constraint, field consistency constraint, and field value uniqueness constraint.

11. The method according to claim 1, characterized in that, The semantic dependency directed acyclic graph is determined in the following way: Assign field labels to the target field; the field labels have label confidence levels; Acquire historical target data; the historical target data includes at least one of the following: historical merge processing records, historical constraint verification results, and repair operation feedback data. The historical target data is dynamically analyzed to determine the field relationships and update the field labels; Based on the field relationships, the target field, the field labels, and the label confidence, a semantic dependency directed acyclic graph is constructed.

12. The method according to claim 11, characterized in that, Dynamically analyzing the historical target data and updating the field labels using at least one of the following methods includes: Analyze the frequency of use of fields in the historical target data and adjust the label confidence of the field labels accordingly; If a new usage pattern of a target field is identified in the historical target data, then a new field label is determined, and the new field label is automatically added to the target field. If a change in the field structure of the target field in the historical target data is detected, the field label migration is performed according to the new field structure.

13. A data processing apparatus, characterized in that, include: The difference map determination module is used to perform difference analysis on the target data and obtain the difference map. The difference map includes the changed fields, the types of the changed fields, and the associated information of the changed fields; A conflict information determination module is used to determine at least one conflict information based on the difference map. The conflict information includes the conflict field, the conflict field type, and the field value corresponding to the conflict field; The field influence chain determination module is used to determine the field influence chain based on the semantic dependency directed acyclic graph and the changed field; The topology order determination module is used to determine the order of topology changes based on the influence chain of the field. The dependency impact determination module is used to determine the dependency impact of the conflict information based on the conflict field and the changed topology order. The conflict resolution result determination module is used to process the at least one conflict information according to the dependency impact degree to obtain the conflict resolution result.

14. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-12.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-12.

Images