Data recovery method and related equipment

Abstract

The embodiment of the invention discloses a data recovery method and related equipment, which are used for carrying out data recovery under the condition of improving the accuracy, comprehensiveness and efficiency of data recovery. The method comprises the steps that when field values of fields with the same meaning in a source table and a target table are inconsistent, a data restoration trigger signal initiated by a user or automatically triggered by a system is obtained, and a target source field, corresponding to the data restoration trigger signal, in the target source table is determined, and determining at least one target associated field corresponding to the target source field, and determining and executing a recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field so as to perform corresponding data recovery on the target source field and the at least one target associated field based on the recovery data of the target source field.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of data repair, and more particularly, to a data repair method, a data repair apparatus, a data repair device, a computer readable storage medium, and a computer program product containing instructions. BACKGROUND

[0002] With the increase of data volume and the complexity of business requirements, the demand for data repair is rising.

[0003] The existing data repair method is to directly modify the data in the database by writing SQL scripts, that is, a developer or a database administrator uses a structured query language (SQL) to write a series of commands or scripts that can directly modify the data stored in the database. For example, the position information of an employee may be stored in multiple tables, when the position information in one of the tables is updated, the developer must manually write SQL scripts to update the values of all associated fields.

[0004] However, the determination and re-repair of each associated field are determined by the developer one by one, which is easy to miss some associated fields that should be repaired, and the accuracy, comprehensiveness and efficiency of determining the associated fields are low, and the developer needs to manually determine the associated fields one by one and write SQL scripts to modify the field values of the associated fields, so the accuracy, comprehensiveness and efficiency of data repair are low. SUMMARY

[0005] Embodiments of the present application provide a data repair method, a data repair apparatus, a data repair device, a computer readable storage medium, and a computer program product containing instructions, for improving the accuracy, comprehensiveness and efficiency of data repair.

[0006] In a first aspect, embodiments of the present application provide a data repair method, comprising:

[0007] When the field values of the same meaning fields in the source table and the target table are inconsistent, a data repair trigger signal initiated by a user or automatically triggered by a system is obtained;

[0008] A target source field in the target source table corresponding to the data repair trigger signal is determined, and at least one target associated field corresponding to the target source field is determined, the at least one target associated field including at least one target redundant field in each target table having a preset mapping relationship with the target field value of the target source field, or at least one target redundant field and at least one target redundant source field;

[0009] The data recovery unit determines and executes a recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field, so as to perform corresponding data recovery on the target source field and the at least one target associated field based on the recovery data of the target source field.

[0010] In a second aspect, the embodiments of the present application provide a data repair device, comprising:

[0011] The acquisition unit is configured to acquire a data repair trigger signal initiated by a user or triggered automatically by a system when field values of the same meaning fields in the source table and the target table are inconsistent.

[0012] The determination unit is configured to determine a target source field in the target source table corresponding to the data repair trigger signal, and determine at least one target associated field corresponding to the target source field, the at least one target associated field comprising at least one target redundant field or at least one target redundant field and at least one target redundant source field in each target table, the field values of which have a preset mapping relationship with a target field value of the target source field.

[0013] The data recovery unit is configured to determine and execute a recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field, so as to perform corresponding data recovery on the target source field and the at least one target associated field based on the recovery data of the target source field.

[0014] In a third aspect, the embodiments of the present application provide a data repair device, comprising:

[0015] The central processor, the memory, the input and output interface, the wired or wireless network interface and the power supply;

[0016] The memory is a volatile memory or a persistent memory;

[0017] The central processor is configured to communicate with the memory and execute instruction operations in the memory to perform the foregoing data repair method.

[0018] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which comprises instructions, when the instructions are run on a computer, causing the computer to execute the foregoing data repair method.

[0019] In a fifth aspect, the embodiments of the present application provide a computer program product comprising instructions, when the computer program product is run on a computer, causing the computer to execute the foregoing data repair method.

[0020] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: They can automatically determine the target source field in the target source table corresponding to the data repair trigger signal, and automatically determine at least one target associated field corresponding to the target source field; that is, they can automatically determine all associated fields related to the source field, including redundant fields and redundant source fields. Then, based on the recovered data of the target source field, the data recovery task for the target source field and its associated fields is automatically executed. By identifying all associated fields that have a mapping relationship with the source field, the system can comprehensively understand the dependency relationship between the source field and associated fields, which can improve the accuracy and comprehensiveness of determining all associated fields of the source field. This ensures that when restoring the source field, all associated fields can be correctly and automatically restored automatically, without the need for manual determination of associated fields one by one, avoiding the omission of some associated fields that should be repaired, and eliminating the need for manual writing of SQL scripts to modify the field values ​​of associated fields. The accuracy, comprehensiveness, and efficiency of data repair are high.

[0021] Accordingly, the data repair apparatus, data repair equipment, computer-readable storage medium, and computer program product containing instructions provided in this application also have the aforementioned technical effects. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of a data repair system disclosed in an embodiment of this application;

[0023] Figure 2-1 This is a flowchart illustrating a data repair method disclosed in an embodiment of this application;

[0024] Figure 2-2 This is a schematic diagram illustrating redundant relationships in a data lineage definition disclosed in an embodiment of this application;

[0025] Figure 2-3 This is a schematic diagram illustrating the relationship between a source field, a redundant field, and a redundant source field as disclosed in an embodiment of this application.

[0026] Figure 2-4 This is a schematic diagram illustrating an example of a source field, a redundant field, and the relationship between redundant source fields disclosed in an embodiment of this application.

[0027] Figure 2-5 This is a schematic diagram illustrating the execution process of a data repair task as disclosed in an embodiment of this application;

[0028] Figure 2-6 This is a schematic diagram illustrating a task splitting method disclosed in an embodiment of this application;

[0029] Figure 2-7 This is a flowchart illustrating a business verification logic disclosed in an embodiment of this application;

[0030] Figure 2-8 This is a schematic diagram of the architecture of another data repair system disclosed in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of the structure of a data repair device disclosed in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of a data repair device disclosed in an embodiment of this application. Detailed Implementation

[0033] This application provides a data repair method, a data repair apparatus, a data repair device, a computer-readable storage medium, and a computer program product containing instructions, for performing data repair while improving the accuracy, comprehensiveness, and efficiency of data repair.

[0034] Please see Figure 1 The architecture of the data repair system in this application embodiment includes:

[0035] Data repair device 101 and client 102. When performing data repair, data repair device 101 can connect to client 102. Data repair device 101 can receive a data repair trigger signal sent by client 102, determine the target source field in the target source table corresponding to the data repair trigger signal, determine at least one target associated field corresponding to the target source field, and determine and execute the recovery task for the target source field and at least one target associated field based on the recovered data of the target source field. This allows for the corresponding data recovery of the target source field and at least one target associated field, and the recovery result is returned to client 102.

[0036] based on Figure 1 Please refer to the data recovery system shown. Figure 2-1 , Figure 2-1 This is a flowchart illustrating a data repair method disclosed in an embodiment of this application. The method includes:

[0037] 201. When the values ​​of fields with the same meaning in the source table and the target table are inconsistent, obtain the data repair trigger signal initiated by the user or automatically triggered by the system.

[0038] In one alternative implementation, the source table refers to a database table containing original or source data. During the data repair process of this application, the source table is the starting point for data changes, and all related updates are based on the data in the source table. The target table refers to other tables that depend on the data in the source table; these tables may contain redundant data with the same fields as the source table. During the data repair process of this application, the target table needs to be updated accordingly based on the updated data in the source table.

[0039] 202. Determine the target source field in the target source table corresponding to the data repair trigger signal, and determine at least one target associated field corresponding to the target source field. The at least one target associated field includes at least one target redundant field in each target table where the field value and the target field value of the target source field have a preset mapping relationship, or at least one target redundant field and at least one target redundant source field.

[0040] In one alternative implementation, a target source field refers to a field in the source table that is depended upon by other fields. A target related field refers to a field that has a direct or indirect relationship with the target source field; these fields are stored in the target table, but their values ​​depend on the values ​​of the target source field. A target redundant source field refers to a field in the target table that can be depended upon by other fields and also depends on other fields. A target redundant field refers to a field in the target table that depends on other fields. For details, please refer to [link to relevant documentation]. Figure 2-2 , Figure 2-2 This is a schematic diagram illustrating redundant relationships in a data lineage definition (data lineage defines the dependency and flow relationships between data) disclosed in an embodiment of this application. Figure 2-2 It can be seen that maintaining strong consistency for redundant fields is as follows: Vertical relationships (field redundancy) indicate that field B in table A is the source field, while field G in table B is a redundant field, meaning field G depends on field B. Horizontal relationships (data row relationship mapping) indicate that the value of field A in table A is equal to the value of field H in table B, indicating that these two rows of data are logically related. This definition of lineage helps the system identify and repair all related fields that depend on the source field when data inconsistencies occur, ensuring data consistency. For more details, please refer to [link to relevant documentation]. Figure 2-3 , Figure 2-3 This is a schematic diagram illustrating the relationship between a source field, a redundant field, and a redundant source field as disclosed in an embodiment of this application. Figure 2-3 As can be seen, a source field can be depended on by a redundant field, and vice versa. Please refer to [link to relevant documentation]. Figure 2-4 , Figure 2-4 This is a schematic diagram illustrating an example of a source field, a redundant field, and the relationship between redundant source fields disclosed in an embodiment of this application. Figure 2-4 As can be seen, HR position (organization) is the source field, work experience (administrative organization) is a redundant source field that depends on HR position (organization), attendance personnel work experience (administrative organization) and attendance file (administrative organization) are both redundant fields that depend on work experience (administrative organization), and organizer (administrative organization) and work experience summary table (administrative organization) are both redundant fields that depend on HR position (organization).

[0041] It is understandable that the lineage relationship between each source field can be constructed based on the dependencies between various fields (source fields, redundant fields, and redundant source fields) and the mapping between data rows (the system can identify logically related rows in different tables; for example, if the value of a field in table A is equal to the value of a field in table B, then these two rows of data are logically related). It is also understandable that the mapping relationship can be based not only on equal field values ​​(such as the employee ID in the source table being equal to the employee ID in the target table), but also on logical or business rules (for example, if an employee's job title is updated to "Manager" in the employee information table, then the employee's salary grade in the salary table may automatically be updated to "Senior"), or other reasonable mapping relationships; specific details are not limited here.

[0042] It's worth noting that by clearly defining the vertical (field redundancy) and horizontal (data row relationship mapping) relationships of the data and establishing data transfer standards, the matching of data relationships is ensured, and data points requiring repair are identified. Furthermore, modification is only permitted on the source data, thereby reducing errors in database table relationships. This approach mitigates the risk of data inconsistency caused by arbitrary modifications from the outset, ensuring data quality.

[0043] 203. Based on the recovery data of the target source field, determine and execute the recovery task of the target source field and at least one target associated field, so as to perform corresponding data recovery on the target source field and at least one target associated field based on the recovery data of the target source field.

[0044] In one alternative implementation, performing a recovery task ensures that all related fields maintain consistency with their values ​​in the source tables. For example, suppose in a company database system, employee information is stored in table A, while employee job information may also be stored in several other tables, such as table B1 (salary table) and table B2 (report table). When the employee job field in table A is updated, the system detects inconsistencies in the corresponding field values ​​in tables B1 (salary table) and B2 (report table). The system determines that the job field in table A is the source field, and the job fields in tables B1 (salary table) and B2 (report table) are the target related fields. The system performs a recovery task, using the latest job information from table A to update the job fields in tables B1 (salary table) and B2 (report table), ensuring consistency of job information across all tables.

[0045] This approach automatically identifies the target source field in the target source table corresponding to the data repair trigger signal, and automatically determines at least one target related field corresponding to the target source field. In other words, it automatically identifies all related fields associated with the source field, including redundant fields and redundant source fields. Then, based on the recovered data of the target source field, it automatically executes data recovery tasks for the target source field and its related fields. By identifying all related fields that map to the source field, the system can comprehensively understand the dependencies between the source field and related fields, improving the accuracy and comprehensiveness of identifying all related fields of the source field. This ensures that when restoring the source field, all related fields can be correctly and automatically restored automatically, eliminating the need for manual identification of each related field. This avoids missing any related fields that should be repaired and eliminates the need for manually writing SQL scripts to modify the values ​​of related fields, resulting in high accuracy, comprehensiveness, and efficiency in data repair.

[0046] In an optional implementation, before determining at least one target associated field corresponding to the target source field, the method further includes: determining each source table and each source field of each source table; for each source field, determining at least one redundant field, or at least one redundant field and at least one redundant source field, in each target table where the field value has a preset mapping relationship with the field value of the source field; and taking at least one redundant field, or at least one redundant field and at least one redundant source field, corresponding to each source field as at least one associated field corresponding to each source field.

[0047] Specifically, it can identify all source tables and their source fields. For each source field, at least one redundant field is identified in the target table. The value of this redundant field has a predefined mapping relationship with the value of the source field. The redundant field can be a copy of the source field in another table or a related field. Before data repair, the system will determine at least one redundant field or redundant source field corresponding to each source field. These fields will be used as related fields of the source field.

[0048] By identifying at least one related field for each source field in advance, these related fields can be quickly and accurately located when data repair is needed. This method reduces the need to repeatedly identify related fields during data repair, thereby saving time and improving the overall efficiency of data repair.

[0049] In one optional implementation, performing a recovery task on the target source field and at least one target associated field includes: determining transactional and non-transactional tasks based on the recovery task on the target source field and at least one target associated field, performing transactional tasks in parallel through an object-relational mapping framework, and calling the corresponding business microservices to perform non-transactional tasks.

[0050] Specifically, transactional tasks refer to tasks that need to be executed under database transaction control. It's a mechanism to ensure data consistency and integrity, guaranteeing that a set of operations either all succeed or all fail, leaving no intermediate states. In database operations, transactions ensure that multiple steps either all complete or are rolled back if an error occurs, thus avoiding data inconsistency. Non-transactional tasks, in contrast to transactional tasks, do not require strict atomicity guarantees. They involve more complex business logic or require cross-service or cross-database operations, which are not suitable for traditional transaction management. Non-transactional tasks can be executed asynchronously, do not require immediate completion, and do not enforce a strict order between operations. An Object-Relational Mapping Framework (ORM) is a technology used to map programming object models to database tables. ORM frameworks allow developers to manipulate databases in an object-oriented way without writing large amounts of SQL statements. ORM frameworks are responsible for transforming data between object models and database tables, simplifying database operations. In a microservice architecture, a business microservice refers to a small, independent service unit that implements specific business functions. Each microservice typically handles a portion of the functionality and can be deployed and scaled independently. During the data repair process, business microservices are used to handle complex logic or business rules that are not suitable for direct database transaction processing.

[0051] For more details, please refer to Figure 2-5 , Figure 2-5 This is a schematic diagram illustrating the execution flow of a data repair task disclosed in an embodiment of this application. Figure 2-5 As can be seen, the core of the data repair task execution process is to perform corresponding data update operations based on the task type. The specific steps include: The process is initiated to begin the data repair task; then, the system breaks down complex tasks into multiple smaller, independent task units based on data lineage to ensure orderly operations and data consistency. The broken-down repair tasks are executed, and it is determined whether the task is a transaction. If it is a transactional task, the database entity objects are directly updated through the ORM framework; if it is not a transactional task or contains complex business logic, the business microservice is called for processing. Detailed logs of task execution are recorded, and it is checked whether there are any subsequent tasks. If so, execution continues until all tasks are completed. Finally, the process ends, marking the successful completion of the data repair task.

[0052] Understandably, the method of splitting small tasks into batches can be replaced by pipelined processing, which breaks down large-scale data repair tasks into multiple sequential stages (such as splitting, verification, repair, and review), with each stage processing a batch of data, thereby improving process efficiency.

[0053] In this way, by distinguishing between transactional and non-transactional tasks, data update operations can be managed more precisely, ensuring the atomicity of critical data operations and optimizing resource utilization through parallel processing. Secondly, for ORM-based redundant data repair, this application differentiates between cross-database and non-cross-database tasks, resolving the issue of transactional consistency during cross-database redundant data repair. Finally, in large-scale data scenarios, transactional data repair adopts a strategy of splitting data into smaller transactions in batches. It clearly defines the data volume division principles for each batch of smaller transactions (e.g., by fixed size or business logic units) and how to ensure the coherence and consistency checks of the overall data after batch processing. This avoids system lock waiting and performance bottlenecks caused by long transactions with large data volumes, improving transaction commit efficiency and system reliability.

[0054] In one optional implementation, the target source field and at least one target associated field are marked with corresponding database identifiers. The recovery task based on the target source field and at least one target associated field determines the transactional and non-transactional tasks, including: merging multiple recovery tasks of the same database to generate transactional tasks based on the database identifiers corresponding to the target source field and at least one target associated field, and generating non-transactional tasks separately for recovery tasks across databases and / or recovery tasks corresponding to complex repair operations.

[0055] Specifically, a database identifier is a unique identifier used to distinguish different database instances or different parts of a database. In a distributed database system or microservice architecture, data may be stored in different databases, each with different configurations, rules, or business logic. Database identifiers help the system identify the specific location of the data, thereby enabling the correct operations to be performed. When data requiring repair is identified as being located in the same database, the system merges these repair tasks into a single transactional task. Transactional tasks are those that need to be executed together within the same database transaction to ensure data consistency and integrity. If recovery tasks involve cross-database operations or complex repair operations, these tasks will be handled separately as non-transactional tasks. Non-transactional tasks may require invoking specific business logic or microservices to execute, as they may involve more complex business rules or coordination between multiple services.

[0056] More specifically, data fields requiring repair can be identified by analyzing data lineage, and repair tasks can be categorized as transactional or non-transactional based on the database identifiers of these fields. The specific process includes initiating the repair process, querying lineage, identifying source and redundant fields, grouping by database, generating transactional and non-transactional tasks, and ending the process. This method can efficiently process large amounts of data, ensuring the orderly execution of repair operations and data consistency, while optimizing database operations, reducing the number of connections and operations, and improving the overall efficiency of the system. In this way, the system can ensure the comprehensiveness and accuracy of data repair, while simplifying the troubleshooting and auditing process.

[0057] For example, task breakdown is as follows: Figure 2-6 , Figure 2-6 This is a schematic diagram of a task splitting method disclosed in an embodiment of this application. Figure 2-6 As we can see, root node A is the starting point of the task, involving operations across multiple databases. Nodes B, C, D, and E directly depend on A, while node B further depends on nodes F and G. The task splitting step merges nodes A and C, which can be processed in parallel, into a single task because they have no dependencies on each other. Node B, because it depends on A and is a prerequisite for nodes F and G, is executed as a separate task. Nodes F and G, which share a common parent node B and have no mutual dependencies, are merged into a single task to reduce scheduling overhead, while the independent nodes D and E are executed separately as branch tasks. In this way, the data repair process can be effectively managed and optimized, ensuring data consistency while improving repair efficiency.

[0058] This approach improves operational efficiency and ensures data consistency by merging multiple recovery tasks within the same database into a single transactional task. Furthermore, differentiating between transactional and non-transactional tasks enhances system flexibility and maintainability to accommodate diverse business needs.

[0059] In an optional implementation, before determining and executing the recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field, the method further includes: generating verification parameters based on the recovery data of the target source field, and determining whether the verification parameters conform to the preset business verification rules corresponding to the verification service. Furthermore, determining and executing the recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field includes: if the verification parameters conform to the preset business verification rules corresponding to the verification service, then determining and executing the recovery task of the target source field and the at least one target associated field based on the recovery data of the target source field.

[0060] Specifically, preset business verification rules refer to a series of pre-defined rules that define the conditions data should meet, such as data format, logical relationships, and integrity, to ensure data quality and consistency. In the data repair process, the business verification stage is a critical control point. The system checks the imported data according to the preset business rules to ensure correct data format, reasonable logical relationships, and data integrity. If the data passes verification, the process continues to the data repair stage, performing operations such as data cleaning, correction, updating, or completion. If verification fails, the process terminates directly to prevent erroneous data from being modified and to ensure data security. Verification parameters refer to the key data in the recovered data of the target source field that needs to be verified. For more details, please refer to [link to relevant documentation]. Figure 2-7 , Figure 2-7 This is a flowchart illustrating a business verification logic disclosed in an embodiment of this application. Figure 2-7 This process is used to verify the accuracy and consistency of data during data repair tasks. The process includes steps such as starting the data repair task, querying the verification service, determining the service's existence, generating verification parameters, calling the verification service, determining subsequent verifications, returning verification results, and ending the process. This process ensures that the data has passed necessary verifications before data repair is performed, thus guaranteeing data quality and system reliability. If the verification service does not exist, the process will skip the verification step and end directly. If it exists, the system will generate parameters and call the verification service to perform data verification, which may require looping verification until all verifications are completed. Finally, it returns the verification results and ends the process. In this way, it ensures that data conforms to business rules before updates, preventing erroneous data from being applied to the system.

[0061] In this way, the accuracy of data repair is ensured through preset business verification rules, thereby improving data quality. Furthermore, executing the recovery task after verification ensures data consistency and reduces risks and problems caused by data errors. Secondly, determining whether the verification parameters conform to the preset business verification rules corresponding to the verification service to verify the recovered data of the target source fields improves the accuracy of verification, thus ensuring data consistency and correctness during the data repair process.

[0062] In one optional implementation, determining whether the verification parameters conform to the preset business verification rules corresponding to the verification service includes at least one of the following: calling the reverse registration verification service of each target table through a microservice invocation mechanism to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification services of each target table; or routing the verification request to the corresponding verification service of each target table through an API gateway to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification services of each target table.

[0063] Specifically, the microservice invocation mechanism refers to the ability of a service (or component) in a microservice architecture to invoke the interface or function of another service over a network to perform a specific task. This invocation is typically remote, allowing services to communicate and collaborate. The reverse registration of validation services for each target table refers to the fact that each target table (i.e., the table that depends on the source field) has registered its own validation logic or service. These services know how to validate data according to business rules and can be invoked by other services or processes for data validation. The preset business rules corresponding to the validation services of each target table refer to the specific business rules set for each target table, defining how data should be processed and validated. For example, a table might have a rule stipulating that the price field must be between 0 and 1000. For example, suppose in an online bookstore system, the price field in the books table is updated; this field also has redundant fields in the orders and inventory tables. The system uses the microservice invocation mechanism to request the validation services registered in the orders and inventory tables. These services validate the updated data in the price field based on their respective preset business rules (for example, the rule for the orders table might be that the price field must be consistent with the books table, and the rule for the inventory table might be that the inventory quantity must be adjusted according to the new price).

[0064] Specifically, an API gateway is a type of middleware that acts as an intermediary between frontend requests and backend services, responsible for request routing, load balancing, authentication, monitoring, and other functions. In a microservice architecture, an API gateway helps simplify the interaction between clients and services. For example, when the `price` field in the `books` table is updated, the system sends a validation request through the API gateway to the validation services of the `orders` and `inventory` tables. The API gateway routes these requests to the appropriate validation services, which validate the data according to preset business rules (such as price consistency and inventory adjustment rules).

[0065] It's important to understand that in the reverse registration verification service method, each target table registers its verification service through a microservice architecture. The reverse-registered verification service of each target table is then invoked to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification services of each target table, thereby verifying the recovered data of the target source fields. In the unified verification method via the API gateway, all verification requests are routed through the API gateway, which uniformly schedules downstream services to perform verification, determining whether the verification parameters conform to the preset business verification rules corresponding to the verification services of each target table.

[0066] This solves the technical challenges in distributed environments, such as cross-database consistency repair and effective verification of distributed systems, enhances the robustness of the system, and represents a new trend in reliable repair in complex data environments.

[0067] In one optional implementation, when the field values ​​of the same meaning fields in the source table and the target table are inconsistent, a data repair trigger signal automatically triggered by the system is obtained, including at least one of the following situations: when the situation that the field values ​​of the same meaning fields in the source table and the target table are inconsistent reaches a preset condition, a data repair trigger signal automatically triggered by the system is obtained.

[0068] Specifically, the condition of inconsistent values ​​for fields with the same meaning in the source and target tables reaching a preset threshold means that when data inconsistency exceeds a certain threshold or condition, the system will automatically identify this situation and trigger a data repair process. For example, the system may set a threshold, such as a data inconsistency rate exceeding 5%, and once this threshold is exceeded, the repair process will automatically begin.

[0069] This allows for timely identification and response to data inconsistencies, reducing manual intervention and improving the efficiency and accuracy of data processing. Furthermore, it helps maintain data consistency and integrity, ensuring the reliability and stability of system operation.

[0070] In one optional implementation, performing a recovery task for the target source field and at least one target associated field includes: determining whether the recovery task for the target source field and at least one target associated field was successfully performed; if unsuccessful, retrying the task.

[0071] Specifically, in the data repair process, the system attempts to update the target source field (i.e., the field that needs repair) and at least one associated target field. The system checks whether these recovery tasks were successfully executed. If the tasks fail—that is, the data is not updated correctly or validation fails—the system will attempt to re-execute these tasks. It is understood that retry strategies may include, but are not limited to, exponential backoff algorithms, alerting mechanisms, and compensatory transactions (rollbacks). An exponential backoff algorithm refers to avoiding consecutive failures by gradually increasing the retry interval. An alerting mechanism means that if a retry still fails, the system will trigger an alert to notify the administrator or relevant personnel for manual intervention. Compensatory transactions (rollbacks) mean that if the task ultimately cannot be completed, the system may execute a compensatory transaction, such as a rollback operation, to undo all previous changes and ensure that the data returns to a consistent state.

[0072] This provides a more robust data repair mechanism that can automatically handle failures, reduce reliance on human intervention, and ensure data consistency and integrity.

[0073] To facilitate understanding of this application, the overall architecture diagram of this application is illustrated below. Please refer to the following for details. Figure 2-8 , Figure 2-8 This is a schematic diagram of the architecture of another data repair system disclosed in the embodiments of this application. Figure 2-8 As shown, a layered structure of a data repair system is presented, comprising three main layers: the presentation layer, the business logic layer, and the basic configuration layer. Each layer contains different components and functional modules that work together to automate and systematize data repair. The presentation layer, at the top, is responsible for user interaction and includes modules such as data repair scenarios, function lists, summary statistics, details, and task monitoring, allowing users to trigger repair tasks, view repair options, and monitor repair progress and results. The business logic layer contains the core logic for handling data repair, including functions such as data import, lineage query, business validation, data assembly, task splitting, and task creation. It also involves source entity modification, related entity query, and microservice calls to ensure the accuracy and consistency of data repair. The basic configuration layer provides underlying support and configuration, including lineage configuration, query field definition, entity query field configuration, validator configuration, and data repair scenario configuration, providing necessary settings and rules for the upper layers.

[0074] Overall, this application improves the efficiency and reliability of data recovery through automated processes and a clear hierarchical design. This architecture helps simplify data management tasks, reduce human error, and ensure data consistency throughout the system.

[0075] For further details, please refer to Figure 3 One embodiment of the data repair device in this application includes:

[0076] The acquisition unit is used to acquire data repair trigger signals initiated by the user or automatically triggered by the system when the field values ​​of the same meaning fields in the source table and the target table are inconsistent.

[0077] The determining unit is used to determine the target source field in the target source table corresponding to the data repair trigger signal, and to determine at least one target associated field corresponding to the target source field. The at least one target associated field includes at least one target redundant field in each target table where the field value and the target field value of the target source field have a preset mapping relationship, or at least one target redundant field and at least one target redundant source field.

[0078] A data recovery unit is configured to determine and execute recovery tasks for the target source field and the at least one target associated field based on the recovery data of the target source field, so as to perform corresponding data recovery for the target source field and the at least one target associated field based on the recovery data of the target source field.

[0079] In one alternative implementation, the determining unit may be used for:

[0080] Identify each source table and each source field of each source table;

[0081] For each source field, at least one redundant field is identified in each target table that has a preset mapping relationship between the field value and the field value of the source field, or at least one redundant field and at least one redundant source field.

[0082] Each source field has at least one redundant field, or at least one redundant field and at least one redundant source field, as at least one associated field corresponding to each source field.

[0083] In one alternative implementation, the data recovery unit can be used for:

[0084] The recovery task determines transactional and non-transactional tasks based on the target source field and the at least one target associated field;

[0085] The transactional tasks are executed in parallel through an object-relational mapping framework, and the non-transactional tasks are executed by calling the corresponding business microservices.

[0086] In one alternative implementation, the data recovery unit can be used for:

[0087] Based on the database identifier corresponding to the target source field and the at least one target association field, multiple recovery tasks of the same database are merged into transactional tasks, and recovery tasks corresponding to cross-database recovery tasks and / or complex repair operations are generated into non-transactional tasks separately.

[0088] In one alternative implementation, the determining unit may be used for:

[0089] Based on the recovered data of the target source field, a verification parameter is generated, and it is determined whether the verification parameter conforms to the preset business verification rule corresponding to the verification service.

[0090] If the verification parameters conform to the preset business verification rules corresponding to the verification service, then the recovery task of the target source field and the at least one target associated field is determined and executed based on the recovery data of the target source field.

[0091] In one alternative implementation, the determining unit may be used for:

[0092] Through the microservice invocation mechanism, the reverse registration verification service of each target table is invoked to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification service of each target table.

[0093] The API gateway routes the verification request to the verification service of each target table to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification service of each target table.

[0094] In one alternative implementation, the acquisition unit may be used for:

[0095] When a preset condition is met, and the values ​​of fields with the same meaning in the source table and the target table are inconsistent, a data repair trigger signal is obtained that is automatically triggered by the system.

[0096] For further details, please refer to Figure 4 One embodiment of the data repair device in this application includes:

[0097] Central processing unit 401, memory 405, input / output interface 404, wired or wireless network interface 403, and power supply 402;

[0098] Memory 405 is either a short-term storage memory or a persistent storage memory;

[0099] The central processing unit 401 is configured to communicate with the memory 405 and execute instructions stored in the memory 405 to perform the aforementioned operations. Figure 2-1 The method in the illustrated embodiment.

[0100] Furthermore, embodiments of this application also provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned... Figure 2-1 The method in the illustrated embodiment.

[0101] Furthermore, embodiments of this application also provide a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 2-1 The method in the illustrated embodiment.

[0102] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data repair method, characterized in that, include: When the values ​​of fields with the same meaning in the source table and the target table are inconsistent, retrieve the data repair trigger signal initiated by the user or automatically triggered by the system; Determine the target source field in the target source table corresponding to the data repair trigger signal, and determine at least one target associated field corresponding to the target source field. The at least one target associated field includes at least one target redundant field in which the field value in each target table has a preset mapping relationship with the target field value of the target source field, or at least one target redundant field and at least one target redundant source field. Based on the recovery data of the target source field, the recovery tasks of the target source field and the at least one target associated field are determined and executed to perform corresponding data recovery on the target source field and the at least one target associated field based on the recovery data of the target source field.

2. The method according to claim 1, characterized in that, Before determining at least one target associated field corresponding to the target source field, the method further includes: Identify each source table and each source field of each source table; For each source field, at least one redundant field is identified in each target table that has a preset mapping relationship between the field value and the field value of the source field, or at least one redundant field and at least one redundant source field. Each source field has at least one redundant field, or at least one redundant field and at least one redundant source field, as at least one associated field corresponding to each source field.

3. The method according to claim 1, characterized in that, The task of performing the recovery of the target source field and the at least one target associated field includes: The recovery task determines transactional and non-transactional tasks based on the target source field and the at least one target associated field; The transactional tasks are executed in parallel through an object-relational mapping framework, and the non-transactional tasks are executed by calling the corresponding business microservices.

4. The method according to claim 3, characterized in that, The target source field and the at least one target associated field are marked with corresponding database identifiers. The step of determining transactional and non-transactional tasks based on the recovery task using the target source field and the at least one target associated field includes: Based on the database identifier corresponding to the target source field and the at least one target association field, multiple recovery tasks of the same database are merged into transactional tasks, and recovery tasks corresponding to cross-database recovery tasks and / or complex repair operations are generated into non-transactional tasks separately.

5. The method according to claim 1, characterized in that, Before determining and executing the recovery task for the target source field and the at least one target associated field based on the recovery data of the target source field, the method further includes: Based on the recovered data of the target source field, a verification parameter is generated, and it is determined whether the verification parameter conforms to the preset business verification rule corresponding to the verification service. The step of determining and executing the recovery task based on the recovery data of the target source field and the at least one target associated field includes: If the verification parameters conform to the preset business verification rules corresponding to the verification service, then the recovery task of the target source field and the at least one target associated field is determined and executed based on the recovery data of the target source field.

6. The method according to claim 5, characterized in that, Determining whether the verification parameters conform to the preset business verification rules corresponding to the verification service includes at least one of the following situations: Through the microservice invocation mechanism, the reverse registration verification service of each target table is invoked to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification service of each target table. The API gateway routes the verification request to the verification service of each target table to determine whether the verification parameters conform to the preset business verification rules corresponding to the verification service of each target table.

7. A data repair device, characterized in that, include: The acquisition unit is used to acquire data repair trigger signals initiated by the user or automatically triggered by the system when the field values ​​of the same meaning fields in the source table and the target table are inconsistent. The determining unit is used to determine the target source field in the target source table corresponding to the data repair trigger signal, and to determine at least one target associated field corresponding to the target source field. The at least one target associated field includes at least one target redundant field in each target table where the field value and the target field value of the target source field have a preset mapping relationship, or at least one target redundant field and at least one target redundant source field. A data recovery unit is configured to determine and execute recovery tasks for the target source field and the at least one target associated field based on the recovery data of the target source field, so as to perform corresponding data recovery for the target source field and the at least one target associated field based on the recovery data of the target source field.

8. A data repair device, characterized in that, include: Central processing unit and memory; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

10. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 6.

Images