Data recovery method and device, storage medium and electronic device

By combining the data rule engine and the data prediction generation algorithm, data can be updated and repaired in layers in real time, solving the problem of data repair not being able to be updated in real time in existing technologies, improving the repair effect and efficiency, and adapting to the rapid changes of multi-source heterogeneous data.

CN120723539APending Publication Date: 2025-09-30ZTE CORP
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202510811236.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-17
Publication Date
2025-09-30

AI Technical Summary

Technical Problem

Existing data repair technology cannot be updated in real time as data changes, resulting in poor repair effects and inability to cope with rapid changes in data source formats or business rules. In addition, manual review is inefficient and cannot meet the real-time needs of massive data.

Method used

It adopts a combination of data rule engine and data prediction generation algorithm, updates in real time through preset monitoring methods, configures rules and algorithms according to data business scenarios, performs format and logic checks, repairs data in layers and categories, and uses database pre-write logs to ensure the atomicity and rollback of repair operations.

Benefits of technology

It achieves real-time and high efficiency of data repair, improves the accuracy and consistency of data repair, reduces human errors and data delays, and adapts to the rapid changes of multi-source heterogeneous data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120723539A_ABST
    Figure CN120723539A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data recovery method and device, a storage medium and an electronic device.The method comprises the steps that abnormal second data in first data is determined according to a data rule engine, and a data rule applied in the data rule engine is configured according to a data service scene corresponding to the first data; the second data is restored through a preset processing mode, the preset processing mode comprises a data prediction generation algorithm, and the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring mode. By the adoption of the technical scheme, the problem that the data restoration effect is poor due to the fact that a data restoration method in the related technology cannot be updated in real time along with changes of the restored data is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular to a data repair method, device, storage medium, and electronic device. Background Art

[0002] With the acceleration of digitalization, big data and real-time data have become core drivers of enterprise decision-making, business operations, and technological innovation. According to IDC, global data volume is projected to reach 393.8 zettabytes (ZB) by 2028, with real-time data accounting for over 35%. Data sources are diverse (e.g., IoT devices, sensors, and user behavior logs). However, data quality defects are common. For example, field types that do not match definitions can lead to parsing failures or calculation errors; values ​​outside the acceptable range can cause system misjudgments or security vulnerabilities; logical failures in field associations can distort analysis results; and missing key fields can impact data relevance and business processes. If these data quality defects are not promptly addressed, they can lead to serious business risks. For example, erroneous data entering decision-making systems can lead to financial fraud, medical misdiagnosis, or production accidents. Abnormal data can consume storage resources or trigger calculation anomalies, causing system crashes or performance degradation.

[0003] Current data repair technologies primarily rely on the following methods: 1) Static repair based on predefined rules, but static repair based on predefined rules often has poor dynamic adaptability and cannot cope with rapid changes in data source formats or business rules (such as new fields reported by new devices); 2) Intelligent repair based on machine learning, but intelligent repair based on machine learning often lacks real-time performance, and complex models are difficult to repair in real time in high-concurrency scenarios, resulting in data delays; 3) Manual review and post-repair, but manual review and post-repair are inefficient, and manual processing cannot cope with the real-time requirements of massive data. In other words, the data repair methods in related technologies often have poor real-time performance and cannot be updated in real time as the data being repaired changes, resulting in poor data repair effects.

[0004] Regarding related technologies, data repair methods cannot be updated in real time as the data being repaired changes, resulting in poor data repair effects. Currently, no effective solution has been proposed.

[0005] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the Invention

[0006] The embodiments of the present application provide a data repair method, device, storage medium, and electronic device to at least solve the problem that the data repair method cannot be updated in real time as the data being repaired changes, resulting in poor data repair effect.

[0007] According to one aspect of an embodiment of the present application, a data repair method is provided, including: determining abnormal second data in first data based on a data rule engine, wherein the data rules applied in the data rule engine are configured according to the data business scenario corresponding to the first data; repairing the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0008] In an exemplary embodiment, before determining the abnormal second data in the first data according to the data rule engine, the method also includes: monitoring a first event through a first thread to obtain the data rule, and updating the data rule engine through the data rule, wherein the first event is used to indicate that there is a configuration change in the data rule; and monitoring a second event through a second thread to obtain the data prediction generation algorithm, and updating the data prediction generation algorithm to the preset processing method, wherein the second event is used to indicate that there is a change in the data prediction generation algorithm.

[0009] In an exemplary embodiment, determining an abnormal second data in first data based on a data rule engine includes: performing a format check on the first data through the data rule engine to determine that third data with a format abnormality exists in the first data, wherein the data rules applied in the data rule engine include: a data format rule and a first repair rule corresponding to the data format rule, a data logic rule and a second repair rule corresponding to the data logic rule; performing a logical check on fourth data through the data rule engine to determine that fifth data with a logical abnormality exists in the fourth data, wherein the fourth data is data in the first data other than the third data; and determining the third data and the fifth data as the second data.

[0010] In an exemplary embodiment, the data rule engine performs a format check on the first data to determine that third data with a format abnormality exists in the first data, including: checking the target format information of the first data according to the data format rule by the data rule engine; repairing the sixth data with target format information abnormality in the first data according to the first repair rule, and using the sixth data that fails to be repaired as the third data.

[0011] In an exemplary embodiment, a logical check is performed on the fourth data by the data rule engine to determine that fifth data with logical anomalies exists in the fourth data, including: performing a logical check on the fourth data according to the data logic rule by the data rule engine, and taking the data in the fourth data that fails the logical check as the fifth data.

[0012] In an exemplary embodiment, the second data is repaired by a preset processing method, including: dividing the second data into first dynamic data and first static data according to data type; repairing the first dynamic data by the data prediction generation algorithm, and repairing the first static data by a target audit method.

[0013] In an exemplary embodiment, before repairing the first dynamic data through the data prediction generation algorithm, the method also includes: performing local data dimension expansion on the first static data; repairing the seventh data according to the logical conflict type of the seventh data to obtain eighth data, wherein the seventh data includes: the first static data that failed dimension expansion and the first dynamic data; splitting the eighth data into second dynamic data and second static data, and updating the first dynamic data and the first static data respectively through the second dynamic data and the second static data.

[0014] In an exemplary embodiment, the seventh data is repaired according to the logical conflict type of the seventh data to obtain the eighth data, including: updating the first sub-data through an external interface; wherein the first sub-data is the data in the seventh data where the logical conflict type is a single data conflict; and repairing the second sub-data through a second repair rule, and updating the third sub-data through the external interface, wherein the second sub-data is the data in the seventh data where the logical conflict type is a conflict between data, and the third sub-data is the data in the second sub-data that fails to be repaired by the second repair rule; and the first sub-data that fails to be updated and the third sub-data that fails to be updated are determined as the eighth data.

[0015] In an exemplary embodiment, repairing the first dynamic data through the data prediction generation algorithm includes: decomposing the first dynamic data into target components, wherein the target components include: trend component, seasonal component and residual component; determining multiple single-component algorithms for processing the target components according to the component strength of the target components; fusing the first output results after processing the target components by the multiple single-component algorithms to obtain a second output result, wherein the second output result is the repair result of the first dynamic data.

[0016] In an exemplary embodiment, before determining the abnormal second data in the first data according to the data rule engine, the method also includes: establishing a database write-ahead log corresponding to the first data, and backing up the first data to the database write-ahead log, wherein the database write-ahead log is used to record any operation performed on the first data through the data rule engine and the preset processing method, and verify the operation result of the any operation; the database write-ahead log is also used to roll back the first data if the verification result of the operation result is a verification failure.

[0017] According to another aspect of the embodiment of the present application, a data repair device is also provided, including: a determination module, used to determine abnormal second data in the first data based on a data rule engine, wherein the data rules applied in the data rule engine are configured according to the data business scenario corresponding to the first data; a repair module, used to repair the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0018] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned data repair method when running.

[0019] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the data repair method through the computer program.

[0020] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the above-mentioned data repair method when executed by a processor.

[0021] This application determines abnormal second data within first data using a data rule engine, wherein the data rules applied in the data rule engine are configured based on the data business scenario corresponding to the first data; and repairs the second data using a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time via a preset monitoring method. This solves the technical problem in related technologies where data repair methods cannot be updated in real time as the data being repaired changes, resulting in poor data repair results, thereby improving the data repair effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The exemplary embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 This is a hardware structure block diagram of a computer terminal of the data repair method according to an embodiment of the present application;

[0024] Figure 2 is a flow chart (1) of a data repair method according to an embodiment of the present application;

[0025] Figure 3 is an architectural diagram of an automated data repair system according to an embodiment of the present application;

[0026] Figure 4 Flowchart (2) of the data repair method according to an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of a data repair device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 : is a hardware structure block diagram of a computer terminal of the data repair method of the embodiment of the present application. Figure 1As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 Shown are different configurations with more functionality.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data repair method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] Figure 2 Flowchart (1) of the data repair method according to an embodiment of the present application, such as Figure 2 As shown, the steps of the method include:

[0034] Step S202: determining abnormal second data in the first data according to a data rule engine, wherein the data rules applied in the data rule engine are configured according to a data business scenario corresponding to the first data.

[0035] Step S204: repairing the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0036] An embodiment of the present application determines abnormal second data within first data based on a data rule engine, wherein the data rules applied in the data rule engine are configured according to the data business scenario corresponding to the first data; and repairs the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method. This solves the technical problem in related technologies that data repair methods cannot be updated in real time as the data being repaired changes, resulting in poor data repair results, thereby improving the data repair effect.

[0037] In an exemplary embodiment, before determining the abnormal second data in the first data according to the data rule engine, the method also includes: monitoring a first event through a first thread to obtain the data rule, and updating the data rule engine through the data rule, wherein the first event is used to indicate that there is a configuration change in the data rule; and monitoring a second event through a second thread to obtain the data prediction generation algorithm, and updating the data prediction generation algorithm to the preset processing method, wherein the second event is used to indicate that there is a change in the data prediction generation algorithm.

[0038] Optionally, the data rule engine is deployed in Figure 3 In the case of the data error detection and repair module in the automated data repair system shown, the data error detection and repair can be configured with a rule configuration listener and an intelligent algorithm listener, which are respectively used to monitor the first event issued by the data rule configuration module and the second event issued by the intelligent algorithm generation module. Optionally, the first event and the second event can be sent to the message middleware, and then the rule configuration listener and the intelligent algorithm listener can monitor the message middleware to monitor the first event and the second event. The first thread includes the rule configuration listener, and the second thread includes the intelligent algorithm listener.

[0039] In an exemplary embodiment, determining an abnormal second data in first data based on a data rule engine includes: performing a format check on the first data through the data rule engine to determine that third data with a format abnormality exists in the first data, wherein the data rules applied in the data rule engine include: a data format rule and a first repair rule corresponding to the data format rule, a data logic rule and a second repair rule corresponding to the data logic rule; performing a logical check on fourth data through the data rule engine to determine that fifth data with a logical abnormality exists in the fourth data, wherein the fourth data is data in the first data other than the third data; and determining the third data and the fifth data as the second data.

[0040] Among them, data format rules include: data type, data unit, and time format standards; data logic rules cover single data logic rules (such as data size range, data enumeration range, data length range, etc.) and data logic rules (that is, a certain data is generated by calculating other data values).

[0041] Optionally, the data rule engine performs a format check on the first data to determine whether third data with a format abnormality exists in the first data, including: checking the target format information of the first data according to the data format rule by the data rule engine; repairing the sixth data with target format information abnormality in the first data according to the first repair rule, and using the sixth data that fails to be repaired as the third data.

[0042] The target format information includes: Figure 4 The data encoding, data type, data format, data overflow, data unit, and time format are shown in the format layer.

[0043] That is to say, first, the data rule engine performs a format check on the first data according to the data format rules in the data rules, and the first repair rule corresponding to the data format rule is used to repair the sixth data with target format information abnormality. The sixth data that fails to be repaired will be used as the third data output after the format check.

[0044] Optionally, a logical check is performed on the fourth data through the data rule engine to determine that there is fifth data with logical anomalies in the fourth data, including: performing a logical check on the fourth data according to the data logic rule through the data rule engine, and taking the data in the fourth data that fails the logical check as the fifth data.

[0045] In an exemplary embodiment, the second data is repaired by a preset processing method, including: dividing the second data into first dynamic data and first static data according to data type; repairing the first dynamic data by the data prediction generation algorithm, and repairing the first static data by a target audit method.

[0046] Optionally, before repairing the first dynamic data through the data prediction generation algorithm, the method also includes: performing local data dimension expansion on the first static data; repairing the seventh data according to the logical conflict type of the seventh data to obtain eighth data, wherein the seventh data includes: the first static data that failed to be expanded and the first dynamic data; splitting the eighth data into second dynamic data and second static data, and updating the first dynamic data and the first static data respectively through the second dynamic data and the second static data.

[0047] Furthermore, the seventh data is repaired according to the logical conflict type of the seventh data to obtain the eighth data, including: updating the first sub-data through an external interface; wherein the first sub-data is the data in the seventh data whose logical conflict type is a single data conflict; and repairing the second sub-data through a second repair rule, and updating the third sub-data through the external interface, wherein the second sub-data is the data in the seventh data whose logical conflict type is a conflict between data, and the third sub-data is the data in the second sub-data that fails to be repaired by the second repair rule; the first sub-data that fails to be updated and the third sub-data that fails to be updated are determined as the eighth data.

[0048] Among them, the preset processing methods also include: local data dimension expansion, logical conflict repair, and external interface update.

[0049] That is, before repairing the first dynamic data using the data prediction generation algorithm, the first static data obtained by splitting can also be subjected to local data expansion; then, logical conflict repair is performed on the first dynamic data obtained by splitting and the first static data that failed to be expanded (i.e., the seventh data). Logical conflict repair means repairing the seventh data according to the second repair rule corresponding to the data logic rule. Data logic rules can also be further classified into single data conflict rules and data conflict rules.

[0050] Optionally, the first dynamic data is repaired by the data prediction generation algorithm, including: decomposing the first dynamic data into target components, wherein the target components include: trend component, seasonal component and residual component; determining multiple single-component algorithms for processing the target components according to the component strength of the target components; fusing the first output results after the multiple single-component algorithms process the target components to obtain a second output result, wherein the second output result is the repair result of the first dynamic data.

[0051] It should be noted that the first dynamic data is decomposed into the above-mentioned target components by using the STL (Seasonal-Trend decomposition using LOESS) algorithm.

[0052] Determining a plurality of single-component algorithms for processing the target component according to the component strength of the target component, including:

[0053] The absolute value of the slope of the trend component is calculated. If the absolute value of the slope exceeds the trend threshold, it is determined to be a strong trend; otherwise, it is determined to be a weak trend. The SARIMA model is used for strong trends, and the Prophet model is used for weak trends.

[0054] A Fourier transform is used to extract the dominant frequency energy ratio of the seasonal component. If it exceeds the seasonal threshold, it is considered strong seasonality; otherwise, it is considered weak seasonality. Strong seasonality is detected using the SARIMA model with seasonal terms, while weak seasonality is detected using the LSTM neural network.

[0055] ARIMA modeling is performed on the residual components. If the optimal model ends with p + q > 3, the residual is considered complex, otherwise it is considered simple. Complex residuals use the XGBoost model, and simple residuals use the ARMA model.

[0056] The first output results after the target component is processed by the multiple single-component algorithms are fused to obtain a second output result, including: splicing the multiple first output results to obtain a splicing vector; and using a multi-layer perceptron (MLP) to fuse the splicing vectors to obtain the second output result.

[0057] In an exemplary embodiment, before determining the abnormal second data in the first data according to the data rule engine, the method also includes: establishing a database write-ahead log corresponding to the first data, and backing up the first data to the database write-ahead log, wherein the database write-ahead log is used to record any operation performed on the first data through the data rule engine and the preset processing method, and verify the operation result of the any operation; the database write-ahead log is also used to roll back the first data if the verification result of the operation result is a verification failure.

[0058] Next, the data repair method will be further described in conjunction with the following embodiments.

[0059] In related technologies, data cleaning and repair technology mainly relies on the following methods.

[0060] 1) Static repair based on rule engine.

[0061] Technical Features: Filters or corrects data using predefined rules (such as field type validation and value range restrictions). However, its rule coverage is insufficient, requiring numerous rules for complex scenarios (e.g., multi-field logical consistency), resulting in high maintenance costs and the risk of omissions. It also has poor dynamic adaptability and cannot cope with rapid changes in data source formats or business rules (e.g., new fields reported by new devices). Repair granularity is coarse, requiring only truncation or default value filling for some anomalies (e.g., GPS coordinate deviations), leading to information loss.

[0062] 2) Intelligent repair based on machine learning.

[0063] Technical Features: Leverages clustering, regression, or generative models (such as AutoML and GAN) to predict missing values ​​or correct outliers. However, this relies on labeled data: a large amount of labeled data is required to train the model, and the model is sensitive to noisy data. It also suffers from insufficient real-time performance, making it difficult to implement real-time repairs for complex models in high-concurrency scenarios, leading to data delays. Furthermore, it suffers from poor interpretability, making repair results difficult to trace and potentially introducing uncontrollable "black box" errors.

[0064] 3) Manual review and post-repair.

[0065] Technical Characteristics: This method relies on manual investigation and correction of abnormal data, which is common in critical business scenarios. However, it is inefficient and cannot meet the real-time demands of massive amounts of data. It is subject to subjective bias, and the results of corrections depend on the experience of the reviewer, which is prone to human error.

[0066] In addition, related technologies generally lack atomicity guarantees for repair operations, which will lead to the risk of data inconsistency. If an interruption occurs during the repair process (such as network fluctuations), some fields may be corrected while other fields are not updated, causing data logic conflicts. At the same time, the rollback mechanism is missing, and it is impossible to quickly restore to the state before the repair, especially when batch repairs fail, which may increase data loss. Related technologies are insufficient for collaborative repair of multi-source data. Complex scenarios (such as multi-field association repair) require collaborative analysis across data sources, but related technologies usually process single data independently and cannot globally optimize the repair results. For example, abnormal disk usage may require a comprehensive judgment based on multi-dimensional data such as total capacity, usage, and device model, which is difficult to achieve with traditional methods.

[0067] In order to overcome the defects of the data repair methods in the related art, the embodiments of the present application provide an automated data repair method, including: configuring rules for data fields according to data business scenarios; realizing the dynamic loading function of the rule engine and machine learning algorithm model, and updating the rule engine and data prediction generation algorithm of the data error judgment and repair module in real time; based on the data field rule configuration, performing hierarchical classification of the accessed data for data error judgment and adopting multimodal path decision to repair the data; creating an atomic repair mechanism based on the database write-ahead log (WAL) to ensure automatic rollback when the repair fails to avoid the spread of dirty data; online training of the machine learning model and using it to predict missing data and abnormal data, while updating the latest algorithm model and its confidence.

[0068] Furthermore, data fields are configured with rules based on data business scenarios, including: defining data format rules, data logic rules, and data repair rules, and issuing change information in a timely manner when changes occur.

[0069] Furthermore, data format rules include: data type, data unit, and time format standards; data logic rules cover single data logic rules (such as data size range, data enumeration range, data length range, etc.) and data logic rules (that is, a certain data is generated by calculating other data values).

[0070] Furthermore, the rule engine of the data anomaly detection and repair module is updated in real time. This involves launching a thread in the data anomaly detection module to monitor data rule configuration change events in the message middleware and synchronously updating the data rule engine to ensure that data rule modifications are applied to the data anomaly detection and repair module in real time. The machine learning algorithm model is dynamically loaded by similarly launching a thread in the data anomaly detection module to monitor algorithm model change events in the message middleware to enable the real-time application of the latest model algorithm to the data repair logic.

[0071] Furthermore, hierarchical and categorized data error detection includes prioritizing repairs based on error type (format error > logical conflict > data integrity), and performing tiered processing accordingly. Error types are further categorized as data encoding errors, data type errors, numerical range overflows, data unit anomalies, time data format anomalies, single data logical conflicts, inter-data logical conflicts, and missing data.

[0072] Furthermore, multimodal path decision repair includes: for missing static data, use existing persistent data to fill in; for dynamic data, calculate and fill in according to the generation method in the rule configuration; for data that can be verified through an external authoritative API, fill in after verification through the API; for numerical indicator data, use the model prediction provided by the intelligent prediction algorithm to generate, and determine the processing method based on the model confidence level - when the confidence level exceeds the set threshold, apply the generated data, otherwise turn to manual review.

[0073] Furthermore, an atomic repair mechanism based on WAL logs starts a transaction before the data repair operation, records each repair operation in the WAL log, and executes the repair operations sequentially. If a failure occurs during the repair process, a rollback operation is triggered to restore the data to its pre-repair state, thus preventing the spread of dirty data.

[0074] Furthermore, the machine learning model is continuously trained according to the set frequency and window range, and the generalized machine learning model based on the three base models is stacked and applied online to predict missing data.

[0075] It should be noted that this application is applicable to multi-source heterogeneous data processing scenarios that require automated data quality management. Its application environment must meet the following basic characteristics:

[0076] The system is deployed in digital business environments with data integration requirements. Typical scenarios include IoT device data collection, multi-system enterprise data integration, cross-platform business interactions, and other areas with data anomaly risks. The application environment must include access to heterogeneous data sources and support hybrid processing of structured databases (such as MySQL and PostgreSQL), semi-structured logs (such as JSON and XML), and real-time data streams (such as Kafka message queues).

[0077] The network environment must ensure the reliability of repair instructions and data transmission. When deployed on a local area network or cloud platform, a basic message queue mechanism is required to enable the distribution and status synchronization of repair tasks. For edge computing scenarios (such as industrial sites and smart terminals), the system can adapt to a lightweight deployment model, executing local repairs through an embedded rules engine and maintaining policy synchronization with the central node.

[0078] Typical application scenarios include, but are not limited to, real-time cleaning of sensor data from manufacturing equipment (e.g., correcting temperature threshold drift), repairing logical conflicts in financial transaction records (e.g., handling inconsistent transfer amounts across systems), and completing patient data integrity in medical information systems (e.g., intelligently inferring missing examination indicators). The system's modular design adapts to deployments of varying scales. Small and medium-sized applications can rely solely on rule configuration and basic repair functionality, while large distributed systems can expand upon intelligent algorithms and multi-source collaborative repair capabilities.

[0079] This application can not only meet the high-concurrency repair needs in cloud computing environments (such as e-commerce platforms processing tens of thousands of order data per second), but can also adapt to resource-constrained edge devices (such as local data verification of smart meters). It can realize on-demand activation of functional modules through a dynamic loading mechanism, ensuring that effective data repair services can be provided in different resource configuration scenarios.

[0080] Further, such as Figure 3 As shown, the data repair method proposed in this application can be executed based on a data repair system, which includes: a data rule configuration module, a data error detection and repair module, an intelligent algorithm generation module, and two external dependencies: a local database and an external API interface.

[0081] The functions of each module are described as follows:

[0082] 1) Data rule configuration module: Data rules are the foundation of the entire system and are indispensable. Data error detection and data repair are based on the configuration provided by this module. Data rule configuration is divided into the following types:

[0083] Data type configuration: As the first step in data access, you need to clearly define the data type. Different data types have different exception methods. Data type configuration includes the data type, data length, and data format (such as IP format, geographic location format, etc.). Data length and data format can be defined using regular expressions.

[0084] Data unit configuration: This applies to string types that contain measurement information, such as weight units (jin, gram, kilogram) or disk size units (byte, KB, MB). Inconsistent units can cause problems in subsequent data analysis, so it is necessary to clarify unit standards and conversion formulas.

[0085] Data time format configuration: Since data may come from different regions, terminals, and software, inconsistent time formats or time zones are inevitable. Therefore, it is necessary to define a unified time format and conversion scheme.

[0086] Data logic rule configuration: In addition to the aforementioned scenarios where data anomalies may occur, different business scenarios can also introduce new data anomaly scenarios. Defining these rules requires users to be familiar with the business scenario. This can be done by configuring single-data logic rules or cross-data logic rules. For example, order quantities cannot be negative; pass rates must be between 0 and 100; or even if all students in a class scored 90 in a certain subject, the average score attribute shows 91. For logical conflicts between data, remediation logic must also be defined.

[0087] This module also provides the function of issuing rule configuration updates, and notifies the data error detection and repair module of the changed rule configuration in real time.

[0088] 2) Intelligent algorithm generation module: This module is the key to handling abnormal data repair.

[0089] Data is generally divided into static and dynamic data. Static data is relatively fixed and infrequently changes, while dynamic data evolves over time, reflecting the real-time state of a system or object. Static data is generally not predictable based on patterns, while dynamic data can be predicted based on relationships or trends. Therefore, the intelligent algorithm generation module primarily focuses on predicting dynamic data.

[0090] The changing trends of dynamic data are complex and diverse. In order to cope with various complex scenarios, it is divided into three components: trend, season, and residual. Different base models are used for prediction respectively, and finally integrated learning is achieved through stacking generalization to generate the final prediction data.

[0091] After each model algorithm update is completed, the latest algorithm model is promptly sent to the data error detection and repair module through the message middleware.

[0092] 3) Data Anomaly Identification and Repair Module: This module is the core of the entire system, responsible for identifying and repairing anomalies in the data. This module is divided into three levels of processing: format layer, logic layer, and integrity layer.

[0093] Format layer: This layer handles basic data format anomalies. This layer is divided into two categories: self-driven exception handling (which does not require rule configuration), including encoding format anomalies (such as automatically correcting GBK encoding to UTF-8); and rule-based exception handling, including data type inconsistency, value overflow, data unit anomalies, and time format anomalies.

[0094] Logic layer: Verifies data anomaly logic defined by business scenarios, which is divided into single data logic conflicts and inter-data logic conflicts.

[0095] The integrity layer completes data repair operations, addressing missing essential data and abnormal data. This multimodal repair process includes local data dimensionality expansion, external API acquisition, multi-field calculation generation, machine learning algorithm prediction, and manual review. The system automatically selects the appropriate repair method based on different data types and logical configuration rules.

[0096] The data error detection and repair module also includes a rule configuration listener (which listens to the configuration update results issued by the data rule configuration module), an intelligent algorithm listener (which listens to the updated algorithm model of the intelligent algorithm generation module), and an exception repair transaction manager (which implements an atomic repair mechanism based on the WAL log, automatically rolls back when the repair fails, and avoids the spread of dirty data).

[0097] In addition to the three major modules in the system, you can also choose to add two external dependencies, a local database and an external API interface, based on the data repair method (equivalent to the preset processing method in the above embodiment).

[0098] Combine Figure 3 The system shown in the figure addresses the data quality defects in big data and real-time data access. The automated data anomaly detection and repair method proposed in this application (equivalent to the data repair method in the above embodiment) includes the following steps:

[0099] S301, configure data field rules according to data business scenarios, which requires a detailed understanding of data business rules and data structure design principles. Specific analysis can be performed based on the data processing process, including data collection, processing, storage, and presentation.

[0100] Specifically, the data types used in the data access system should be clearly defined, such as integer, floating-point, string, Boolean, and timestamp types. The possible rule types corresponding to each data type should also be defined. For example, field length rules can be applied to integer, floating-point, and string data; unified unit standards can be applied to data with measurement meanings; time format standards can be applied to all time format fields; and data logic conflict rules can be applied to any field.

[0101] Furthermore, after data rule configuration is complete, the modified data rules should be immediately sent to the messaging middleware. Common messaging middleware options include Kafka, RabbitMQ, and Redis. Subsequently, the data error detection and repair module can immediately receive the data rule configuration information and create a data rule engine, achieving dynamic and smooth data rule engine updates.

[0102] S302: When real data cannot be obtained by other means, a machine learning algorithm is used to predict dynamic data. This provides an abnormal data repair algorithm module based on dynamic data feature decomposition. By dynamically identifying data features and combining targeted models, high-precision outlier prediction and repair is achieved. The following is a specific technical solution:

[0103] Specifically, firstly, the dynamic data feature decomposition and judgment are carried out. The decomposition method is as follows: the original time series data Y is decomposed using the STL algorithm. t Decomposed into trend component T t , seasonal component S t and the residual component R t , satisfying Y t =T t +S t +R t The decomposition steps are as follows:

[0104] Initialization: Set the seasonal period s (e.g., s=12 for monthly data) and separate the trend and seasonal components through locally weighted regression (LOESS).

[0105] Iterative optimization: The decomposition parameters are adjusted through multiple iterations to ensure the independence of each component.

[0106] Residual Extraction: Calculating R t =Y t -T t -S t , and test whether the residuals are white noise (via Ljung-Box test).

[0107] The feature judgment logic is as follows:

[0108] Trend Strength: Calculate the trend component T t The absolute value of the slope |ΔT t |, if it exceeds the threshold δ T (such as 0.1 times the data standard deviation), it is judged as a strong trend.

[0109] Seasonal significance: Extract the seasonal component S through Fourier transform t If the main frequency energy ratio exceeds the threshold δ S (e.g. 50%), it is determined to be strongly seasonal.

[0110] Residual complexity: residual R t Perform ARIMA modeling. If the optimal model order p+q>3, it is determined to be a complex residual.

[0111] Furthermore, three different types of algorithm models are applied to the three components divided above. The model algorithm selection for the trend component is shown in Table 1:

[0112] Table 1 Dynamic feature-model mapping

[0113]

[0114] Combined with Table 1, the model algorithm selection for the trend component is as follows:

[0115] The strong trend adopts the SARIMA model, the formula is:

[0116]

[0117] Where d is the difference order and B is the lag operator.

[0118] Weak trends use the Prophet model to automatically capture change points and fit nonlinear trends.

[0119] The model algorithm selection for the seasonal component is as follows:

[0120] Strong seasonality uses the SARIMA model with seasonal terms, the formula is:

[0121]

[0122] Where s is the seasonal period and D is the seasonal difference order.

[0123] Weak seasonality uses LSTM neural network, through the gate unit (such as forget gate f t , input gate i t , output gate o t ) captures the periodic pattern, and the formula is:

[0124] i t =σ(W ui x t +W hi h t-1 +b i );

[0125] f t =σ(W uf x t +W hf h t-1 +b f );

[0126] o t =σ(W uo x t +W ho h t-1 +b o );

[0127] g t =tanh(W ug x t +Whg h t-1 +b g );

[0128] c t =f t ⊙c t-1 +i t ⊙g t ;

[0129] h t =o t ⊙tanh(c t ).

[0130] Among them, c t is the cell state, h t In hidden state.

[0131] The model algorithm selection for the residual component is as follows:

[0132] The complex residual adopts the XGBoost model, and the objective function is:

[0133]

[0134] Where Ω(f i ) is a regularization term to prevent overfitting.

[0135] The simple residual uses the ARMA model, and the formula is:

[0136]

[0137] Among them, ∈ t is white noise.

[0138] Furthermore, after having different types of component models, we use the stacking integration method to integrate different models for learning. The steps are as follows:

[0139] 1) First-layer base model: Select the above-mentioned single-component model (such as SARIMA, LSTM, XGBoost) according to the data characteristics, and predict the trend, season, and residual components respectively.

[0140] 2) Second-layer meta-model: Multi-layer perceptron (MLP) is used to fuse the output of the base model, and the input is the concatenation vector of the predicted values ​​of each component Output is the final repair value

[0141] 3) Weight optimization: Determine the meta-model parameters by minimizing the loss function (such as MAE) on the validation set.

[0142] Furthermore, the present application predicts dynamic data, so the selection of the training set of the above algorithm cannot randomly disrupt the data order, ensuring that the model only uses sequential historical information for prediction.

[0143] Furthermore, the embodiment of the present application adopts an atomic training and verification system. For the time-constrained data partitioning, a sliding time window stratified sampling method is used. The training set and verification set are strictly separated in chronological order, and cross-time period data mixing is prohibited. The partitioning rules are formally defined:

[0144] Training set = Y 1:T , validation set = Y T+1:T+v ,

[0145] In other words, the application scenario of this application is to continuously access data, allowing for online verification and model updates. A sliding window verification strategy is used: for example, when the window duration is 5 hours and the sliding window duration is 1 hour, the data from the last 5 hours is used for learning and the value for the next hour is predicted. After 1 hour, the model confidence is calculated using real data and the algorithm model is updated.

[0146] For rolling verification error correction: a timing deviation compensation function is proposed:

[0147] ∈ adj =∈ val ×(1+β·max_gap(T)); (correction of cross-period forecast bias)

[0148] Solve the problem of periodic boundary error amplification in traditional rolling verification.

[0149] S303, the data error detection and repair module is the core of the automated data repair method, and its layered repair process is as follows: Figure 4 shown.

[0150] Specifically, S41: at the format layer, firstly, the coding format is checked, and when an abnormal coding format is found, the coding format of the data with the abnormal coding format is converted, and if the operation fails, the data with the failed operation is marked as abnormal.

[0151] Further, S42: perform a data type check. If the check result is inconsistent with the definition in the rule configuration (ie, the data type is abnormal), perform a data format conversion operation. If the operation fails, the data that failed to be converted is marked as abnormal.

[0152] Furthermore, S43 performs checks on numerical overflow, data units, and time formats. Numerical data is checked for data overflow; character strings with measurement meanings are checked for data units; and time format data is checked for time formats. Data is automatically repaired based on the check results. If the operation fails, the data for which repair failed is marked as abnormal (the third data in the above embodiment includes the data marked as abnormal in steps S41, S42, and S43, respectively).

[0153] Furthermore, S44: when performing a logic layer check, each item is checked according to the configured rules, such as missing data, data size not meeting the range, data regular expression mismatch, logical contradictions between data, etc. When a check failure is encountered at this layer, it is marked as an abnormality (the fourth data in the above embodiment is the data input to the logic layer, and the fifth data in the above embodiment is the data marked as abnormal by the logic layer).

[0154] Furthermore, S45: implement a multimodal repair path selector in the integrity layer to complete the last barrier of data repair. For static data, you can first choose to obtain it through local data dimension expansion. If the local data acquisition is unsuccessful, the logical conflict type is determined in the same way as dynamic data processing. If it is a conflict between data, it can be repaired using the logical rules defined between the data. If the repair is unsuccessful, it is obtained from the external API in the same way as single data conflict processing. If the acquisition from the external API fails or cannot be obtained from the external API, proceed to the next step of processing. If it is dynamic data, this type of data generally has some trend or some kind of correlation, and the result can be predicted by a machine learning algorithm. Then, according to the confidence threshold configured by the data rule, choose to accept the system or go to the manual review stage like static data.

[0155] S304, this application realizes atomic control of data repair operations by integrating database pre-write log WAL technology, ensuring that the data status is traceable and rollbackable during the abnormal repair process, and avoiding the spread of dirty data.

[0156] Specifically, the WAL log module is deeply coupled with the data repair engine, and through a three-level mechanism of transaction logging, atomicity verification, and differential rollback, the repair operation is encapsulated as an atomic transaction unit. The log entry contains:

[0157] 1) Transaction metadata: unique transaction ID, operation type (repair / rollback), data entity identifier, and operation timestamp.

[0158] 2) Original data image: a complete data snapshot before repair (including field values ​​and version numbers).

[0159] 3) Repair operation instructions: a serialized description of the specific repair logic (such as field replacement rules and API call parameters).

[0160] 4) Status mark: PENDING (pending execution), COMMITTED (committed), ROLLEDBACK (rolled back).

[0161] Furthermore, the rollback process is as follows:

[0162] 1.Fix transaction initialization.

[0163] 1) Generate transaction ID: Generate a globally unique transaction identifier TXN_ID through UUID.randomUUID(), which serves as the primary key for full-link tracking.

[0164] 2) Create a log write-ahead buffer: Write the transaction start record in the WAL log file.

[0165] 3) Back up the original data image: Take a snapshot of the target data entity (such as the record with ID = 1001 in the order table) and store it in the RAW_DATA field of the WAL log to ensure that the original state can be restored during rollback.

[0166] 2. Perform repair operations.

[0167] Intermediate state temporary storage: Each time a repair sub-operation is completed, a temporary operation record is appended to the WAL log and is not immediately submitted to the database.

[0168] Atomic checkpoint: When connecting cross-layer repairs (such as before switching from format layer repair to logic layer check), a checkpoint record is inserted to mark the current repair progress.

[0169] 3. Transaction commit or rollback strategy.

[0170] Verification of repair results: After each repair is completed, it needs to be verified, such as format compliance, logical consistency, and completeness.

[0171] Decision Logic: If the repair result verification fails, an atomic rollback process is triggered, discarding all uncommitted repair operations. The WAL log status is marked as ROLLEDBACK, along with an error code and failure reason. If the repair result verification satisfies, the WAL log status is marked as COMMITTED, and the repaired data is persisted to the database. A repair audit record is generated, associating the transaction ID with the operation details.

[0172] 4. Log archiving and cleaning.

[0173] Transaction closure: Regardless of commit or rollback, the WAL log status is updated to the final status (COMMITTED / ROLLEDBACK) and written to the audit log.

[0174] Log lifecycle management: Logs of completed transactions are retained for a certain period (e.g., 30 days) for traceability. Logs exceeding the retention period are batch compressed and archived through background tasks to free up storage resources.

[0175] Based on the above, compared with the existing technology, the present application shows significant advantages in terms of technical value. First, through the dynamic rule engine loading and real-time updating of the machine learning model, the present application can quickly respond to changes in data rules, greatly improving the accuracy and timeliness of data repair. Secondly, the atomic repair mechanism based on the WAL log ensures that it can be safely rolled back even if an error occurs during the repair process, avoiding the problem of dirty data diffusion caused by repair failure in traditional methods. Furthermore, the hierarchical classification data discrimination and multimodal path decision repair method proposed in this application not only solves complex logical contradictions, but also effectively handles missing data, improving the overall data quality and system robustness. In summary, the technological progress of this application is not achieved by a simple combination of technologies, but through a series of innovative designs and technology integrations, a solution with outstanding technical effects has been formed.

[0176] In this embodiment, a data repair device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0177] Figure 5 : is a structural block diagram of a data repair device according to an embodiment of the present application. Figure 5 As shown, the data repair device includes:

[0178] a determination module 52 configured to determine abnormal second data in the first data according to a data rule engine, wherein the data rules applied in the data rule engine are configured according to a data business scenario corresponding to the first data;

[0179] The repair module 54 is used to repair the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0180] The above-described device determines abnormal second data within first data based on a data rule engine, wherein the data rules applied in the data rule engine are configured based on the data business scenario corresponding to the first data. The second data is then repaired using a preset processing method, wherein the preset processing method includes a data prediction generation algorithm. The data rule engine and the data prediction generation algorithm are updated in real time via a preset monitoring method. This solves the technical problem in related technologies where data repair methods cannot be updated in real time with changes in the repaired data, resulting in poor data repair results, thereby improving the data repair effect.

[0181] In an exemplary embodiment, the device also includes a monitoring module, which is used to: monitor a first event through a first thread to obtain the data rules, and update the data rule engine through the data rules, wherein the first event is used to indicate that there is a configuration change in the data rules; and monitor a second event through a second thread to obtain the data prediction generation algorithm, and update the data prediction generation algorithm to the preset processing method, wherein the second event is used to indicate that there is a change in the data prediction generation algorithm.

[0182] In an exemplary embodiment, the determination module is further used to: perform a format check on the first data through the data rule engine to determine that third data with a format abnormality exists in the first data, wherein the data rules applied in the data rule engine include: a data format rule and a first repair rule corresponding to the data format rule, a data logic rule and a second repair rule corresponding to the data logic rule; perform a logic check on the fourth data through the data rule engine to determine that fifth data with a logical abnormality exists in the fourth data, wherein the fourth data is the data in the first data other than the third data; and determine the third data and the fifth data as the second data.

[0183] In an exemplary embodiment, the determination module is also used to: check the target format information of the first data according to the data format rule through the data rule engine; repair the sixth data with abnormal target format information in the first data according to the first repair rule, and use the sixth data that failed to be repaired as the third data.

[0184] In an exemplary embodiment, the determination module is further configured to: perform a logic check on the fourth data according to the data logic rule by the data rule engine, and use the data in the fourth data that fails the logic check as the fifth data.

[0185] In an exemplary embodiment, the repair module is further used to: divide the second data into first dynamic data and first static data according to data type; repair the first dynamic data through the data prediction generation algorithm, and repair the first static data through target auditing.

[0186] In an exemplary embodiment, the repair module is further used to: perform local data dimension expansion on the first static data; repair the seventh data according to the logical conflict type of the seventh data to obtain eighth data, wherein the seventh data includes: the first static data that failed to be expanded and the first dynamic data; split the eighth data into second dynamic data and second static data, and update the first dynamic data and the first static data respectively through the second dynamic data and the second static data.

[0187] In an exemplary embodiment, the repair module is further used to: update the first sub-data through an external interface; wherein the first sub-data is the data in the seventh data whose logical conflict type is a single data conflict; and repair the second sub-data through a second repair rule, and update the third sub-data through the external interface, wherein the second sub-data is the data in the seventh data whose logical conflict type is a conflict between data, and the third sub-data is the data in the second sub-data that fails to be repaired by the second repair rule; and the first sub-data that fails to be updated and the third sub-data that fails to be updated are determined as the eighth data.

[0188] In an exemplary embodiment, the repair module is also used to: decompose the first dynamic data into target components, wherein the target components include: trend component, seasonal component and residual component; determine multiple single-component algorithms for processing the target components according to the component strength of the target components; and fuse the first output results after processing the target components by the multiple single-component algorithms to obtain a second output result, wherein the second output result is a repair result of the first dynamic data.

[0189] In an exemplary embodiment, the device also includes a data rollback module, which is used to: establish a database write-ahead log corresponding to the first data, and back up the first data to the database write-ahead log, wherein the database write-ahead log is used to record any operation performed on the first data through the data rule engine and the preset processing method, and verify the operation result of the any operation; the database write-ahead log is also used to roll back the first data if the verification result of the operation result is a verification failure.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0191] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0192] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0193] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0194] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0195] S1, determining abnormal second data in first data according to a data rule engine, wherein the data rules applied in the data rule engine are configured according to a data business scenario corresponding to the first data;

[0196] S2. Repair the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0197] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0198] Optionally, in this embodiment, the above-mentioned electronic device can also be configured to execute the above-mentioned step S1 through a computer program, determine the abnormal second data in the first data according to the data rule engine, wherein the data rules applied in the data rule engine are configured according to the data business scenario corresponding to the first data; S2, repair the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

[0199] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0200] An embodiment of the present application further provides another computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above method embodiments are implemented.

[0201] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above method embodiments.

[0202] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0203] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0204] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data repair method, characterized in that: include: Determining, according to a data rule engine, abnormal second data in the first data, wherein the data rules applied in the data rule engine are configured according to a data business scenario corresponding to the first data; The second data is repaired through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

2. The data repair method according to claim 1, characterized in that: Before determining, according to the data rule engine, the abnormal second data in the first data, the method further includes: monitoring a first event through a first thread to obtain the data rule, and updating the data rule engine through the data rule, wherein the first event is used to indicate that a configuration change has occurred in the data rule; and The second event is monitored through the second thread to obtain the data prediction generation algorithm and update the data prediction generation algorithm to the preset processing method, wherein the second event is used to indicate that there is a change in the data prediction generation algorithm.

3. The data repair method according to claim 1, wherein: Determining, according to a data rule engine, abnormal second data in the first data, including: Performing a format check on the first data by the data rule engine to determine whether third data with an abnormal format exists in the first data, wherein the data rules applied in the data rule engine include: a data format rule and a first repair rule corresponding to the data format rule, a data logic rule and a second repair rule corresponding to the data logic rule; Performing a logic check on the fourth data by the data rule engine to determine whether fifth data with a logical anomaly exists in the fourth data, wherein the fourth data is data in the first data other than the third data; The third data and the fifth data are determined as the second data.

4. The data repair method according to claim 3, characterized in that: Performing a format check on the first data by the data rule engine to determine whether third data with an abnormal format exists in the first data includes: checking, by the data rule engine, target format information of the first data according to the data format rule; The sixth data having target format information abnormality in the first data is repaired according to the first repair rule, and the sixth data on which the repair fails is used as the third data.

5. The data repair method according to claim 3, characterized in that: Performing a logic check on the fourth data by the data rule engine to determine whether fifth data with a logical anomaly exists in the fourth data includes: The data rule engine performs a logic check on the fourth data according to the data logic rule, and uses the data in the fourth data that fails the logic check as the fifth data.

6. The data repair method according to claim 1, wherein: Repairing the second data by a preset processing method includes: dividing the second data into first dynamic data and first static data according to data type; The first dynamic data is repaired by the data prediction generation algorithm, and the first static data is repaired by a target audit method.

7. The data repair method according to claim 6, characterized in that: Before repairing the first dynamic data using the data prediction generation algorithm, the method further includes: Performing local data dimension expansion on the first static data; Repairing the seventh data according to the logical conflict type of the seventh data to obtain eighth data, wherein the seventh data includes: the first static data that failed to expand the dimension and the first dynamic data; The eighth data is split into second dynamic data and second static data, and the first dynamic data and the first static data are updated respectively by the second dynamic data and the second static data.

8. The data repair method according to claim 7, characterized in that: Repairing the seventh data according to the logical conflict type of the seventh data to obtain eighth data includes: Updating the first sub-data through the external interface; wherein the first sub-data is the data in the seventh data whose logical conflict type is a single data conflict; and Repairing the second sub-data using the second repair rule, and updating the third sub-data through the external interface, wherein the second sub-data is data in the seventh data whose logical conflict type is a data conflict, and the third sub-data is data in the second sub-data that fails to be repaired by the second repair rule; The first sub-data that failed to be updated and the third sub-data that failed to be updated are determined as the eighth data.

9. The data repair method according to claim 6, characterized in that: Repairing the first dynamic data by using the data prediction generation algorithm includes: Decomposing the first dynamic data into target components, wherein the target components include: a trend component, a seasonal component, and a residual component; determining a plurality of single-component algorithms for processing the target component according to the component strength of the target component; The first output results after the target component is processed by the multiple single-component algorithms are fused to obtain a second output result, wherein the second output result is a repair result of the first dynamic data.

10. The data repair method according to claim 1, wherein: Before determining, according to the data rule engine, the abnormal second data in the first data, the method further includes: Establish a database write-ahead log corresponding to the first data, and back up the first data to the database write-ahead log, wherein the database write-ahead log is used to record any operation performed on the first data through the data rule engine and the preset processing method, and verify the operation result of the any operation; the database write-ahead log is also used to roll back the first data if the verification result of the operation result is a verification failure.

11. A data repair device, characterized in that: include: a determination module, configured to determine abnormal second data in the first data according to a data rule engine, wherein the data rules applied in the data rule engine are configured according to a data business scenario corresponding to the first data; A repair module is used to repair the second data through a preset processing method, wherein the preset processing method includes: a data prediction generation algorithm, wherein the data rule engine and the data prediction generation algorithm are updated in real time through a preset monitoring method.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 10 when executed.

13. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

14. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 10 when the computer program is executed by a processor.

Citation Information

Cited By

  • Method and device for automatically normalizing road network data

    CN121563325A