A dual-channel dirty data processing system and method based on dynamic priority

Abstract

The present invention provides a dual-channel dirty data processing system and method based on dynamic priority, comprising the following steps: a dirty data monitoring module monitors the data status in the database in real time; a dynamic priority assessment module automatically prioritizes data; a hot repair channel module repairs data in real time; a cold processing channel module asynchronously batch processes data; a channel coordination mechanism isolates resources for hot repairs and controls the data versions of hot repairs and cold processing in real time; and a logging and monitoring module is responsible for real-time recording and monitoring of relevant information about dirty data and its processing process. The present invention has the following beneficial effects: a dynamic priority scheduling mechanism, hierarchical processing, and resource isolation architecture address the problem that traditional database dirty data processing methods cannot flexibly process dirty data based on its urgency and importance.

Description

Technical Field

[0001] The present invention belongs to the technical field of database management, and in particular relates to a dual-channel dirty data processing system and method based on dynamic priority. Background Art

[0002] In database systems, data consistency and integrity are crucial. However, due to various unforeseen factors, such as hardware failures, software errors, network problems, etc., some data may become dirty. The presence of dirty data will affect the performance and accuracy of the database and the reliability of business decisions. Traditional dirty data processing methods often adopt a single processing method, which cannot be flexibly processed according to the urgency and importance of dirty data, resulting in low processing efficiency and may even affect the normal operation of the system. Therefore, there is an urgent need for a system and method that can efficiently process dirty data based on its dynamic priority. In the existing technology, dirty data cleaning usually adopts an offline batch processing mode, which has the following defects:

[0003] 1. Significant business impact: Full scans or table locks cause online business interruptions (for example, order tables are frequently locked during e-commerce promotions).

[0004] 2. Low repair efficiency: Relying on manual rule-making, it cannot dynamically adapt to changes in data distribution (for example, new abnormal data types are not handled in a timely manner).

[0005] 3. Sensitive data risk: Failure to prioritize data leads to mixing high-risk data (such as user payment information) with low-frequency data, increasing the risk of leakage. Summary of the Invention

[0006] In view of this, the present invention aims to propose a dual-channel dirty data processing system and method based on dynamic priority to solve the problem that traditional database dirty data processing methods cannot flexibly process dirty data according to the urgency and importance of dirty data.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] In the first aspect, the present invention proposes a dual-channel dirty data processing system based on dynamic priority, including a dirty data monitoring module, a dynamic priority evaluation module, a hot repair channel module, a cold processing channel module, a channel coordination mechanism and a log recording and monitoring module. The dirty data monitoring module outputs dirty data information to the dynamic priority evaluation module, and the dynamic priority evaluation module outputs dirty data information to the hot repair channel module or the cold processing channel module. The channel coordination mechanism exchanges data information with the hot repair channel module and the cold processing channel module respectively. The dirty data monitoring module, dynamic priority evaluation module, hot repair channel module, cold processing channel module and channel coordination mechanism all output log information to the log recording and monitoring module.

[0009] In a second aspect, based on the same concept, the present invention further proposes a dual-channel dirty data processing method based on dynamic priority, comprising the following steps:

[0010] S1. The dirty data monitoring module monitors the data status in the database in real time: it captures dirty data information based on preset rules and algorithms and sends it to the dynamic priority evaluation module;

[0011] S2. After receiving the dirty data information, the dynamic priority assessment module automatically prioritizes the data based on data access frequency and field sensitivity, and sends the dirty data information to the hot repair channel module or the cold processing channel module according to the data priority;

[0012] S3, after receiving high-priority data information, the hot repair channel module repairs the data in real time;

[0013] S4. After receiving the low-priority data information, the cold processing channel module performs asynchronous batch processing on the data;

[0014] S5. The channel coordination mechanism isolates resources in the hot repair thread pool and controls the data versions of hot repair and cold processing in real time.

[0015] S6, the logging and monitoring module is responsible for real-time recording and monitoring of relevant information about dirty data and its processing process. Further, in step S1, the dirty data monitoring module monitors the data status in the database in real time, including: S11, the data acquisition layer captures change events through database logs;

[0016] S12. Verify the changed data using preset rules;

[0017] S13. If dirty data information is captured, it is sent to the logging and monitoring module and the dynamic priority evaluation module.

[0018] Furthermore, in step S2, data priority is automatically divided based on data access frequency and field sensitivity, including:

[0019] S21. Receive dirty data information sent by the dirty data monitoring module;

[0020] S22. The dynamic weight adjustment mechanism performs priority judgment. The formula of the priority expression is as follows:

[0021] Priority value = (α × sensitivity coefficient) + (β × access frequency) + (γ × last modification time decay factor) + (δ × data magnitude);

[0022] Where α is the sensitivity coefficient weight, β is the access frequency weight, γ is the time decay factor weight, and δ is the data magnitude factor weight. The sensitivity coefficient is automatically mapped based on the field type. The access frequency statistical period is the number of queries within 24 hours. The last modified time decay factor expression is as follows: 1 / (1+e^(-λ(t_current-t_last_modified))).

[0023] Where t_current is the current time of data evaluation, t_last_modified is the time point of the most recent change, and λ is the adjustment coefficient;

[0024] The formula for the data magnitude expression is as follows:

[0025] Data level = log10(V) / log10(1TB);

[0026] Where V is the actual data volume of the field or table;

[0027] S23. If the priority is P0-P1, the data information is sent to the hot repair channel module;

[0028] S24. If the priority is P2, the data information is sent to the cold processing channel module;

[0029] S25. Send the log information to the log recording and monitoring module.

[0030] Furthermore, in step S3, after receiving the high-priority data information, the hot repair channel module performs real-time repair on the data, including:

[0031] S31, receiving data information sent by the dynamic priority evaluation module;

[0032] S32. If the priority is P0, the data is stored in an independent memory area and replaced with a legal value;

[0033] S33. If the priority is P1, check whether there is any unrepaired data with a priority of P0. If so, proceed to step S32. Otherwise, cache the data with a priority of P1 to the corresponding memory area and replace it with a legal value.

[0034] S34. Double write transaction log: Utilize a lock-free hash table, adopt compare-and-exchange operations to implement memory data replacement, and pre-allocate a memory pool; adopt a double-write log consistency protocol to write to the database transaction log and independent repair log at the same time, and complete the atomic commit of the two through the log location identifier, and send the log information to the logging and monitoring module.

[0035] Furthermore, in step S4, after receiving the low-priority data information, the cold processing channel module performs asynchronous batch processing on the data, including:

[0036] S41, receiving data information sent by the dynamic priority evaluation module;

[0037] S42, distributed task sharding: Divide the data information with priority P2 into multiple shards and distribute them to multiple nodes for parallel processing through the Kafka message queue;

[0038] S43. Batch mode optimization: merge multiple updates into a single batch SQL, and merge single updates into batch operations; S44. Asynchronous verification: generate a data snapshot before updating. If the repair fails, roll back to the snapshot version. After the cold treatment is completed, the repair result is verified through sampling. Failed tasks are automatically retried and an alarm notification is triggered. If they are retried three times, they are marked as "manual review" tasks.

[0039] Furthermore, in step S5, the channel coordination mechanism isolates resources of the hot repair thread pool and controls the data versions of hot repair and cold processing in real time, including:

[0040] S51, Resource Isolation: The hotfix thread pool is independent of the business thread pool, and the maximum concurrency is limited through the Hystrix-style isolation strategy;

[0041] S52, Memory Area Division: Hot fix uses Java Virtual Machine off-heap memory;

[0042] S53. Data version control: Use multi-version concurrent control, hot fixes modify the current active version, cold fixes are based on historical snapshot versions, and conflicts are prevented through global version numbers.

[0043] Furthermore, in step S6, the logging and monitoring module is responsible for real-time recording and monitoring of the relevant information of dirty data and its processing process, including:

[0044] S61, receiving data information sent by the dirty data monitoring module;

[0045] S62: Receive repair information from the hot repair channel module and the cold processing channel module, and mark the repair fields and processing results;

[0046] S63. Receive channel coordination mechanism version information and mark the data version number;

[0047] S64. If the backlog of data in the cold processing queue exceeds 100,000, capacity expansion will be performed.

[0048] Compared with the prior art, the dual-channel dirty data processing system and method based on dynamic priority described in the present invention has the following advantages:

[0049] Advantage 1: Dynamic priority scheduling mechanism

[0050] Based on the business impact weight (e.g., transaction amount, user volume) and sensitivity level (e.g., payment information, privacy fields) of dirty data, processing priorities are dynamically assigned, and task scheduling is driven by a real-time weight calculation model. Business Impact Weight: Quantifies the impact of dirty data on core business by real-time statistics of business indicators associated with dirty data (e.g., order amount, user activity). Sensitivity Level: Based on data classification and grading strategies, high-risk fields (e.g., payment card numbers) are marked and assigned high priority.

[0051] The corresponding beneficial effects are as follows:

[0052] (1) Addressing the “large business impact” defect: high-risk data (such as payment information) is prioritized to avoid table locks or long transactions blocking online business; low-priority tasks (such as historical order error correction) are deferred to reduce business interruption time.

[0053] (2) Solution to the defect of “low repair efficiency”: Dynamically adjust the weights to dynamically adapt to changes in data distribution and promptly handle new abnormal data types.

[0054] (3) Solve the defects of “sensitive data risk”: distinguish data priorities and avoid mixing operations with ordinary data.

[0055] Advantage 2: Hierarchical processing and resource isolation architecture

[0056] We employ hierarchical processing channels, segregating resources based on data sensitivity and business importance, and optimizing storage access through hot and cold data diversion. This channel isolation design ensures that high-risk data is processed in real time through high-speed caches (such as Redis), while low-frequency data is processed in batches through object storage. Our resource allocation strategy ensures that high-priority tasks exclusively occupy compute nodes, while low-priority tasks share idle resources, improving resource utilization.

[0057] The corresponding beneficial effects are as follows:

[0058] (1) Solve the defect of "large business impact": Online business is physically isolated from low-priority repair tasks, table lock operations are reduced by 100%, and database availability reaches 99.99%.

[0059] (2) Solve the defects of "sensitive data risk": Sensitive data such as payment information is encrypted separately, and full-link access rights are controlled to reduce compliance risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0061] Figure 1 A schematic diagram of the execution flow between the functional modules according to an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the dirty data monitoring module execution process according to an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the execution flow of the dynamic priority evaluation module according to an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the execution flow of the hot repair channel module according to an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram of the execution process of the cold treatment channel module according to an embodiment of the present invention;

[0066] Figure 6 The figure is a schematic diagram of the execution flow of the channel coordination mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0068] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more. In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0069] like Figure 1 As shown, a dual-channel dirty data processing system and method based on dynamic priority include a dirty data monitoring module, a dynamic priority evaluation module, a hot repair channel module, a cold processing channel module, a channel coordination mechanism and a log recording and monitoring module. The dirty data monitoring module outputs dirty data information to the dynamic priority evaluation module, and the dynamic priority evaluation module outputs dirty data information to the hot repair channel module or the cold processing channel module. The channel coordination mechanism exchanges data information with the hot repair channel module and the cold processing channel module respectively. The dirty data monitoring module, dynamic priority evaluation module, hot repair channel module, cold processing channel module and channel coordination mechanism all output log information to the log recording and monitoring module.

[0070] A dual-channel dirty data processing method based on dynamic priority includes the following steps:

[0071] S1. The dirty data monitoring module monitors the data status in the database in real time: it captures dirty data information based on preset rules and algorithms and sends it to the dynamic priority evaluation module;

[0072] S2. After receiving the dirty data information, the dynamic priority assessment module automatically prioritizes the data based on data access frequency and field sensitivity, and sends the dirty data information to the hot repair channel module or the cold processing channel module according to the data priority;

[0073] S3, after receiving high-priority data information, the hot repair channel module repairs the data in real time;

[0074] S4. After receiving the low-priority data information, the cold processing channel module performs asynchronous batch processing on the data;

[0075] S5. The channel coordination mechanism isolates resources in the hot repair thread pool and controls the data versions of hot repair and cold processing in real time.

[0076] S6, the log recording and monitoring module is responsible for real-time recording and monitoring of the relevant information of dirty data and its processing process. In this embodiment, Figure 1 As shown in the figure, the dirty data monitoring module enters the dynamic weight adjustment mechanism through the dynamic priority evaluation module, and determines whether to enter the hot repair channel module or the cold treatment channel module through the dynamic weight adjustment mechanism. The hot repair channel module and the cold treatment channel module are both regulated by the channel coordination mechanism.

[0077] The specific implementation is as follows:

[0078] 1. Dirty Data Monitoring Module

[0079] The dirty data monitoring module monitors the data status in the database system in real time and detects possible dirty data by matching it with preset rules and algorithms. Once dirty data is found, the relevant information is immediately sent to the priority assessment module. Figure 2 As shown, the data collection layer determines whether the data is normal data through preset rules. If it is normal data, the monitoring ends directly. If it is abnormal data, it is sent to the dynamic priority evaluation module and the log recording and monitoring module respectively. The specific process is as follows:

[0080] (1) The data collection layer captures change events through database logs;

[0081] (2) Verify the changed data using preset rules, such as the mobile phone number is not 11 digits, the amount exceeds 2 decimal places, etc.;

[0082] (3) If abnormal data is captured, it is sent to the logging and monitoring module and the dynamic priority assessment module.

[0083] 2. Dynamic Priority Evaluation Module

[0084] After receiving dirty data information, the dynamic priority assessment module automatically classifies the data into levels (P0-P2) based on data access frequency and field sensitivity. For example, the order amount field is marked as P0, and the mobile phone number field in the user login table is marked as P1. By comprehensively considering the various attributes of dirty data and business needs, each dirty data item is assigned a priority value.

[0085] A dynamic weight adjustment mechanism is introduced in priority calculation:

[0086] Priority value = (α × sensitivity coefficient) + (β × access frequency) + (γ × last modification time decay factor) + (δ × data magnitude);

[0087] in:

[0088] (1) 'Sensitivity coefficient' is automatically mapped based on the field type (e.g. payment information = 1.5, user ID = 1.2, general field = 1.0);

[0089] (2) The statistical period of 'access frequency' is the number of queries within 24 hours;

[0090] (3) 'Last modification time decay factor' formula:

[0091] 1 / (1+e^(-λ(t_current-t_last_modified)));

[0092] t_current: The current time, used as the reference point for calculating time differences, represents the current moment of data evaluation. t_last_modified: The last modification time, which records the freshness of the data and reflects the time of its most recent change.

[0093] λ is the adjustment coefficient, which controls the decay rate of the time difference.

[0094] Typical value: 0.01≤λ≤0.5;

[0095] The larger the λ is, the steeper the S-curve becomes. A small change in the time difference (t_current - t_last_modified) will cause the attenuation factor to fluctuate dramatically.

[0096] Example: When λ=0.5, if the data is not modified within 1 hour, the decay factor

[0097] ≈1 / (1+e^(-0.5*1))≈0.73; if it is not modified for 2 hours, the attenuation factor

[0098] ≈1 / (1+e^(-0.5*2))≈0.59.

[0099] The smaller λ is, the flatter the curve is and the more gradual the impact of the time difference is.

[0100] Example: When λ = 0.1, the attenuation factor changes less under the same time difference (1 hour ≈ 0.95, 2 hours ≈ 0.91).

[0101] (4) Data volume

[0102] Standardization of data levels:

[0103] Map the raw data magnitude to a uniform numerical range (e.g., 0 to 1) so that it can be added to other parameters (sensitivity coefficient, access frequency):

[0104] Data level = log10(V) / log10(1TB);

[0105] Where V is the actual data volume of the field or table (unit: byte).

[0106] Normalize the data volume to the range of 0 to 1. If it is greater than 1, truncate it and take the value as 1.

[0107] (5)α is the sensitivity coefficient weight

[0108] Function: Controls the contribution of field sensitivity to priority.

[0109] Typical value: 0.5≤α≤1.5;

[0110] Example:

[0111] If the system pays more attention to sensitive data (such as payment information), the α value can be increased (such as α=1.2) to make it easier for highly sensitive fields to be marked as high priority.

[0112] If the business has low privacy requirements, the α value can be lowered (such as α = 0.8) to weaken the sensitivity impact.

[0113] (6)β is the access frequency weight

[0114] Function: Measures the contribution of data access popularity to priority.

[0115] Typical value: 0.3≤β≤1.0;

[0116] Example:

[0117] For frequently accessed hotspot tables (such as order tables), increasing the β value (such as β = 1.0) can prioritize repairing frequently accessed fields and reduce business delays.

[0118] For cold data tables (such as log tables), reduce the β value (such as β = 0.3) to avoid frequent updates interfering with low-frequency services.

[0119] (7)γ is the time decay factor weight

[0120] Function: Adjust the impact of data "freshness" on priority.

[0121] Typical value: 0.1≤γ≤0.5;

[0122] Example:

[0123] If you need to prioritize recently modified data (such as real-time transaction flow), increase the γ value (such as γ = 0.4) to trigger repairs for recently modified fields more quickly.

[0124] If the historical data repair priority is low, the γ value can be reduced (eg, γ = 0.1).

[0125] (8)δ is the data magnitude factor weight

[0126] Function: Used to adjust the impact of data size on priority

[0127] Typical value: 0≤δ≤0.5

[0128] If δ = 0.5, when the data level is 1, the contribution to the priority is 0.5 × 1 = 0.5;

[0129] If δ = 0, the influence of data magnitude is completely ignored.

[0130] Business adaptation:

[0131] In high-concurrency write scenarios (such as e-commerce order tables): Increase the δ value (for example, δ = 0.4) to prioritize processing large-scale data and avoid write bottlenecks.

[0132] In low-frequency analysis scenarios (such as historical log tables): Reduce the δ value (for example, δ = 0.1) to prevent large table repairs from occupying too many resources.

[0133] Through the dynamic weight adjustment mechanism, the system can flexibly adapt to the requirements of different scenarios (such as real-time vs. cost sensitivity) and achieve refined control of dynamic priorities.

[0134] like Figure 3 As shown in the figure, the specific processing flow of the dynamic priority evaluation module is as follows:

[0135] (1) Receive data sent by the dirty data monitoring module;

[0136] (2) Dynamic weight adjustment mechanism for priority judgment;

[0137] (3) If the priority is P0-P1, it is sent to the hot repair channel module;

[0138] (4) If the priority is P2, it is sent to the cold processing channel module;

[0139] (5) Send the data and priority records to the logging and monitoring module.

[0140] 3. Hot Repair Channel Module

[0141] Real-time repair of high-priority data (such as frequently accessed tables and key business fields) is performed, and the impact on business is reduced through in-memory computing and lock-free operations. The hot repair channel has the following two characteristics:

[0142] Memoized real-time repair:

[0143] Caching P0-P1 data in high-speed caches such as Redis, Memcached, or local caches enables fast read and write speeds through lock-free hash tables. For example, when a "mobile phone number format error" is detected, the legal value is directly replaced in memory, avoiding frequent disk reads and writes. P0 data has a higher priority and involves more highly sensitive data, so it uses a separate memory area for priority processing.

[0144] Transaction log double write:

[0145] Repair operations are recorded in both the database transaction log and the independent "repair log" to ensure data consistency and traceability.

[0146] Implementation method:

[0147] Lock-free hash table implementation: Uses compare and swap (CAS) operations to replace memory data, combined with a pre-allocated memory pool to reduce garbage collection (GC) pressure.

[0148] Dual-write log consistency protocol: Repair operations must be written to the database transaction log and an independent repair log at the same time, and the log position identifier (Log Position Identifier) ​​is used to ensure that both are committed atomically.

[0149] like Figure 4 As shown, the specific implementation process is:

[0150] (1) receiving data and priority levels sent by the dynamic priority assessment module;

[0151] (2) Determine the priority level. If the priority is P0, store the data in an independent memory area for P0-level processing, replace the legal value in the memory, and record the repair operation in the log recording and monitoring module;

[0152] (3) When the priority is P1, check whether there is any unrepaired P0-level data. If so, proceed to the second step. If not, cache the P1-level data in the memory area processed by the P1 level, replace the legal value in the memory, and record the repair operation to the logging and monitoring module.

[0153] 4. Cold treatment channel module

[0154] For asynchronous batch processing of low-priority data (such as log tables and historical data), distributed task queues and batch submission are used to optimize resource consumption. The cold processing channel has the following three characteristics:

[0155] Distributed task sharding:

[0156] Divide data with priority level P2 into multiple shards and distribute them to multiple nodes for parallel processing through the Kafka message queue.

[0157] Distributed task sharding strategy:

[0158] P2-level data is sharded into Kafka partitions using a consistent hashing algorithm, ensuring that all shards of the same data are processed by the same consumer, avoiding cross-node data contention. The message queue can be Kafka, RabbitMQ, or RocketMQ, without affecting the core logic of the system architecture.

[0159] Batch mode optimization:

[0160] Merge multiple updates into a single batch SQL statement (e.g., 'UPDATE table SET field = CASE id WHEN ... THEN ... END') to reduce database connection overhead.

[0161] Batch SQL optimization example:

[0162]

[0163]

[0164] Merge single updates into batch operations to reduce database connection consumption.

[0165] Asynchronous verification mechanism:

[0166] Generate a data snapshot before updating, and roll back to the snapshot version if a repair fails. After the cold treatment is complete, verify the repair results through sampling (e.g., randomly selecting 5% of the data). Failed tasks are automatically retried and trigger an alert notification. After three retries, the task is marked as "manual review."

[0167] like Figure 5 As shown, the specific process is:

[0168] (1) receiving data and priority levels sent by the dynamic priority assessment module;

[0169] (2) Distribute data to multiple node partitions;

[0170] (3) Generate snapshot version data before updating;

[0171] (4) Perform batch processing to update data;

[0172] (5) If the update fails, roll back to the snapshot version and try again. After three retries, an alarm notification will be triggered and marked as "manual review" in the logging and monitoring module;

[0173] (6) If the update is successful, sample 5% of the data to verify the repair results. If the verification passes, record it in the log recording and monitoring module; if the verification fails, proceed to step 5.

[0174] 5. Channel Collaboration Mechanism

[0175] Resource isolation:

[0176] Allocate independent thread pools and memory areas for hot fixes to avoid competing for resources with business queries.

[0177] Data version control: Hot repair only modifies the current active data version, while cold repair is based on snapshots to prevent mutual interference.

[0178] Implementation method:

[0179] Resource isolation strategy:

[0180] The hotfix thread pool is independent of the business thread pool and uses a Hystrix-style isolation strategy to limit the maximum concurrency. Memory area division: Hotfixes use Java Virtual Machine (JVM) off-heap memory (DirectByteBuffer) to avoid GC pauses affecting business.

[0181] Data version control:

[0182] Multiversion Concurrency Control (MVCC) is used. Hot fixes modify the current active version, while cold fixes are based on historical snapshot versions. Conflicts are prevented through a global version number (e.g., version = timestamp + unique identifier, i.e., version = timestamp + UUID.hashCode()) to ensure global uniqueness.

[0183] like Figure 6 As shown, the specific process is:

[0184] For the hotfix channel:

[0185] (1) Generate the current global version number;

[0186] (2) Read the current global version number as the baseline version for this hotfix;

[0187] (3) When the hot repair channel receives a P0-P1 level data repair task, it allocates independent memory;

[0188] (4) After the repair is completed, a new global version number is generated, the memory version number is updated to the new global version number, and it is synchronized to the logging and monitoring module.

[0189] For cold treatment channels:

[0190] (1) When the cold processing queue receives the P2-level data repair task, it reads the current version number and marks the snapshot version as the current version number;

[0191] (2) After the cold processing task is completed, a new global version number is generated;

[0192] (3) During the execution of the cold treatment task, the hot fix may have updated the global version number. By comparing the version numbers, the data is finally consistent, and the global version number is updated and synchronized to the logging and monitoring modules.

[0193] 6. Logging and Monitoring: The log management module is responsible for recording information about dirty data and the entire processing process, including the time of discovery, priority, processing method, and results. The log management module also monitors and audits the processing process in real time to promptly identify and resolve potential issues. If anomalies are detected or processing results do not meet expectations, the scheduling control module can adjust the processing strategy or reassign tasks based on the log information.

[0194] Audit log content:

[0195] Record field-level repair details.

[0196] Alarm mechanism:

[0197] Set thresholds to trigger capacity expansion (for example, when the backlog in the cold processing queue exceeds 100,000, automatically increase the Kafka consumer instance).

[0198] The specific process is:

[0199] (1) Receive data information sent by the dirty data monitoring module;

[0200] (2) Receive the repair information of the hot repair channel module and the cold processing channel module, and mark the repair field and processing result;

[0201] (3) Receive channel coordination mechanism version information and mark the data version number;

[0202] (4) When the backlog of data in the cold processing queue exceeds 100,000, expansion processing is performed.

[0203] The advantages and beneficial effects of the present invention are as follows:

[0204] Advantage 1: Dynamic priority scheduling mechanism

[0205] Technical features:

[0206] Based on the business impact weight (such as transaction amount, user scale) and sensitivity level (such as payment information, privacy fields) of dirty data, processing priority is dynamically allocated, and task scheduling is driven by a real-time weight calculation model.

[0207] Technical solution analysis:

[0208] Business impact weight: Quantify the impact of dirty data on core business by collecting real-time statistics on business indicators associated with dirty data (such as order amount and user activity).

[0209] Sensitivity level: Based on data classification and grading strategies, high-risk fields (such as payment card numbers) are marked and given high priority.

[0210] Beneficial effects:

[0211] 1. Addressing the issue of “significant business impact”:

[0212] High-risk data (such as payment information) is processed first to avoid table locks or long transactions that block online business.

[0213] Low-priority tasks (such as historical order error correction) are deferred to reduce business interruption time.

[0214] 2. Solution: The defect of "low repair efficiency":

[0215] Dynamically adjust based on weights, dynamically adapt to changes in data distribution, and promptly handle new abnormal data types.

[0216] 3. Addressing the flaws of “sensitive data risk”:

[0217] Differentiate data priorities and avoid mixing operations with normal data.

[0218] Advantage 2: Hierarchical processing and resource isolation architecture

[0219] Technical features:

[0220] Adopt hierarchical processing channels, isolate resources according to data sensitivity and business importance, and optimize storage access through hot and cold data diversion technology.

[0221] Technical solution analysis:

[0222] Channel isolation design: High-risk data is processed in real time through a high-speed cache unit (such as Redis), and low-frequency data is processed in batches through object storage.

[0223] Resource allocation strategy: high-priority tasks exclusively occupy computing nodes, while low-priority tasks share idle resources, thereby improving resource utilization.

[0224] Beneficial effects:

[0225] 1. Addressing the issue of “significant business impact”:

[0226] Online services are physically isolated from low-priority repair tasks, table lock operations are reduced by 100%, and database availability reaches 99.99%.

[0227] 2. Addressing the flaws of “sensitive data risk”:

[0228] Sensitive data such as payment information is encrypted separately, and access rights are controlled throughout the entire process, reducing compliance risks. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A dual-channel dirty data processing method based on dynamic priority, implemented by a dual-channel dirty data processing system based on dynamic priority, characterized by: The dual-channel dirty data processing system based on dynamic priority includes a dirty data monitoring module, a dynamic priority evaluation module, a hot repair channel module, a cold processing channel module, a channel coordination mechanism and a log recording and monitoring module. The dirty data monitoring module outputs dirty data information to the dynamic priority evaluation module, and the dynamic priority evaluation module outputs dirty data information to the hot repair channel module or the cold processing channel module. The channel coordination mechanism exchanges data information with the hot repair channel module and the cold processing channel module respectively. The dirty data monitoring module, the dynamic priority evaluation module, the hot repair channel module, the cold processing channel module and the channel coordination mechanism all output log information to the log recording and monitoring module. The method comprises the following steps: S1. The dirty data monitoring module monitors the data status in the database in real time: it captures dirty data information based on preset rules and algorithms and sends it to the dynamic priority evaluation module; S2. After receiving the dirty data information, the dynamic priority assessment module automatically prioritizes the data based on data access frequency and field sensitivity, and sends the dirty data information to the hot repair channel module or the cold processing channel module according to the data priority; S3, after receiving high-priority data information, the hot repair channel module repairs the data in real time; S4. After receiving the low-priority data information, the cold processing channel module performs asynchronous batch processing on the data; S5. The channel coordination mechanism isolates resources in the hot repair thread pool and controls the data versions of hot repair and cold processing in real time. S6. The logging and monitoring module is responsible for real-time recording and monitoring of the relevant information of dirty data and its processing process.

2. The dual-channel dirty data processing method based on dynamic priority according to claim 1, characterized in that: In step S1, the dirty data monitoring module monitors the data status in the database in real time, including: S11, the data collection layer captures change events through database logs; S12. Verify the changed data using preset rules; S13. If dirty data information is captured, it is sent to the logging and monitoring module and the dynamic priority evaluation module.

3. The dual-channel dirty data processing method based on dynamic priority according to claim 2, characterized in that: In step S2, data is automatically prioritized based on data access frequency and field sensitivity, including: S21. Receive dirty data information sent by the dirty data monitoring module; S22. The dynamic weight adjustment mechanism performs priority judgment. The formula of the priority expression is as follows: Priority value = (α × sensitivity coefficient) + (β × access frequency) + (γ × last modification time decay factor) + (δ × data magnitude); In the formula, α is the sensitivity coefficient weight, β is the access frequency weight, γ is the time decay factor weight, and δ is the data magnitude factor weight. The sensitivity coefficient is automatically mapped based on the field type. The access frequency statistical period is the number of queries within 24 hours. The formula for the most recently modified time decay factor expression is as follows: 1 / (1+e^(-λ(t_current-t_last_modified))); Where t_current is the current time of data evaluation, t_last_modified is the time point of the most recent change, and λ is the adjustment coefficient; The formula for the data magnitude expression is as follows: Data level = log10(V) / log10(1TB); Where V is the actual data volume of the field or table; S23. If the priority is P0-P1, the data information is sent to the hot repair channel module; S24. If the priority is P2, the data information is sent to the cold processing channel module; S25. Send the log information to the log recording and monitoring module.

4. The dual-channel dirty data processing method based on dynamic priority according to claim 3, characterized in that: In step S3, after receiving the high-priority data information, the hot repair channel module performs real-time repair on the data, including: S31, receiving data information sent by the dynamic priority evaluation module; S32. If the priority is P0, the data is stored in an independent memory area and replaced with a legal value; S33. If the priority is P1, check whether there is any unrepaired data with a priority of P0. If so, proceed to step S32. Otherwise, cache the data with a priority of P1 to the corresponding memory area and replace it with a legal value. S34. Double write transaction log: Utilize a lock-free hash table, adopt compare-and-exchange operations to implement memory data replacement, and pre-allocate a memory pool; adopt a double-write log consistency protocol to write to the database transaction log and independent repair log at the same time, and complete the atomic commit of the two through the log location identifier, and send the log information to the logging and monitoring module.

5. The dual-channel dirty data processing method based on dynamic priority according to claim 4, characterized in that: In step S4, after receiving the low-priority data information, the cold processing channel module performs asynchronous batch processing on the data, including: S41, receiving data information sent by the dynamic priority evaluation module; S42, distributed task sharding: Divide the data information with priority P2 into multiple shards and distribute them to multiple nodes for parallel processing through the Kafka message queue; S43, Batch mode optimization: merge multiple updates into a single batch SQL, merge single updates into a batch operation; S44, Asynchronous Verification: Generate a data snapshot before updating. If the repair fails, roll back to the snapshot version. After the cold treatment is completed, the repair results are verified through sampling. Failed tasks are automatically retried and an alarm notification is triggered. If they are retried three times, the task is marked as "manual review".

6. The method for processing dirty data in a dual-channel manner based on dynamic priority according to claim 5, characterized in that: In step S5, the channel coordination mechanism isolates resources in the hot repair thread pool and controls the data versions of hot repair and cold processing in real time, including: S51, Resource Isolation: The hotfix thread pool is independent of the business thread pool, and the maximum concurrency is limited through the Hystrix-style isolation strategy; S52, Memory Area Division: Hot fix uses Java Virtual Machine off-heap memory; S53. Data version control: Use multi-version concurrent control, hot fixes modify the current active version, cold fixes are based on historical snapshot versions, and conflicts are prevented through global version numbers.

7. The dual-channel dirty data processing method based on dynamic priority according to claim 6, characterized in that: In step S6, the logging and monitoring module is responsible for real-time recording and monitoring of dirty data information and its processing process, including: S61, receiving data information sent by the dirty data monitoring module; S62: Receive repair information from the hot repair channel module and the cold processing channel module, and mark the repair fields and processing results; S63. Receive channel coordination mechanism version information and mark the data version number; S64. If the backlog of data in the cold processing queue exceeds 100,000, capacity expansion will be performed.

Images