A retail omni-channel transaction data big data cleaning system

Abstract

The application discloses a retail full-channel transaction data big data cleaning system and relates to the technical field of big data cleaning.The system comprises a multi-channel data acquisition module, a traffic buffer module, an elastic resource scheduling module, an intelligent data cleaning module and a data quality monitoring and alarming module; historical data of a data source is pulled, new data is acquired by calling a timestamp window, standardized source data is sent to a corresponding subject partition and sequentially written in a log form to a disk for traffic buffering, a traffic peak period is determined according to a threshold comparison result, elastic resource scheduling is performed by calling an underlying cloud platform and intelligent data cleaning is performed, full-process core indexes are collected and real-time calculation is performed on the full-process core indexes.Through threshold comparison of a full-process core index quality evaluation value, the elastic resource scheduling module is further linked to perform emergency elastic resource scheduling, so that the problem that system stability is reduced under a traffic flood peak and data is lost in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of big data cleaning technology, and in particular to a big data cleaning system for retail omnichannel transaction data. Background Technology

[0002] As businesses expand their operations across diverse channels such as online e-commerce, live-streaming sales, and private mini-programs, transaction data sources become extremely dispersed and require extremely high real-time performance. Especially during major promotional events or flash sales, the instantaneous write pressure on the system can reach hundreds or even thousands of times the daily peak traffic. However, many traditional or early-built data cleaning architectures are often based on fixed physical servers or traditional monolithic databases, with a static upper limit to their processing capacity. When traffic surges, this rigid architecture cannot quickly expand computing or storage resources, leading to a backlog in the data ingestion pipeline. A more critical issue is that, to ensure real-time reporting and inventory synchronization, systems typically employ a memory-first, fast write strategy, where data resides briefly in a memory buffer before being confirmed for disk writes. Once the request backlog exceeds the system's preset load threshold, it triggers service degradation or even a crash. At this point, transaction details in the buffer that have not yet been persisted to disk are permanently lost with the process crash. This loss is not only a technical data deficiency, but also directly leads to serious business consequences such as unbalanced accounts, distorted sales performance calculations, and incorrect distribution of user points or coupons. Therefore, existing big data cleaning systems for retail omnichannel transaction data suffer from decreased system stability and data loss under traffic surges. Summary of the Invention

[0003] This application provides a retail omnichannel transaction data big data cleaning system, which solves the problem of decreased system stability and data loss under traffic surges in existing retail omnichannel transaction data big data cleaning systems, and achieves the effect of improving system stability.

[0004] This application provides a retail omnichannel transaction data big data cleaning system, including: Multi-channel data acquisition module: Establishes a connection with the online data source based on the preset database connection string, pulls historical data from the data source, calls the timestamp window to obtain new data, and standardizes the historical data and new data to record as standardized source data; Traffic buffer module: Marks all retail business types, creates different topic partitions in the message queue according to the retail business type, sends standardized source data to the corresponding topic partitions and writes it to disk in the form of logs, and other modules actively call standardized source data from the queue in pull mode; Elastic resource scheduling module: Creates data cleaning tasks based on standardized source data traffic, monitors the backlog of standardized source data in the message queue, compares the backlog of standardized source data with a threshold, determines the peak traffic period based on the threshold comparison result, calls the underlying cloud platform for elastic resource scheduling, monitors the backlog of standardized source data in the message queue again, if the backlog of standardized source data is less than the preset backlog, gradually shrinks and releases idle computing resources, if the backlog of standardized source data is greater than or equal to the preset backlog, connects to the intelligent data cleaning module; Intelligent data cleaning module: pulls standardized source data from the message queue, removes invalid data from the standardized source data, associates customer master data through key fields, constructs a unified customer identifier for preliminary cleaning, calls intelligent model to supplement missing values, and performs desensitization processing on sensitive information. The processed standardized source data is written to the memory buffer. When the memory buffer reaches the preset size or preset time threshold, the processed standardized source data is written to the final storage file. Data quality monitoring and alarm module: Collects core indicators from various modules throughout the entire process, collects and optimizes the logs of the entire process, performs real-time calculations on the core indicators of the entire process, and obtains the quality assessment value of the core indicators of the entire process. When the quality assessment value of the core indicators of the entire process is greater than or equal to the preset quality threshold, the system automatically triggers an alarm and links the elastic resource scheduling module to perform emergency elastic resource scheduling.

[0005] Furthermore, a connection is established with the online data source based on a preset database connection string, historical data from the data source is retrieved, and the changed data is obtained by calling the timestamp window, including: A secure connection is established with the online data source using a preset database connection string and encryption credentials. The system uses paginated queries to retrieve all historical data from the source database and records the current time as a watermark. Window queries based on timestamp fields are executed at a fixed frequency to obtain the changed data within that window. The full historical data includes: order master table data, order detail data, and payment transaction data. The modified data includes: data after addition, status update data, and attribute change data.

[0006] Furthermore, all retail business types are tagged, and different topic partitions are created in the message queue based on the retail business type, including: Enumerate and define all retail business types supported by the system, assign a globally unique topic name to each business type, create corresponding topic partitions in the message queue middleware according to these topic names, and configure an appropriate number of partitions for each topic based on the expected data throughput. Establish a routing mapping table between service types and topic partitions, and accurately route messages to the corresponding topic partitions based on the service type.

[0007] Furthermore, data cleaning tasks are created based on standardized source data traffic, and the backlog of standardized source data in the message queue is monitored, including: Create an independent data cleaning task for each business theme defined in the traffic buffer module. Each task is bound to one or more theme partitions to be consumed, and the initial parallelism is set. The system queries the current offset and partition end offset of each message queue on each topic partition at a fixed frequency, calculates the difference between the current offset and the partition end offset and records it as the backlog of that partition, and adds up the backlog of each partition to obtain the standardized source data backlog of the business topic.

[0008] Furthermore, a threshold comparison is performed on the backlog of standardized source data. Based on the threshold comparison results, peak traffic periods are determined, and the underlying cloud platform is invoked for elastic resource scheduling, including: The backlog of standardized source data is compared with a preset backlog threshold. If the backlog of standardized source data is greater than or equal to the preset backlog threshold, it is determined to be a peak traffic period. The specific process of calling the underlying cloud platform for elastic resource scheduling is as follows: Based on the total backlog of the backlog business topic and the single concurrent processing speed, calculate the theoretical number of concurrent processes required to clear the current backlog, combine the total number of partitions of the topic to determine the target parallelism, call the cloud platform scaling API, poll the cloud platform API to query the status of the new instance, and wait for the new resources to be ready.

[0009] Furthermore, standardized source data is pulled from the message queue, and invalid data in the standardized source data is removed, including: Standardized source data from each business topic partition is pulled in batches from the message queue of the traffic buffer module. Each cleaning task instance consumes one or more specified partitions and records the consumption offset. For each piece of standardized source data, the following invalid data filtering rules are executed sequentially: field null value validation, format validation, logical validation, and test data filtering. Records that fail the validation are discarded and the discard log is recorded.

[0010] Furthermore, by linking key fields to customer master data, a unified customer identifier is constructed for initial cleansing, including: Extract key fields for identifying customers from the standardized source data after removing invalid data, and use the key fields as joint query conditions to initiate a query request to the customer master data storage to retrieve whether there is a corresponding customer master data record. If a matching customer master data record is found, its unique customer identifier is directly obtained and associated with the current data record. If no matching record is found, it is determined that the data corresponds to an unknown customer. A temporary unique customer identifier is generated for the unknown customer according to preset rules, and the temporary record is written to the pending queue. The data record after invalid data removal and association with the unique customer identifier is recorded as the result of the preliminary cleaning and output.

[0011] Furthermore, the intelligent model is invoked to fill in missing values, and sensitive information is anonymized. The processed, standardized source data is then written into a memory buffer, including: Fields with missing values ​​that still exist in data records that have completed invalid data removal and unique customer identification association are encapsulated into feature vectors and then a pre-trained intelligent completion model is called to perform inference and prediction. After the intelligent completion model outputs the completion value, it is backfilled into the corresponding field, and the source of the completion for that field is marked. For sensitive fields in data records, the field type is identified according to the preset desensitization rule table, and the corresponding desensitization algorithm is called to convert them; Complete data records of all cleaning steps are written in batches to the memory state buffer.

[0012] Furthermore, when the memory buffer reaches a preset size or a preset time threshold, the processed, standardized source data is written to the final storage file, including: The system continuously monitors the current size of the memory buffer and the time data stays in the buffer, and compares it in real time with preset trigger thresholds, including write size thresholds and time thresholds. When the buffer reaches a preset size or a preset time threshold, the system will submit the data in the buffer in batches and write it to the final storage file of the distributed file system in columnar or row-based storage format. The system submits the consumption offset of the currently processed data to the message queue, then clears the memory buffer to prepare for receiving the next batch of data. If the write fails, a retry mechanism is triggered. If the retry still fails, a notification is sent to the data quality monitoring and alarm module to record the anomaly. The buffer data is retained without submitting the offset, waiting for reprocessing.

[0013] Furthermore, core metrics are collected from various modules throughout the entire process, and the process logs are collected and optimized. Real-time calculations are performed on these core metrics to obtain quality assessment values ​​for the entire process, including: Monitoring probes are pre-embedded at key processing nodes of each module to actively collect core indicator values ​​of each module at a fixed frequency, including data access latency and throughput of the multi-channel data acquisition module, backlog of each topic partition of the traffic buffer module, current parallelism and resource utilization of the elastic resource scheduling module, and processing latency and discard rate of the intelligent data cleaning module. Runtime logs are collected in real time from the log files of each module and pushed to the log aggregation center. Indexes are created by module name and business theme. Unstructured raw logs are converted into structured fields through log parsing rules, and redundant logs and duplicate stack information are removed. The core indicators of the entire process are calculated in real time to obtain the quality assessment value of the core indicators of the entire process. The specific process is as follows: ;in, This is the quality assessment value of the core indicators throughout the entire process. For the i-th core indicator value, is the weight coefficient of the i-th core indicator value, and n is the number of core indicator values.

[0014] The technical effects and advantages of the big data cleaning system for retail omnichannel transaction data of this invention are as follows: 1. The system employs a timestamp-based left-open, right-closed interval query, coupled with a displacement commit confirmation mechanism, to ensure that incremental data is neither repeatedly retrieved nor missed. The full historical data is fetched in pages to avoid system memory overflow caused by loading massive amounts of data at once. When writing data, a multi-replica synchronization mechanism with write-ahead logs is used to ensure that each piece of data has been saved by at least two physical nodes before a successful confirmation is returned. Even if a single node fails, the data can be recovered from other replicas, achieving a recovery point target close to zero.

[0015] 2. By calling the scaling up and down interface of the underlying cloud platform, the elastic expansion of computing resources can be completed within seconds. The parallelism after expansion is capped at the smaller of twice the current parallelism and the total number of topic partitions, ensuring that the expansion operation is effective under the constraint of the number of partitions and avoiding resource waste caused by unlimited expansion.

[0016] 3. Through end-to-end data lineage tracing and structured log indexing, when the system malfunctions, operations and maintenance personnel can quickly pinpoint the specific business theme, processing stage, and time point of the problem, significantly shortening troubleshooting time. Attached Figure Description

[0017] Figure 1 A schematic diagram of a big data cleaning system for retail omnichannel transaction data provided in this application embodiment; Figure 2 This is a schematic diagram illustrating the process of marking all retail business types and establishing different topic partitions in the message queue based on the retail business type, as provided in the embodiments of this application. Figure 3 This is a schematic diagram illustrating the process of calling the underlying cloud platform for elastic resource scheduling, as provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This application example provides a big data cleaning system for retail omnichannel transaction data. The multi-channel data acquisition module connects to online data sources via database connections, pulls full historical data in pages, and continuously pulls incremental data in timestamp windows. After standardizing both, the data is output. The traffic buffer module establishes independent topic partitions in the message queue according to retail business types, routes standardized source data to the corresponding partitions, and sequentially writes the data to disk in log form for downstream retrieval. The elastic resource scheduling module creates cleaning tasks for each business topic and monitors the backlog of each partition. When the backlog continuously exceeds the threshold, it calls the cloud platform interface to expand the capacity (the target parallelism is double the smaller of the current parallelism and the total number of partitions). When the backlog decreases, it gradually shrinks the capacity. The intelligent data cleaning module pulls data from each partition and sequentially performs invalid data filtering, format standardization, customer master data association to build a unified identifier, intelligent model missing value completion, and sensitive information desensitization. After cleaning, the data is written to the memory buffer in batches. When the data reaches the preset size or time threshold, it is written to the distributed file system in columnar storage. The data quality monitoring and alarm module collects the core indicators of the entire process, calculates the weighted quality assessment value in real time, and triggers an alarm when the preset threshold is reached, and links the elastic scheduling module to expand the capacity urgently. The entire process, from data collection, buffering, scheduling, cleaning to monitoring and alerting, forms a closed loop, achieving triple protection of high availability without crashes, zero data loss, and on-demand elastic scaling of resources.

[0020] Please see Figure 1 This is a flowchart illustrating a retail omnichannel transaction data big data cleaning system provided in an embodiment of this application. The method includes the following steps: Multi-channel data acquisition module: Establishes a connection with the online data source based on the preset database connection string, pulls historical data from the data source, calls the timestamp window to obtain new data, and standardizes the historical data and new data to record as standardized source data; Traffic buffer module: Marks all retail business types, creates different topic partitions in the message queue according to the retail business type, sends standardized source data to the corresponding topic partitions and writes it to disk in the form of logs, and other modules actively call standardized source data from the queue in pull mode; Elastic resource scheduling module: Creates data cleaning tasks based on standardized source data traffic, monitors the backlog of standardized source data in the message queue, compares the backlog of standardized source data with a threshold, determines the peak traffic period based on the threshold comparison result, calls the underlying cloud platform for elastic resource scheduling, monitors the backlog of standardized source data in the message queue again, if the backlog of standardized source data is less than the preset backlog, gradually shrinks and releases idle computing resources, if the backlog of standardized source data is greater than or equal to the preset backlog, connects to the intelligent data cleaning module; Intelligent data cleaning module: pulls standardized source data from the message queue, removes invalid data from the standardized source data, associates customer master data through key fields, constructs a unified customer identifier for preliminary cleaning, calls intelligent model to supplement missing values, and performs desensitization processing on sensitive information. The processed standardized source data is written to the memory buffer. When the memory buffer reaches the preset size or preset time threshold, the processed standardized source data is written to the final storage file. Data quality monitoring and alarm module: Collects core indicators from various modules throughout the entire process, collects and optimizes the logs of the entire process, performs real-time calculations on the core indicators of the entire process, and obtains the quality assessment value of the core indicators of the entire process. When the quality assessment value of the core indicators of the entire process is greater than or equal to the preset quality threshold, the system automatically triggers an alarm and links the elastic resource scheduling module to perform emergency elastic resource scheduling.

[0021] Furthermore, a connection is established with the online data source based on a preset database connection string, historical data from the data source is retrieved, and the changed data is obtained by calling the timestamp window, including: A secure connection is established with the online data source using a preset database connection string and encryption credentials. The system uses paginated queries to retrieve all historical data from the source database and records the current time as a watermark. Window queries based on timestamp fields are executed at a fixed frequency to obtain the changed data within that window. The full historical data includes: order master table data, order detail data, and payment transaction data. The modified data includes: data after addition, status update data, and attribute change data.

[0022] In this embodiment, when a retail enterprise first connects to the cleaning system, it establishes a secure connection with the online MySQL transaction database through a preset JDBC connection string (containing the database IP, port, instance name, and encrypted account password). Then, it uses paginated queries (e.g., fetching 1000 records at a time, looping with the auto-incrementing primary key ID as the pagination condition) to completely fetch all historical records from the order master table (order number, order time, payment amount, etc.), the order details table (product SKU, unit price, quantity, etc.), and the payment transaction table (payment channel, third-party transaction number, payment amount, etc.) to the cleaning pipeline, and records the current time as the water level in the Redis database.

[0023] The system switches to incremental synchronization mode, performing a window query based on the timestamp field every 5 minutes. If successful, it retrieves order records that have changed within the past 5 minutes, including newly generated orders (new data), orders that have changed from "pending payment" to "paid" (status update data), and orders with modified delivery addresses (attribute change data). After successfully writing to the target storage, the watermark is pushed up to the current time, and this process is repeated in a loop.

[0024] Full pagination retrieval avoids memory overflow caused by loading tens of millions of data points at once. The left-open, right-closed time window design ensures that incremental data is neither duplicated nor omitted. Index-based timestamp queries keep the pressure on the source database under control. The waterline advancement mechanism, combined with failure retry, ensures the integrity of data synchronization. That is, if a write fails, the waterline is not advanced, and the same window is retrieved again in the next task. Overall, it achieves efficient, lossless, and continuous collection of online transaction data, providing complete and timely data raw materials for downstream cleaning and business analysis.

[0025] Furthermore, all retail business types are tagged, and different topic partitions are created in the message queue based on the retail business type, including: Enumerate and define all retail business types supported by the system, assign a globally unique topic name to each business type, create corresponding topic partitions in the message queue middleware according to these topic names, and configure an appropriate number of partitions for each topic based on the expected data throughput. Establish a routing mapping table between service types and topic partitions, and accurately route messages to the corresponding topic partitions based on the service type.

[0026] In this embodiment, please refer to Figure 2 This document illustrates the process of marking all retail business types and establishing different topic partitions in the message queue based on the retail business type, as provided in this application embodiment. During system initialization, the architect enumerates and defines six core business types: orders, refunds, payment records, product master data, member master data, and inventory changes. A globally unique topic name is assigned to each type. Subsequently, these six topics are created sequentially, and the number of partitions for each topic is configured according to the expected data throughput. For example, the peak daily throughput of the order topic is 20,000 messages per second. Based on the single partition throughput limit of 2,000 messages per second and reserving a 50% margin, it is configured with 16 partitions. The refund topic and payment topic have lower throughput and are each configured with 4 partitions. The product, member, and inventory master data topics have low change frequency and are each configured with 2 partitions.

[0027] The system also maintains an internal routing mapping table. When the multi-channel data acquisition module generates an order data, the system accurately sends it to the order partition according to the mapping table, while the refund data is sent to the refund partition, ensuring that data of different business types enter their respective isolated channels without interference.

[0028] Physical isolation of data streams for different business types ensures that order surges do not crowd out processing resources for refunds or inventory channels, achieving fault isolation and resource assurance between businesses. Differentiated configuration of the number of partitions allows for precise matching of resource allocation with business volume, avoiding resource waste or bottlenecks. Downstream cleaning modules can be expanded independently by topic, adding computing resources only to backlogged business channels, thereby achieving fine-grained elastic scheduling. At the same time, as the basic unit of parallel consumption, the number of partitions determines the maximum parallelism limit of cleaning tasks, providing clear constraints for horizontal scaling and effectively improving the overall throughput and manageability of the system.

[0029] Furthermore, data cleaning tasks are created based on standardized source data traffic, and the backlog of standardized source data in the message queue is monitored, including: Create an independent data cleaning task for each business theme defined in the traffic buffer module. Each task is bound to one or more theme partitions to be consumed, and the initial parallelism is set. The system queries the current offset and partition end offset of each message queue on each topic partition at a fixed frequency, calculates the difference between the current offset and the partition end offset and records it as the backlog of that partition, and adds up the backlog of each partition to obtain the standardized source data backlog of the business topic.

[0030] In this embodiment, if the creation of the six business topics in the traffic buffer module has been completed, the system will automatically create an independent data cleaning task for each topic when it starts up, setting the initial parallelism to 16, and configuring 2 parallelisms for each of the other main data topics.

[0031] After the task starts, the system calls the management client API every 5 seconds at a fixed frequency to query the current offset and the offset at the end of each partition for each consumer group. For example, if the offset at the end of the order partition is 10000 and the current offset is 8000, the difference is calculated to be 2000 backlog entries for that partition. The backlog entries of the 16 partitions are added together to get the total backlog entry for the order topic as 32000 entries, while the total backlog entry for the order partition is only 500 entries. The system records the backlog quantification value for each topic for subsequent threshold determination.

[0032] Each business theme has its own independent cleaning task and consumer group, achieving resource and fault isolation between businesses. Order surges will not affect the cleaning progress of refund data. The initial parallelism is aligned with the number of partitions, ensuring that each partition has an independent consumer instance, maximizing the parallel throughput capacity brought by the partitions. Partition-level backlog monitoring can accurately identify which specific partition the backlog occurs in, providing data basis for accurately locating hot partitions during subsequent elastic scaling. At the same time, the backlog, as a quantitative indicator of the degree of data congestion, provides a unique and reliable trigger signal for the elastic scheduling module, making scaling decisions accurate and effective, and avoiding resource waste caused by blind scaling.

[0033] Furthermore, a threshold comparison is performed on the backlog of standardized source data. Based on the threshold comparison results, peak traffic periods are determined, and the underlying cloud platform is invoked for elastic resource scheduling, including: The backlog of standardized source data is compared with a preset backlog threshold. If the backlog of standardized source data is greater than or equal to the preset backlog threshold, it is determined to be a peak traffic period. The specific process of calling the underlying cloud platform for elastic resource scheduling is as follows: Based on the total backlog of the backlog business topic and the single concurrent processing speed, calculate the theoretical number of concurrent processes required to clear the current backlog, combine the total number of partitions of the topic to determine the target parallelism, call the cloud platform scaling API, poll the cloud platform API to query the status of the new instance, and wait for the new resources to be ready.

[0034] In this embodiment, please refer to Figure 3 The diagram below illustrates the process of calling the underlying cloud platform for elastic resource scheduling in this application embodiment. The system monitors that the total backlog of order topics has climbed to 120,000, while the preset backlog threshold for this topic is 100,000, and the threshold has been exceeded for three consecutive samplings. The system determines that the order business channel has entered a peak traffic period.

[0035] Subsequently, the elastic resource scheduling module obtains the current topic cleaning task's parallelism as 8, single concurrent processing speed as 500 records per second, and total number of partitions as 16. It calculates that the theoretical concurrency required to clear the current backlog of 120,000 records is 120,000 / 500 = 240 seconds. If it is cleared within this time, the required parallelism is 120,000 / (500 × 240) = 10. Taking the smaller value between twice the current parallelism of 8 and the total number of partitions of 16, the target parallelism is determined to be 16.

[0036] The system then calls the Scale sub-resource API in Kubernetes to expand the Deployment, adjusting the number of replicas of the order cleaning task from 8 to 16, and polling the API every 2 seconds to check the new status. After all 8 newly added replicas are in running status and pass the health check, the consumer group automatically joins the topic and starts pulling data from the allocated partition. After the expansion is completed, the backlog is monitored, and the system gradually shrinks down to release resources after the backlog continues to fall below the safe level.

[0037] The continuous sampling judgment mechanism effectively filters out instantaneous fluctuations in backlog volume, avoiding unnecessary expansion operations triggered by brief traffic spikes; the theoretical concurrency calculation combines actual processing capacity and expected clearing time, providing a clear quantitative basis for expansion decisions; the target parallelism limit is constrained by the total number of partitions, ensuring that expansion operations are always within an effective range and will not generate idle consumer instances due to exceeding the number of partitions; the polling waiting mechanism guarantees the integrity and observability of expansion operations, with new instances only starting to consume after they are fully ready, avoiding data duplication or omission in a half-startup state; targeted expansion is only executed on backlogged order topics, without affecting the normal operation of other business channels, achieving precise on-demand allocation of computing resources.

[0038] Furthermore, standardized source data is pulled from the message queue, and invalid data in the standardized source data is removed, including: Standardized source data from each business topic partition is pulled in batches from the message queue of the traffic buffer module. Each cleaning task instance consumes one or more specified partitions and records the consumption offset. For each piece of standardized source data, the following invalid data filtering rules are executed sequentially: field null value validation, format validation, logical validation, and test data filtering. Records that fail the validation are discarded and the discard log is recorded.

[0039] In this embodiment, it is assumed that the order cleaning task pulls a batch of standardized source data from the partition and records the current consumption offset as 8000. Then, it iterates through this batch of data one by one, and performs invalid data filtering rules on each record, including but not limited to, if the order number field of a record is empty, it triggers a null value validation failure and is directly discarded; if the order time format of a record does not meet the system requirements, it triggers a format validation failure and is discarded; if the order amount of a record is -100 yuan, it triggers a logic validation failure and is discarded (the amount cannot be negative); if the order source of a record is marked as a test environment, it triggers a test data filtering rule and is discarded. All valid records that pass the validation enter the subsequent standardization process, and each discarded record is recorded in JSON format as a discard log (including the discard time, the topic partition, the reason for discarding, and the original data summary) and written to a separate log topic for operation and maintenance auditing.

[0040] Multi-layered verification rules isolate dirty data at the cleaning entry point, preventing invalid data carrying null values ​​or incorrect formats from entering subsequent standardization, master data association, and AI completion stages, thus reducing the waste of resources caused by invalid calculations. Test data is automatically filtered in the production environment to ensure that business analysis reports are not contaminated by test orders. The partition-level retrieval method allows multiple instances to consume different partitions in parallel, maximizing the throughput capacity brought by partitions. The offset recording mechanism ensures that consumers can continue consuming from the last interrupted position after a failure restart, avoiding data duplication or omission. The complete record of discard logs provides auditable original evidence for data quality traceability and rule optimization, helping operations and maintenance personnel to continuously optimize filtering rules.

[0041] Furthermore, by linking key fields to customer master data, a unified customer identifier is constructed for initial cleansing, including: Extract key fields for identifying customers from the standardized source data after removing invalid data, and use the key fields as joint query conditions to initiate a query request to the customer master data storage to retrieve whether there is a corresponding customer master data record. If a matching customer master data record is found, its unique customer identifier is directly obtained and associated with the current data record. If no matching record is found, it is determined that the data corresponds to an unknown customer. A temporary unique customer identifier is generated for the unknown customer according to preset rules, and the temporary record is written to the pending queue. The data record after invalid data removal and association with the unique customer identifier is recorded as the result of the preliminary cleaning and output.

[0042] In this embodiment, it is assumed that the order cleaning task pulls an order record with invalid data removed from the message queue. This record contains two customer identification fields: mobile phone number and membership card number. The system uses these two fields as joint query conditions to initiate a query request to the customer master data storage cached in the Redis database to check if there is a corresponding customer master data record. If a unified customer identifier is found in the customer master data record corresponding to the mobile phone number, the identifier is directly associated with the current order record. If no matching record is found, it is determined that the order comes from an unknown customer. A temporary unique customer identifier is generated according to the preset snowflake algorithm and associated with the current record. The temporary customer information is written to the pending processing queue for subsequent manual or system merging processing. The order record that has completed invalid data filtering and unique customer identifier association is marked as the preliminary cleaning result and output to the subsequent standardized process.

[0043] By enabling joint queries of key fields, customer data from various online and offline channels can be integrated, allowing transactions from the same customer across different platforms to be uniformly aggregated under a unique customer identifier. This provides a data foundation for building a 360-degree customer view and precision marketing. When a match is successful, existing master data is directly reused to avoid duplicate creation, ensuring the uniqueness and consistency of customer identifiers. The fallback mechanism of generating temporary identifiers when a match fails ensures that transaction data of unknown customers will not be blocked in the data cleaning process due to the inability to associate with master data, guaranteeing the continuous smooth flow of data. The pending queue provides a traceable list of pending processes for subsequent customer merging and master data supplementation, creating a closed loop for data governance.

[0044] Furthermore, the intelligent model is invoked to fill in missing values, and sensitive information is anonymized. The processed, standardized source data is then written into a memory buffer, including: Fields with missing values ​​that still exist in data records that have completed invalid data removal and unique customer identification association are encapsulated into feature vectors and then a pre-trained intelligent completion model is called to perform inference and prediction. After the intelligent completion model outputs the completion value, it is backfilled into the corresponding field, and the source of the completion for that field is marked. For sensitive fields in data records, the field type is identified according to the preset desensitization rule table, and the corresponding desensitization algorithm is called to convert them; Complete data records of all cleaning steps are written in batches to the memory state buffer.

[0045] In this embodiment, assuming that an order record that has completed invalid data removal and customer identification association has a missing district-level information in the delivery address field, the system encapsulates the province and city fields of the record, as well as contextual information such as the order source channel and product category, into a feature vector. The system then calls a pre-trained natural language processing address parsing model to perform inference and prediction. The model outputs the district-level information as a completion value to fill back into the field and marks the source of the completion. At the same time, the mobile phone number and ID card number contained in the record are identified as sensitive fields by a preset desensitization rule table, and are converted into desensitized values ​​by a masking algorithm. After completing the above completion and desensitization, the system writes the complete order record in batches into the memory state buffer maintained by the task.

[0046] The intelligent completion model uses contextual information to infer and fill in missing values, effectively improving data integrity and usability, and avoiding deviations in downstream logistics or regional sales analysis caused by incomplete address information. The completion source tag provides a traceable basis for subsequent manual verification and model iteration. The anonymization process ensures that customer privacy is not leaked throughout the entire data cleaning and storage process, meeting the compliance requirements of personal information protection regulations, and enabling the cleaned data to be safely used in scenarios such as reporting and precision marketing. Batch writing to the memory buffer, combined with the subsequent dual-threshold disk push triggering mechanism, effectively reduces performance overhead and significantly improves overall throughput efficiency.

[0047] Furthermore, when the memory buffer reaches a preset size or a preset time threshold, the processed, standardized source data is written to the final storage file, including: The system continuously monitors the current size of the memory buffer and the time data stays in the buffer, and compares it in real time with preset trigger thresholds, including write size thresholds and time thresholds. When the buffer reaches a preset size or a preset time threshold, the system will submit the data in the buffer in batches and write it to the final storage file of the distributed file system in columnar or row-based storage format. The system submits the consumption offset of the currently processed data to the message queue, then clears the memory buffer to prepare for receiving the next batch of data. If the write fails, a retry mechanism is triggered. If the retry still fails, a notification is sent to the data quality monitoring and alarm module to record the anomaly. The buffer data is retained without submitting the offset, waiting for reprocessing.

[0048] In this embodiment, it is assumed that the memory buffer of the order cleaning task is configured with a write size threshold of 10,000 records or 64MB and a time threshold of 60 seconds. The system continuously monitors the current accumulated data volume in the buffer, which is 9,800 records, occupies approximately 58MB of memory, and the earliest data has been in the buffer for 55 seconds. At this time, no trigger condition has been met, and new data continues to be received. When the next batch of 200 data records is written, the accumulated data volume reaches 10,000 records (reaching the size threshold). The system immediately triggers the disk write operation, submitting the 10,000 completely cleaned order records in a columnar storage format to the distributed file system. The system submits the consumption offset of the currently processed data and then clears the memory buffer to prepare for receiving the next batch of data.

[0049] If a write timeout error is returned during this write process due to network jitter, the system will automatically trigger a retry mechanism and resubmit the write request after a 2-second interval. If the retry fails after 3 attempts, the exception information will be encapsulated as an alarm notification and sent to the data quality monitoring and alarm module to record the exception. At the same time, 10,000 data entries in the buffer will be retained without submitting the offset, and will be reprocessed after the fault is recovered.

[0050] The dual-threshold triggering mechanism (data volume threshold and time threshold are both met before data is written to disk) strikes a balance between the efficiency of batch writing and the real-time nature of data. The data volume threshold ensures high throughput efficiency for large-volume writing, while the time threshold prevents data from remaining in the buffer for a long time during low-traffic periods, which would lead to excessive latency. This fundamentally eliminates the risk of data loss or duplicate processing. The abnormal alarm linkage monitoring module ensures the observability of faults and timely response.

[0051] Furthermore, core metrics are collected from various modules throughout the entire process, and the process logs are collected and optimized. Real-time calculations are performed on these core metrics to obtain quality assessment values ​​for the entire process, including: Monitoring probes are pre-embedded at key processing nodes of each module to actively collect core indicator values ​​of each module at a fixed frequency, including data access latency and throughput of the multi-channel data acquisition module, backlog of each topic partition of the traffic buffer module, current parallelism and resource utilization of the elastic resource scheduling module, and processing latency and discard rate of the intelligent data cleaning module. Runtime logs are collected in real time from the log files of each module and pushed to the log aggregation center. Indexes are created by module name and business theme. Unstructured raw logs are converted into structured fields through log parsing rules, and redundant logs and duplicate stack information are removed. The core indicators of the entire process are calculated in real time to obtain the quality assessment value of the core indicators of the entire process. The specific process is as follows: ;in, This is the quality assessment value of the core indicators throughout the entire process. For the i-th core indicator value, is the weight coefficient of the i-th core indicator value, and n is the number of core indicator values.

[0052] In this embodiment, it is assumed that during the monitoring window at 10:00 AM, the system collects the following metrics data through Prometheus monitoring probes embedded in the key nodes of the five modules: the data access latency of the multi-channel data acquisition module is 8 seconds (threshold 30 seconds), and the throughput is 12,000 records per second (expected 15,000 records); the total backlog of order topics in the traffic buffer module is 60,000 records (threshold 100,000 records); the current parallelism of the elastic resource scheduling module is 8, and the resource utilization rate is 65%; the processing latency of the intelligent data cleaning module is 45 seconds (threshold 120 seconds), and the discard rate is 0.02%.

[0053] Meanwhile, the log collector collects the running logs of each module in real time, creates indexes by module name and business theme, and converts unstructured logs (such as "Order No. 12345 cleaned, time taken 12ms") into structured fields ({"order_id": "12345", "module": "clean", "duration": 12, "level": "INFO"}) through parsing rules, and removes DEBUG level logs and duplicate stack information to reduce storage pressure.

[0054] The streaming engine then calculates the health score for each indicator in a 1-minute rolling window: Acquisition latency health score is (1-8 / 30)×100=73.3 points, throughput health score is min(12000 / 15000×100, 100)=80 points, backlog health score is (1-60000 / 100000)×100=40 points, parallelism matching score is min(8 / 12×100, 100)=66.7 points, resource utilization health score is 100-|65-70|=95 points, and cleaning latency health score is (1-45 / 120)× 100 = 62.5 points, and the discard rate health score is (1-2 / 10000)×100 = 99.98 points. After weighting and summing according to the preset weights (backlog 30%, cleaning delay 20%, resource utilization 15%, collection delay 10%, parallelism matching degree 10%, discard rate 10%, throughput 5%), the comprehensive quality assessment value is 73.3×0.1+80×0.05+40×0.3+66.7×0.1+95×0.15+62.5×0.2+99.98×0.1=66.75 points, which is lower than the preset quality threshold of 80 points, so no alarm is triggered.

[0055] End-to-end metric collection enables comprehensive visualization of system operation status. Backlog volume, with its highest weight, directly reflects the risk of congestion. The weighted evaluation model integrates multiple dimensions of health to avoid misjudgment based on a single metric. Structured logs and indexing provide accurate retrieval capabilities for anomaly localization. Alarm thresholds and weight coefficients can be adjusted online, allowing the system to flexibly adapt to the sensitivity of different business scenarios. Operations personnel can quickly determine the health status of the entire link through quality assessment values, significantly reducing the time cost of fault discovery and troubleshooting.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0061] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A big data cleaning system for retail omnichannel transaction data, characterized in that, include: Multi-channel data acquisition module: Establishes a connection with the online data source based on the preset database connection string, pulls historical data from the data source, calls the timestamp window to obtain new data, and standardizes the historical data and new data to record as standardized source data; Traffic buffer module: Marks all retail business types, creates different topic partitions in the message queue according to the retail business type, sends standardized source data to the corresponding topic partitions and writes it to disk in the form of logs, and other modules actively call standardized source data from the queue in pull mode; Elastic resource scheduling module: Creates data cleaning tasks based on standardized source data traffic, monitors the backlog of standardized source data in the message queue, compares the backlog of standardized source data with a threshold, determines the peak traffic period based on the threshold comparison result, calls the underlying cloud platform for elastic resource scheduling, monitors the backlog of standardized source data in the message queue again, if the backlog of standardized source data is less than the preset backlog, gradually shrinks and releases idle computing resources, if the backlog of standardized source data is greater than or equal to the preset backlog, connects to the intelligent data cleaning module; Intelligent data cleaning module: pulls standardized source data from the message queue, removes invalid data from the standardized source data, associates customer master data through key fields, constructs a unified customer identifier for preliminary cleaning, calls intelligent model to supplement missing values, and performs desensitization processing on sensitive information. The processed standardized source data is written to the memory buffer. When the memory buffer reaches the preset size or preset time threshold, the processed standardized source data is written to the final storage file. Data quality monitoring and alarm module: Collects core indicators from various modules throughout the entire process, collects and optimizes the logs of the entire process, performs real-time calculations on the core indicators of the entire process, and obtains the quality assessment value of the core indicators of the entire process. When the quality assessment value of the core indicators of the entire process is greater than or equal to the preset quality threshold, the system automatically triggers an alarm and links the elastic resource scheduling module to perform emergency elastic resource scheduling.

2. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The process of establishing a connection with the online data source based on a preset database connection string, retrieving historical data from the data source, and calling a timestamp window to obtain the changed data includes: A secure connection is established with the online data source using a preset database connection string and encryption credentials. The system uses paginated queries to retrieve all historical data from the source database and records the current time as a watermark. Window queries based on timestamp fields are executed at a fixed frequency to obtain the changed data within that window. The full historical data includes: order master table data, order detail data, and payment transaction data. The modified data includes: data after addition, status update data, and attribute change data.

3. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The labeling of all retail business types, and the creation of different topic partitions in the message queue based on the retail business type, including: Enumerate and define all retail business types supported by the system, assign a globally unique topic name to each business type, create corresponding topic partitions in the message queue middleware according to these topic names, and configure an appropriate number of partitions for each topic based on the expected data throughput. Establish a routing mapping table between service types and topic partitions, and accurately route messages to the corresponding topic partitions based on the service type.

4. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The process of creating a data cleaning task based on standardized source data traffic and monitoring the backlog of standardized source data in the message queue includes: Create an independent data cleaning task for each business theme defined in the traffic buffer module. Each task is bound to one or more theme partitions to be consumed, and the initial parallelism is set. The system queries the current offset and partition end offset of each message queue on each topic partition at a fixed frequency, calculates the difference between the current offset and the partition end offset and records it as the backlog of that partition, and adds up the backlog of each partition to obtain the standardized source data backlog of the business topic.

5. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The process of comparing the backlog of standardized source data to a threshold, determining peak traffic periods based on the threshold comparison results, and then invoking the underlying cloud platform for elastic resource scheduling includes: The backlog of standardized source data is compared with a preset backlog threshold. If the backlog of standardized source data is greater than or equal to the preset backlog threshold, it is determined to be a peak traffic period. The specific process of calling the underlying cloud platform for elastic resource scheduling is as follows: Based on the total backlog of the backlog business topic and the single concurrent processing speed, calculate the theoretical number of concurrent processes required to clear the current backlog, combine the total number of partitions of the topic to determine the target parallelism, call the cloud platform scaling API, poll the cloud platform API to query the status of the new instance, and wait for the new resources to be ready.

6. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The step of retrieving standardized source data from the message queue and removing invalid data from the standardized source data includes: Standardized source data from each business topic partition is pulled in batches from the message queue of the traffic buffer module. Each cleaning task instance consumes one or more specified partitions and records the consumption offset. For each piece of standardized source data, the following invalid data filtering rules are executed sequentially: field null value validation, format validation, logical validation, and test data filtering. Records that fail the validation are discarded and the discard log is recorded.

7. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The initial cleansing process involves linking customer master data through key fields to construct a unified customer identifier, including: Extract key fields for identifying customers from the standardized source data after removing invalid data, and use the key fields as joint query conditions to initiate a query request to the customer master data storage to retrieve whether there is a corresponding customer master data record. If a matching customer master data record is found, its unique customer identifier is directly obtained and associated with the current data record. If no matching record is found, it is determined that the data corresponds to an unknown customer. A temporary unique customer identifier is generated for the unknown customer according to preset rules, and the temporary record is written to the pending queue. The data record after invalid data removal and association with the unique customer identifier is recorded as the result of the preliminary cleaning and output.

8. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The process of calling the intelligent model to fill in missing values, de-sensitizing sensitive information, and writing the processed standardized source data into a memory buffer includes: Fields with missing values ​​that still exist in data records that have completed invalid data removal and unique customer identification association are encapsulated into feature vectors and then a pre-trained intelligent completion model is called to perform inference and prediction. After the intelligent completion model outputs the completion value, it is backfilled into the corresponding field, and the source of the completion for that field is marked. For sensitive fields in data records, the field type is identified according to the preset desensitization rule table, and the corresponding desensitization algorithm is called to convert them; Complete data records of all cleaning steps are written in batches to the memory state buffer.

9. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The step of writing the processed standardized source data to the final storage file when the memory buffer reaches a preset size or a preset time threshold includes: The system continuously monitors the current size of the memory buffer and the time data stays in the buffer, and compares it in real time with preset trigger thresholds, including write size thresholds and time thresholds. When the buffer reaches a preset size or a preset time threshold, the system will submit the data in the buffer in batches and write it to the final storage file of the distributed file system in columnar or row-based storage format. The system submits the consumption offset of the currently processed data to the message queue, then clears the memory buffer to prepare for receiving the next batch of data. If the write fails, a retry mechanism is triggered. If the retry still fails, a notification is sent to the data quality monitoring and alarm module to record the anomaly. The buffer data is retained without submitting the offset, waiting for reprocessing.

10. The retail omnichannel transaction data big data cleaning system as described in claim 1, characterized in that, The process involves collecting core metrics from various modules, gathering and optimizing full-process logs, and performing real-time calculations on these core metrics to obtain a quality assessment value for each core metric. This includes: Monitoring probes are pre-embedded at key processing nodes of each module to actively collect core indicator values ​​of each module at a fixed frequency, including data access latency and throughput of the multi-channel data acquisition module, backlog of each topic partition of the traffic buffer module, current parallelism and resource utilization of the elastic resource scheduling module, and processing latency and discard rate of the intelligent data cleaning module. Runtime logs are collected in real time from the log files of each module and pushed to the log aggregation center. Indexes are created by module name and business theme. Unstructured raw logs are converted into structured fields through log parsing rules, and redundant logs and duplicate stack information are removed. The core indicators of the entire process are calculated in real time to obtain the quality assessment value of the core indicators of the entire process. The specific process is as follows: ;in, This is the quality assessment value of the core indicators throughout the entire process. For the i-th core indicator value, is the weight coefficient of the i-th core indicator value, and n is the number of core indicator values.

Images