A big data processing system and method for new retail industry

Abstract

The application discloses a big data processing system and method for a new retail industry, wherein a unified data access module executes CDC capture and stream access on MySQL, MongoDB and SDK buried point data, and outputs the data to a lake warehouse integrated storage module after standardization processing; the lake warehouse integrated storage module realizes stream batch integrated storage based on an ACID transaction, and provides a query interface for a real-time calculation and analysis service module; the real-time calculation and analysis service module writes into Doris after executing calculation and processing through Flink; a cloud native resource and reliability module provides containerized deployment, elastic scaling and fault recovery services based on Kubernetes; a governance and operation module executes blood relationship tracking, monitoring and alarm and permission management on a whole link. The application realizes multi-source unified access, low-delay real-time processing, transaction consistency guarantee and cloud native elastic expansion.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a big data processing system and method for the new retail industry. Background Technology

[0002] With the development of new retail businesses, the integration of online and offline channels has brought about a large amount of data, including transaction data, product data, membership data, inventory data, and offline store behavior tracking. Existing data architectures generally suffer from the following problems:

[0003] (1) Data silos: Business system data is scattered across different databases or log systems, lacking a unified view;

[0004] (2) Insufficient real-time performance: Traditional ETL / offline data warehouses have high latency, making it difficult to support real-time recommendations and real-time risk control;

[0005] (3) Limited scalability: The growth in data volume makes it difficult to expand computing and storage capacity;

[0006] (4) Insufficient consistency and maintainability: lack of transaction guarantee, lineage tracking and unified monitoring and alarm system, making it difficult to locate data problems. Summary of the Invention

[0007] Based on this, the embodiments of this application provide a big data processing system for the new retail industry, which solves the problems of scattered data access, insufficient real-time computing power, fragmented storage and query, difficulty in expansion and consistency guarantee in existing systems, thereby improving business capabilities such as real-time recommendation, intelligent risk control and precision marketing.

[0008] Firstly, a big data processing system for the new retail industry is provided. This system includes a unified data access module, a lake warehouse integrated storage module, a real-time computing and analysis service module, a cloud-native resource and reliability module, and a governance and operation and maintenance module, among which:

[0009] The unified data access module is used to capture and stream data from MySQL, MongoDB and SDK tracking points using CDC, and output the data to the lake warehouse integrated storage module after standardization processing.

[0010] The lake warehouse integrated storage module is used to receive data from the unified data access module, realize stream and batch integrated storage based on ACID transactions, and provide a query interface for the real-time computing and analysis service module;

[0011] The real-time computing and analysis service module is used to obtain real-time streaming data from the unified data access module, perform computing and processing through Flink, and write it to Doris to achieve sub-second OLAP queries.

[0012] The cloud-native resource and reliability module is used to provide Kubernetes containerized deployment, elastic scaling and fault recovery services for the unified data access module, lake warehouse integrated storage module and real-time computing and analysis service module.

[0013] The governance and operation module is used to perform lineage tracing, monitoring and alarm management, and access control for the unified data access module, lake warehouse integrated storage module, real-time computing and analysis service module, and cloud-native resource and reliability module.

[0014] Optionally, the unified data access module specifically includes:

[0015] The data source configuration unit is used to manage the connection parameters and access strategies of MySQL, MongoDB, and SDK tracking systems.

[0016] The CDC capture unit is used to capture change data for MySQL and MongoDB and output incremental data to the lake warehouse integrated storage module in real time.

[0017] The streaming access unit is used to establish a millisecond-level streaming channel for SDK embedded data;

[0018] The standardization processing unit is used to perform schema mapping, field standardization, and quality verification.

[0019] Optionally, the lake warehouse integrated storage module specifically includes:

[0020] A stream-batch integrated storage engine is used to support batch writing of historical data and real-time writing of CDC / tracking stream data;

[0021] The transaction management unit is used to enable ACID transactions to ensure concurrent read-write consistency.

[0022] The intelligent optimization unit is used to perform automatic hierarchical storage and columnar storage optimization;

[0023] The lifecycle management unit is used to perform data archiving and expiration cleanup according to preset strategies.

[0024] Optionally, the real-time computing and analysis service module specifically includes:

[0025] Flink's real-time computing engine is used to perform data cleaning, aggregation, feature calculation, and complex event processing, and achieves exactly-Once semantic guarantees through the Checkpoint mechanism.

[0026] The Doris analytics engine receives Flink computation results and uses vectorized execution and materialized views to achieve sub-second OLAP queries.

[0027] The application interface layer provides standardized query interfaces for real-time data dashboards and reporting systems.

[0028] Optionally, the cloud-native resource and reliability module specifically includes:

[0029] Kubernetes orchestration units are used to deploy compute and storage components and enable elastic scaling.

[0030] High availability assurance unit, used to perform multi-active deployment and automatic failover;

[0031] The fault-tolerant recovery unit is used to ensure data reliability through Flink state recovery and storage of multiple replicas for redundancy.

[0032] Optionally, the governance and operation module specifically includes:

[0033] The lineage tracking unit is used to record information on the entire data flow process.

[0034] The metadata management unit is used for unified management of data schema and asset catalog;

[0035] The monitoring and alarm unit is used to collect and process latency, throughput, and error rate metrics and trigger alarms.

[0036] The Data Services and Permissions Unit is used to provide a unified API gateway and fine-grained permission control.

[0037] Optionally, the unified data access module is connected to the lake warehouse integrated storage module through a message queue, the real-time computing and analysis service module obtains real-time streaming data from the unified data access module through a data subscription method, and the cloud-native resource and reliability module establishes resource scheduling relationships with each module through a container orchestration interface.

[0038] Optionally, the Flink real-time computing engine supports event-time-based window computing, and the Doris analytics engine supports mixed loads of real-time import and batch import, and optimizes query execution plans through dynamic partitioning and pruning.

[0039] Secondly, a big data processing method for the new retail industry is provided, implemented in any of the systems described in the first aspect above, the method comprising:

[0040] CDC change capture is performed on MySQL and MongoDB, and streaming access is performed on SDK data points. After standardization processing, a standardized data stream is formed and output to the lake warehouse integrated storage module.

[0041] The system receives the standardized data stream, implements integrated stream and batch storage based on ACID transactions, and provides a query interface for the real-time computing and analysis service module.

[0042] Real-time streaming data is acquired, processed by Flink, and then written to Doris to achieve sub-second OLAP queries.

[0043] Provides Kubernetes containerized deployment, elastic scaling, and fault recovery services;

[0044] Perform lineage tracing, monitoring and alerting, and access control across the entire supply chain.

[0045] Optionally, performing CDC capture and streaming access on multi-source data through the unified data access module specifically includes:

[0046] Enable CDC change data capture for MySQL and MongoDB, parse the database transaction logs to obtain incremental data, and ensure that the capture latency is controlled within the second level;

[0047] A high-throughput streaming channel is established for SDK data tracking, and real-time distribution is achieved through a message queue to ensure that the end-to-end transmission latency reaches the millisecond level.

[0048] The system performs automatic schema mapping, field type standardization, business timestamp alignment, and integrity verification on the heterogeneous data received, forming a standardized data stream that is output to the lake warehouse integrated storage module.

[0049] The beneficial effects of the technical solutions provided in this application include at least the following:

[0050] (1) Achieve unified access across business systems and eliminate data silos;

[0051] (2) CDC second-level capture + Flink millisecond-level processing to achieve low-latency real-time link;

[0052] (3) Lakeware supports ACID transactions and stream-batch integration, achieving consistency and unified storage;

[0053] (4) Doris provides sub-second OLAP queries to meet the needs of real-time dashboards, analysis and reporting;

[0054] (5) Cloud-native elastic scaling + multi-active disaster recovery improves system stability and scalability;

[0055] (6) Improve maintainability and data governance capabilities through lineage, monitoring and alarm and permission system. Attached Figure Description

[0056] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0057] Figure 1 A schematic diagram of the overall process of a big data processing system for the new retail industry is provided as an embodiment of this application;

[0058] Figure 2 A flowchart illustrating the steps of a big data processing method for the new retail industry, provided as an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0061] With the development of new retail businesses, the integration of online and offline channels has brought about a large amount of data, including transaction data, product data, membership data, inventory data, and offline store behavior tracking. Existing data architectures generally suffer from the following problems:

[0062] (1) Data silos: Business system data is scattered across different databases or log systems, lacking a unified view;

[0063] (2) Insufficient real-time performance: Traditional ETL / offline data warehouses have high latency, making it difficult to support real-time recommendations and real-time risk control;

[0064] (3) Limited scalability: The growth in data volume makes it difficult to expand computing and storage capacity;

[0065] (4) Insufficient consistency and maintainability: lack of transaction guarantee, lineage tracking and unified monitoring and alarm system, making it difficult to locate data problems.

[0066] Therefore, an enterprise-level big data architecture is needed that can achieve "unified access from multiple sources + integrated batch processing + transaction consistency + sub-second query + cloud-native elasticity" in the new retail scenario.

[0067] The "new retail industry" referred to in this application refers to a modern retail format that is data-driven and deeply integrates online and offline channels, encompassing diverse business scenarios such as physical stores, e-commerce platforms, mobile applications, social media, and logistics. In this scenario, enterprises need to simultaneously process heterogeneous data from multiple sources, including offline store POS systems, online e-commerce transactions, membership management systems, inventory management systems, and user behavior tracking, to support key business operations such as real-time personalized recommendations, dynamic pricing, intelligent inventory scheduling, real-time risk control and anti-fraud, and precision marketing. The big data processing system in this application is specifically designed for this type of new retail scenario characterized by online-offline integration, complex business chains, and high real-time data requirements. By building unified data access, storage, computing, and analysis capabilities, it helps enterprises break down data silos and achieve real-time insights and intelligent decision-making across the entire data domain. Specifically, this system includes a unified data access module, a lake-warehouse integrated storage module, a real-time computing and analysis service module, a cloud-native resource and reliability module, and a governance and operation module, among which:

[0068] The unified data access module is used to capture and stream data from MySQL, MongoDB, and SDK tracking points using CDC, and then output the data to the lake warehouse integrated storage module after standardized processing.

[0069] This module specifically includes: unified access to MySQL relational databases, MongoDB document databases, and SDK event tracking data; support for full and incremental synchronization; CDC change capture for MySQL and MongoDB with capture latency controlled within seconds; end-to-end millisecond-level access and processing of event tracking data; and schema standardization, data cleaning, and time series unification after access.

[0070] The lake warehouse integrated storage module is used to receive data from the unified data access module, realize stream and batch storage based on ACID transactions, and provide a query interface for the real-time computing and analysis service module.

[0071] This module specifically includes: building an integrated lake warehouse storage system to achieve a balance between the flexibility of a data lake and the query performance of a data warehouse; supporting ACID transactions and integrated stream / batch write / query; unified management of structured / semi-structured / unstructured data; providing automatic tiering, lifecycle management, and cost optimization; and using columnar storage and intelligent indexes to improve query performance.

[0072] The real-time computing and analysis service module is used to obtain real-time streaming data from the unified data access module, perform computation and processing through Flink, and write it to Doris to achieve sub-second OLAP queries.

[0073] This module specifically includes: using Flink to complete real-time data stream processing, complex event processing, and real-time feature calculation, and supporting Exactly-Once semantics; writing real-time calculation results into the Doris MPP analysis engine; Doris provides capabilities such as vectorized execution and materialized views to achieve sub-second OLAP queries; and supporting real-time dashboards, interactive analysis, and real-time reports.

[0074] The cloud-native resources and reliability module provides Kubernetes containerized deployment, elastic scaling, and fault recovery services for the unified data access module, lake warehouse integrated storage module, and real-time computing and analysis service module.

[0075] This module specifically includes Kubernetes containerized deployment to achieve 99.9% availability; support for multi-active deployment and automatic failover; Flink checkpoint + state recovery to ensure fault tolerance; and storage layer multi-replica redundancy and automatic repair.

[0076] The governance and operation module is used to perform lineage tracing, monitoring and alarm management, and access control for the unified data access module, lake warehouse integrated storage module, real-time computing and analysis service module, and cloud-native resource and reliability module.

[0077] This module specifically includes: providing end-to-end data lineage tracing and metadata management; a unified monitoring and alerting system (Prometheus + Grafana); providing unified data service APIs and permission management; and supporting latency and quality indicator monitoring and rapid location.

[0078] Please refer to Figure 1This document illustrates the overall workflow of a big data processing system for the new retail industry, as provided in an embodiment of this application. The system adopts a layered architecture: the left-hand data source layer includes three heterogeneous data sources: MongoDB document database, MySQL relational database, and user tracking data. After unified access and preliminary processing by the Flink real-time computing engine, the data is written to the Paimon (ODS) lakehouse integrated storage layer. This layer achieves integrated stream and batch data ingestion through Flink+Kafka and provides unified metadata management through Catalog. After layered processing in the lakehouse layer, the data flows sequentially to the Doris data warehouse's DWD (detailed data layer), DWS (summary data layer), and ADS (application data layer), forming a complete data layering system. Finally, it provides sub-second OLAP query services to upper-layer real-time business needs through various interfaces such as BI, JDBC, multidimensional analysis, Spark, and Flink. This architecture achieves end-to-end connectivity from multi-source heterogeneous data access, lakehouse integrated storage, real-time computing processing to multidimensional analysis services, meeting the business requirements for data real-time performance, consistency, and high availability in the new retail scenario.

[0079] Furthermore, such as Figure 2 This application also provides a big data processing method for the new retail industry, including the following steps:

[0080] S1, Unified Access and Standardized Processing: Multi-source data is captured and streamed through a unified data access module, and then output after standardized processing;

[0081] S2, Lakehouse Integrated Storage and Transaction Consistency Guarantee: Data is received through the lakehouse integrated storage module and stream-batch integrated storage is achieved based on ACID transactions;

[0082] S3, Flink Real-Time Computing + Doris Analytics Service: The real-time computing and analytics service module performs Flink computing on real-time streaming data and writes it to Doris to achieve sub-second queries;

[0083] S4, Cloud-Native High Availability and Governance Operations: Provides containerized deployment and fault recovery services for each module through cloud-native resource and reliability modules, and performs lineage tracing, monitoring and alarms, and access control for the entire chain.

[0084] Specifically, step S1, unified access and standardization processing, includes:

[0085] Step S101: Configure the data source through the unified access platform, including MySQL, MongoDB and SDK tracking system;

[0086] Step S102: Enable CDC to capture change logs for MySQL and MongoDB to ensure that incremental capture latency is in the second range;

[0087] Step S103: Adopt a streaming access and real-time distribution mechanism for SDK data entry points to ensure end-to-end millisecond-level accuracy;

[0088] Step S104: Perform schema mapping, field standardization, timestamp alignment, and quality verification on the accessed data.

[0089] Specifically, step S2, lakeware integrated storage and transaction consistency guarantee, includes:

[0090] Step S201: Write all historical data into the lake warehouse integrated storage layer in batches, and simultaneously write CDC / tracking data in real time.

[0091] Step S202: The storage layer enables ACID transactions to ensure consistency between concurrent reads and writes;

[0092] Step S203: Automatically partition, bucket, and store the data in layers, and set lifecycle policies;

[0093] Step S204: Enable columnar storage, indexing, and metadata acceleration to improve query and backtracking performance.

[0094] Specifically, the S3Flink real-time computation + Doris analysis service step includes:

[0095] Step S301: Flink obtains real-time data streams from the access layer and performs cleaning, aggregation, feature calculation, and complex event processing;

[0096] Step S302: Flink enables Exactly-Once semantics, using checkpointing to ensure result consistency and recoverability with the state backend;

[0097] Step S303: Write real-time metrics, feature wide tables, etc. into Doris;

[0098] Step S304: Doris utilizes vectorized execution and materialized views to achieve sub-second OLAP queries, supporting real-time dashboards, reports, and interactive analysis.

[0099] Specifically, step S4, cloud-native high availability and governance operations, includes:

[0100] Step S401: The compute and storage components are deployed based on Kubernetes to achieve elastic scaling;

[0101] Step S402: Multi-active deployment and automatic failover improve availability;

[0102] Step S403: The monitoring and alarm system collects and processes latency, throughput, error rate, and quality indicators;

[0103] Step S404: Support data governance and compliant use through lineage tracing and access management.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A big data processing system for the new retail industry, characterized in that, The system includes a unified data access module, a lake warehouse integrated storage module, a real-time computing and analysis service module, a cloud-native resource and reliability module, and a governance and operation and maintenance module, wherein: The unified data access module is used to capture and stream data from MySQL, MongoDB and SDK tracking points using CDC, and output the data to the lake warehouse integrated storage module after standardization processing. The lake warehouse integrated storage module is used to receive data from the unified data access module, realize stream and batch integrated storage based on ACID transactions, and provide a query interface for the real-time computing and analysis service module; The real-time computing and analysis service module is used to obtain real-time streaming data from the unified data access module, perform computing and processing through Flink, and write it to Doris to achieve sub-second OLAP queries. The cloud-native resource and reliability module is used to provide Kubernetes containerized deployment, elastic scaling and fault recovery services for the unified data access module, lake warehouse integrated storage module and real-time computing and analysis service module. The governance and operation module is used to perform lineage tracing, monitoring and alarming, and access control for the unified data access module, lake warehouse integrated storage module, real-time computing and analysis service module, and cloud-native resource and reliability module.

2. The system according to claim 1, characterized in that, The unified data access module specifically includes: The data source configuration unit is used to manage the connection parameters and access strategies of MySQL, MongoDB, and SDK tracking systems. The CDC capture unit is used to capture change data for MySQL and MongoDB and output incremental data to the lake warehouse integrated storage module in real time. The streaming access unit is used to establish a millisecond-level streaming channel for SDK embedded data; The standardization processing unit is used to perform schema mapping, field standardization, and quality verification.

3. The system according to claim 1, characterized in that, The integrated lake warehouse storage module specifically includes: A stream-batch integrated storage engine is used to support batch writing of historical data and real-time writing of CDC / tracking stream data; The transaction management unit is used to enable ACID transactions to ensure concurrent read-write consistency. The intelligent optimization unit is used to perform automatic hierarchical storage and columnar storage optimization; The lifecycle management unit is used to perform data archiving and expiration cleanup according to preset strategies.

4. The system according to claim 1, characterized in that, The real-time computing and analysis service module specifically includes: Flink's real-time computing engine is used to perform data cleaning, aggregation, feature calculation, and complex event processing, and achieves exactly-Once semantic guarantees through the Checkpoint mechanism. The Doris analytics engine receives Flink computation results and uses vectorized execution and materialized views to achieve sub-second OLAP queries. The application interface layer provides standardized query interfaces for real-time data dashboards and reporting systems.

5. The system according to claim 1, characterized in that, The cloud-native resource and reliability module specifically includes: Kubernetes orchestration unit, used to deploy compute and storage components and enable elastic scaling; High availability assurance unit, used to perform multi-active deployment and automatic failover; The fault-tolerant recovery unit is used to ensure data reliability through Flink state recovery and storage of multiple replicas for redundancy.

6. The system according to claim 1, characterized in that, The governance and operation module specifically includes: The lineage tracking unit is used to record information on the entire data flow process. The metadata management unit is used for unified management of data schema and asset catalog; The monitoring and alarm unit is used to collect and process latency, throughput, and error rate metrics and trigger alarms. The Data Services and Permissions Unit is used to provide a unified API gateway and fine-grained permission control.

7. The system according to claim 1, characterized in that, The unified data access module is connected to the lake warehouse integrated storage module through a message queue. The real-time computing and analysis service module obtains real-time streaming data from the unified data access module through a data subscription method. The cloud-native resource and reliability module establishes resource scheduling relationships with each module through a container orchestration interface.

8. The system according to claim 4, characterized in that, The Flink real-time computing engine supports event-time-based window computing, and the Doris analytics engine supports mixed workloads of real-time import and batch import, and optimizes query execution plans through dynamic partitioning and pruning.

9. A big data processing method for the new retail industry, applied to the system described in any one of claims 1-8, characterized in that, The method includes: CDC change capture is performed on MySQL and MongoDB, and streaming access is performed on SDK data points. After standardization processing, a standardized data stream is formed and output to the lake warehouse integrated storage module. The system receives the standardized data stream, implements integrated stream and batch storage based on ACID transactions, and provides a query interface for the real-time computing and analysis service module. Real-time streaming data is acquired, processed by Flink, and then written to Doris to achieve sub-second OLAP queries. Provides Kubernetes containerized deployment, elastic scaling, and fault recovery services; Perform lineage tracing, monitoring and alerting, and access control across the entire supply chain.

10. The method according to claim 9, characterized in that, The unified data access module performs CDC capture and streaming access on multi-source data, specifically including: Enable CDC change data capture for MySQL and MongoDB, parse the database transaction logs to obtain incremental data, and ensure that the capture latency is controlled within the second level; A high-throughput streaming channel is established for SDK data tracking, and real-time distribution is achieved through a message queue to ensure that the end-to-end transmission latency reaches the millisecond level. The system performs automatic schema mapping, field type standardization, business timestamp alignment, and integrity verification on the heterogeneous data received, forming a standardized data stream that is output to the lake warehouse integrated storage module.

Images