Shopping mall data analysis system

Abstract

The invention discloses a shopping mall data analysis system, which relates to the technical field of electronic commerce data processing, and comprises a data access and preprocessing module, a streaming batch integrated calculation engine, a dynamic mode recognition and prediction module, an intelligent decision support module and a visual interaction platform, high-throughput and low-delay data processing and dynamic mode recognition are realized through stream batch integrated calculation and an incremental learning mechanism, and the real-time performance, the adaptive ability and the decision-making efficiency of shopping mall data analysis are improved in combination with a multi-objective optimization and asynchronous rendering technology. The invention aims to solve the problems of insufficient real-time performance of data processing, high calculation delay and poor adaptability to dynamic user behaviors.

Description

Technical Field

[0001] This invention relates to the field of e-commerce data processing technology, and in particular to an online marketplace data analysis system. Background Technology

[0002] In the field of e-commerce technology, e-commerce data analysis systems are becoming increasingly important as core tools supporting business decisions and optimizing operational efficiency. These systems typically involve the collection, storage, processing, and analysis of massive amounts of user behavior data, transaction data, and product data to uncover potential business value and market trends.

[0003] The key task of the e-commerce data analysis system is to transform raw data into insightful information that can be used for decision-making through efficient data processing. Its basic goal is to assist managers in conducting precise marketing, inventory optimization, and service improvement by monitoring and analyzing multi-dimensional indicators such as sales trends, user preferences, and inventory status.

[0004] In existing technologies, e-commerce data analysis systems generally employ traditional data warehouses and batch processing frameworks, which are insufficient to meet the demands of real-time or near-real-time data analysis. When handling high concurrency and multi-source heterogeneous data, these systems often face problems such as low data integration efficiency and high computational latency, resulting in analysis results lagging behind business changes. Furthermore, existing analytical models are insufficiently adaptable to dynamically changing user behavior patterns, making it difficult to quickly identify sudden sales trends or abnormal transaction behaviors, and easily leading to response bottlenecks during promotional activities or peak traffic periods. Simultaneously, the performance of data visualization and interactive query functions is limited, impacting the decision-making efficiency of managers and the user experience. Summary of the Invention

[0005] The purpose of this invention is to provide an e-commerce data analysis system to solve the problems of insufficient real-time data processing, high computational latency, poor adaptability to dynamic user behavior patterns, and significant response bottlenecks during peak traffic periods in existing technologies.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] An e-commerce data analysis system includes: a data access and preprocessing module, a stream-batch integrated computing engine, a dynamic pattern recognition and prediction module, an intelligent decision support module, and a visual interactive platform.

[0008] The data access and preprocessing module is responsible for receiving multi-source heterogeneous data inputs, performing data cleaning, format standardization, and preliminary feature extraction operations. It is internally equipped with a data quality verification unit and a real-time data distribution channel to ensure that the raw data is standardized and transmitted to downstream computing nodes within millisecond latency.

[0009] The integrated stream and batch computing engine adopts a unified architecture that supports both real-time streaming data processing and offline batch computing tasks. The engine includes a stream processing sub-engine and a batch processing sub-engine, both of which share the same set of memory resource management and task scheduling strategies. They also maintain the consistency of the computing context through a state synchronization mechanism, thereby achieving high throughput and low latency data processing capabilities.

[0010] The dynamic pattern recognition and prediction module, built on an incremental learning algorithm, continuously receives output data from the integrated batch processing engine, automatically detects sudden changes in user behavior patterns and abnormal sales trends, and generates short-term predictive indicators. This module integrates a pattern drift detection unit and an adaptive model update unit, which can dynamically adjust the parameters and structure of the recognition model without interrupting the service.

[0011] The intelligent decision support module receives the analysis results from the dynamic pattern recognition and prediction module, and combines them with preset business rules and optimization objectives to generate inventory allocation suggestions, marketing strategy adjustment plans, and risk warning signals. This module has a built-in multi-objective optimization solver, which is used to solve the optimal decision combination under resource constraints.

[0012] The visualization and interactive platform provides data display and query functions based on a web graphical interface. It supports multi-dimensional data drill-down, real-time dashboard refresh, and interactive report generation. Its front-end rendering engine adopts asynchronous loading and data sharding technology to ensure a smooth user experience in high-concurrency access scenarios.

[0013] Furthermore, the data access and preprocessing module receives input data streams from user terminals, transaction servers, logistics databases, and third-party data interfaces via a distributed message middleware. To ensure data quality, the data cleaning unit uses a rule engine and outlier detection algorithm to automatically identify and filter missing fields, duplicate records, and data with format errors. Subsequently, the format standardization unit converts heterogeneous data into a unified JSON-LD structured format and adds a timestamp and data source identifier to each data record. After data cleaning, the preliminary feature extraction unit extracts key attributes from the raw data based on preset feature templates, including user session duration, product click sequence, transaction amount distribution, and geographic location information. To continuously monitor data quality, the data quality verification unit dynamically evaluates the input data quality by calculating data integrity rate, consistency index, and timeliness score, and triggers data retransmission or alarm processes when anomalies are detected. Finally, the real-time data distribution channel adopts a publish-subscribe model to push the preprocessed data in parallel to the input buffer of the stream-batch integrated computing engine.

[0014] Furthermore, the integrated stream and batch computing engine includes: a stream processing sub-engine that processes real-time data streams based on event-time semantics, internally implementing a sliding window aggregation operator and a state snapshot mechanism, capable of completing user behavior funnel analysis and real-time transaction statistics within any time window; simultaneously, the batch processing sub-engine periodically executes full-data computation tasks, including historical sales trend modeling, user profile updates, and inventory turnover rate analysis; to balance the resource requirements of stream and batch processing, the memory resource management unit adopts a dynamic partitioning allocation strategy, automatically adjusting the allocation ratio and reclamation threshold of memory blocks according to real-time load conditions; the task scheduling strategy is based on priority queues and resource prediction models, allocating more computing resources to high real-time tasks while ensuring that the progress of batch tasks is not affected; in addition, the state synchronization mechanism records the intermediate state of the stream processing sub-engine through a distributed transaction log and restores a consistent view when the batch processing sub-engine starts, thereby ensuring the eventual consistency of stream and batch computing results.

[0015] Furthermore, the dynamic pattern recognition and prediction module operates as follows: the incremental learning algorithm employs an online gradient descent optimization method, updating model weights upon receiving a new batch of data, thus avoiding the overhead of retraining the entire historical data set; to cope with changes in the business environment, the pattern drift detection unit automatically identifies pattern shifts in user behavior or sales trends by monitoring changes in the statistical distribution of the prediction error sequence; once a significant drift is detected, this unit immediately sends a model reconstruction instruction to the adaptive model update unit; the adaptive model update unit flexibly selects local parameter fine-tuning or global structure reconstruction strategies based on the magnitude and direction of the pattern drift: for slight drifts, only the weight parameters of the model output layer are adjusted; for severe drifts, feature selection and reorganization, as well as the addition and deletion of hidden layer nodes, are triggered; based on the updated model, the system generates short-term prediction indicators, including sales forecasts for the next 24 hours, the probability of changes in the ranking of popular products, and an abnormal transaction risk index. These indicators are refreshed every 5 minutes and pushed to the intelligent decision support module.

[0016] Furthermore, the decision-making process of the intelligent decision support module is designed as follows: A multi-objective optimization solver constructs a constrained mathematical programming model with parallel objectives of maximizing sales, minimizing inventory backlog, and optimizing customer satisfaction; the solver adopts a hybrid solution strategy of genetic algorithm and linear programming, outputting a Pareto optimal solution set within milliseconds; based on the optimization results, the system generates three types of decision suggestions: inventory allocation suggestions calculate the transfer quantity and replenishment timing between warehouses based on real-time sales forecasts and current inventory levels; marketing strategy adjustment schemes dynamically generate personalized coupon distribution strategies and advertising content update instructions based on changes in user behavior patterns; risk warning signals trigger manual review processes or automatically freeze suspicious account operations for detected abnormal transaction patterns; to meet the needs of different business scenarios, business rules and optimization objectives can be dynamically set by administrators through a visual configuration interface and persistently stored in the rule database.

[0017] Furthermore, the front-end implementation of the visualization interaction platform includes: a web graphical interface developed based on a component-based architecture, supporting drag-and-drop layout customization and theme style switching, providing users with a flexible and personalized experience; a data drill-down function allows users to drill down from the summary view to detailed data layer by layer, with the drill-down path automatically constructed based on the dimensional hierarchy, facilitating multi-angle data analysis; a real-time dashboard maintains synchronization with the backend data service via a WebSocket long connection, ensuring that data updates are reflected on the front-end interface within 2 seconds; an interactive report generator provides various chart templates and filtering conditions, allowing users to quickly generate customized reports by selecting dimensions and indicators; to optimize the user experience in big data scenarios, asynchronous loading technology divides large datasets into multiple data blocks and loads them to the browser on demand, avoiding interface lag; data sharding technology performs pagination processing on ultra-large-scale query results, with each page displaying 100 records by default, allowing users to browse all data through page turning operations.

[0018] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0019] This invention achieves a qualitative leap in data processing capabilities by constructing a stream-batch integrated computing engine and a dynamic pattern recognition module. The system can simultaneously meet the needs of real-time analysis and batch computing, and maintain low latency and high throughput performance in high-concurrency scenarios. The dynamic pattern recognition module, based on incremental learning and adaptive update mechanisms, significantly improves the detection sensitivity and prediction accuracy of sudden changes in user behavior and sales trend drift, enabling the system to respond quickly to market changes.

[0020] The intelligent decision support module of this invention generates comprehensive decision solutions through a multi-objective optimization solver, effectively coordinating multiple business objectives such as sales, inventory, and customer service, and improving the scientific nature and global optimality of business decisions; the visual interaction platform adopts asynchronous loading and data sharding technology, which completely solves the front-end performance bottleneck when rendering large-scale data, providing managers with a smooth and efficient data exploration and decision support experience. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall technical architecture of the e-commerce data analysis system proposed in this invention;

[0022] Figure 2 This is a schematic diagram of the core principle framework of the integrated batch processing computing engine in this invention. Detailed Implementation

[0023] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0024] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0025] In the embodiments of the present invention, the same reference numerals denote the same components, and for the sake of brevity, detailed descriptions of the same components are omitted in different embodiments. It should be understood that the thickness, length, width, and other dimensions of various components in the embodiments of the present invention shown in the accompanying drawings, as well as the overall thickness, length, width, and other dimensions of the integrated device, are merely illustrative and should not constitute any limitation on the present invention; the term "multiple" in the present invention refers to two or more (including two).

[0026] Example 1

[0027] Please refer to the attached document. Figure 1 This embodiment details the technical implementation scheme of an e-commerce data analysis system. The system aims to solve technical challenges in existing technologies, such as insufficient real-time data processing, high computational latency, poor adaptability to dynamic user behavior patterns, and significant response bottlenecks during peak traffic periods. The core architecture of the system consists of five modules: a data access and preprocessing module, a stream-batch integrated computing engine, a dynamic pattern recognition and prediction module, an intelligent decision support module, and a visual interaction platform. These modules collaborate efficiently through strictly defined data interfaces and communication protocols.

[0028] The data access and preprocessing module serves as the system's data entry point, undertaking the crucial tasks of receiving multi-source heterogeneous data inputs, performing data cleaning, format standardization, and preliminary feature extraction. This module receives continuous data streams from user terminals, transaction servers, logistics databases, and third-party data interfaces through a distributed message middleware.

[0029] The data cleaning unit is the primary processing step in this module. It uses a combination of logic based on a rule engine and a statistical outlier detection algorithm to automatically identify and filter missing fields, duplicate records, and data with incorrect formatting in the input data.

[0030] The rules engine has over fifty built-in data validation rules, covering field non-empty validation, numerical range validation, enumeration value validity validation, and cross-field logical consistency validation. The outlier detection algorithm uses the three Sigma principle combined with the isolated forest model to dynamically calculate the reasonable distribution range of each numeric field; any value deviating from this range by more than three standard deviations will be marked as an outlier and removed or replaced with a preset default value.

[0031] The format standardization unit runs immediately after the data cleaning unit. Its core function is to convert the cleaned heterogeneous data into a unified JSON-LD structured format. This conversion process includes field name mapping, data type casting, and nested structure flattening. After the conversion, each data record is automatically appended with a timestamp accurate to the millisecond and a globally unique data source identifier. The timestamp is used for subsequent event-time-based stream processing calculations, while the data source identifier is used for data lineage tracing and problem attribution.

[0032] The preliminary feature extraction unit extracts key business attributes from standardized data records based on predefined feature templates. A feature template is a configurable set of metadata that defines the fields to be extracted and their derived calculation rules.

[0033] Typical extracted features include: user session duration, product click sequence, transaction amount distribution, and geographic location information. Among them, user session duration is obtained by calculating the time difference between consecutive operations of the same user, product click sequence is the order in which a user browses products within a session, transaction amount distribution is the mean, variance, and quantile of transaction amount within a specified time window, and geographic location information is the city, region, and other hierarchical information parsed from the latitude and longitude data reported by the user terminal.

[0034] The data quality verification unit runs throughout the entire preprocessing process, dynamically evaluating the quality level of input data by calculating three core indicators in real time: data integrity rate, consistency index, and timeliness score. Among them, the data integrity rate is the proportion of non-empty fields to the total number of fields, the consistency index measures the degree of consistency of information corresponding to the same entity in different data sources, and the timeliness score evaluates the time delay from data generation to entering the preprocessing module. When any indicator falls below the preset threshold, the unit will immediately trigger the data retransmission mechanism or send an alarm message to the system monitoring center.

[0035] The real-time data distribution channel, serving as the module's output, employs a high-throughput publish-subscribe model to push data records that have completed all preprocessing steps in parallel to the input buffer of the stream-batch computing engine. This channel achieves a data throughput capacity of at least 100,000 records per second and ensures end-to-end processing latency is controlled at the millisecond level.

[0036] Please refer to the attached document. Figure 2 The integrated stream and batch computing engine is the core of the entire system, employing a unified architecture that supports both real-time streaming data processing and offline batch computing tasks. This engine consists of five core sub-engines: a stream processing sub-engine, a batch processing sub-engine, a memory resource management unit, a task scheduling strategy module, and a state synchronization mechanism.

[0037] The stream processing sub-engine is specifically designed for handling real-time data streams. Based on event-time semantics rather than processing-time semantics, it can correctly handle out-of-order data events. The engine implements various window aggregation operators, including scrolling windows, sliding windows, and session windows. Among them, the sliding window aggregation operator is particularly suitable for user behavior funnel analysis, and can dynamically calculate the conversion rate from browsing products to completing payment within any time window. To prevent state loss due to node failure, the state snapshot mechanism periodically persists the intermediate state during the stream processing to the distributed file system.

[0038] The batch processing sub-engine is responsible for periodically executing full-data calculation tasks. Typical tasks include historical sales trend modeling, user profile updates, and inventory turnover analysis. Historical sales trend modeling is performed on a daily or weekly basis, using regression analysis of all transaction data from the past year to identify seasonal patterns and long-term trends. User profile updates integrate users' historical behavioral and attribute data to generate comprehensive profiles that include dimensions such as interests, purchasing power, and loyalty. Inventory turnover analysis calculates the average number of days each product is held in inventory and the number of times it turns over in the past thirty days.

[0039] The memory resource management unit employs a dynamic partitioning allocation strategy, continuously monitoring the real-time load of streaming and batch processing tasks. The system presets both soft and hard limits on memory usage. When the load on streaming tasks surges, the management unit temporarily reclaims some memory blocks from the batch processing task pool and allocates them to the streaming sub-engine, and vice versa. The reclamation threshold is dynamically adjusted based on task priority, with higher-priority real-time tasks enjoying greater memory protection.

[0040] The task scheduling strategy module operates based on a multi-level priority queue and a resource prediction model. All submitted computational tasks first enter the scheduling queue. The resource prediction model predicts the required CPU time, memory space, and input / output bandwidth based on the task's historical execution data and input data volume. Subsequently, the scheduler combines task priority and resource prediction results to allocate more computing resources to high real-time tasks, while ensuring that the progress of batch tasks is not severely affected through a resource reservation mechanism.

[0041] The state synchronization mechanism is a key technology for ensuring the consistency of results in a unified stream and batch computing engine. This mechanism records all critical intermediate state change events of the stream processing sub-engine through a distributed transaction log. When the batch processing sub-engine starts, it reads the latest state snapshot from the transaction log and initializes the computing context based on this snapshot, thereby ensuring that even for the same data source, the computing results generated by stream processing and batch processing remain consistent in their final state.

[0042] The dynamic pattern recognition and prediction module is responsible for automatically detecting sudden changes in user behavior patterns and abnormal sales trends from massive amounts of data, and generating high-precision short-term predictive indicators.

[0043] The core of this module is a recognition and prediction model built based on an incremental learning algorithm. The incremental learning algorithm employs an online gradient descent optimization method, and its core principle can be expressed in the following mathematical form: the model weight vector is iteratively updated in batches based on newly arriving data, with the update formula being:

[0044]

[0045] in, This shows the model weights at the t-th iteration. It's the learning rate parameter, which controls the step size for each update. Let f represent the gradient of the loss function with respect to the weights, and f be the model's prediction function. It is the input feature vector. These are the true label values. This update mechanism allows the model to update its weights every time it receives a new batch of data, without having to retrain on the entire historical dataset, greatly improving the speed and efficiency with which the model adapts to data changes.

[0046] The mode drift detection unit continuously monitors changes in the statistical distribution of the prediction error sequence. This unit calculates the absolute error between the predicted and actual values ​​and maintains a sliding window of error with a length of one thousand. Every five minutes, the unit calculates the mean and standard deviation of the error within the window and compares it with historical benchmark values. If the current mean error exceeds twice the standard deviation of the historical benchmark value, and this condition persists for more than three detection cycles, a significant mode drift is determined to have occurred. At this point, the detection unit immediately sends a model reconstruction instruction to the adaptive model update unit, containing detailed information such as the magnitude, direction, and affected feature dimensions of the drift.

[0047] The adaptive model update unit intelligently selects a model adjustment strategy based on the received drift information. Different processing schemes are adopted for different degrees of mode drift:

[0048] For minor drift (error changes within 10%), the unit only adjusts the weight parameters of the model's output layer, preventing overfitting by adding a regularization term. For severe drift (error changes exceeding 30%), the unit triggers a comprehensive model reconstruction process, including feature selection and reorganization, and adding / removing hidden layer nodes. Feature selection and reorganization uses a forward search algorithm based on mutual information to select the subset of features most relevant to the current data distribution from the original feature set. Hidden layer node adjustment dynamically increases or decreases the number of nodes in the neural network's hidden layers based on the model's performance on the validation set, optimizing the match between model capacity and data complexity.

[0049] The short-term forecasting indicators are the final output of this module, including three main categories of core indicators:

[0050] First, the sales forecast for the next 24 hours, predicting the sales of the entire mall and each product category on an hourly basis.

[0051] Second, the probability of changes in the ranking of popular products is calculated by determining the probability that each product will enter the top ten in sales within the next 6 hours.

[0052] Third, the abnormal transaction risk index assesses the risk level of each transaction as fraudulent by comprehensively considering multiple characteristics such as transaction amount, user behavior sequence, and device fingerprint.

[0053] All these metrics are refreshed every 5 minutes and pushed to the intelligent decision support module via a high-speed data bus.

[0054] The intelligent decision support module receives analysis results from the dynamic pattern recognition and prediction module, and generates executable business decision solutions by combining preset business rules and optimization objectives.

[0055] The core component of this module is a multi-objective optimization solver, which constructs a constrained mathematical programming model with three parallel optimization objectives: maximizing sales, minimizing inventory backlog, and optimizing customer satisfaction.

[0056] The model contains more than 100 decision variables and 50 constraints: decision variables include replenishment quantity for each product, promotional discount rate, and advertising budget allocation; constraints cover business restrictions such as warehouse capacity limit, budget limit, and minimum customer satisfaction requirement.

[0057] The solver employs a hybrid solution strategy combining genetic algorithms and linear programming. The genetic algorithm searches for high-quality solution regions within the global search space, with a population size of 200 individuals, 500 generations, a crossover probability of 0.8, and a mutation probability of 0.1. Linear programming is used to fine-tune the high-quality solutions found by the genetic algorithm locally, aiming to obtain the exact solution point on the Pareto front. The entire solution process is completed within 500 milliseconds, outputting a Pareto optimal solution set containing 10 to 15 non-dominated solutions.

[0058] Inventory allocation recommendations are generated based on real-time sales forecasts and current inventory levels. The system scans the inventory status of all products every minute. When the projected sales of a product in the next 24 hours exceed 90% of the current inventory, the system automatically generates an allocation recommendation. The allocation recommendation details the warehouse to be transferred out, the warehouse to be transferred in, the types of products to be transferred, the quantity to be transferred, and the latest execution time. The replenishment timing is calculated using an economic order quantity model, comprehensively considering ordering costs, holding costs, and stockout risks to determine the optimal reorder point and order quantity.

[0059] Marketing strategy adjustment plans are dynamically generated based on changes in user behavior patterns. The system analyzes user clickstream data in real time, and automatically generates personalized coupon distribution strategies when it detects a significant increase in attention to a certain type of product. The coupon value, usage conditions, and validity period are precisely calculated based on users' historical purchasing power and product profit margins. Simultaneously, the system sends content update instructions to the ad placement management system, pushing high-attention products to the homepage's featured images and recommended positions.

[0060] Risk warning signals are triggered upon detection of abnormal transaction patterns. Each transaction is evaluated by the risk scanning engine within 0.5 seconds of completion. If the abnormal transaction risk index exceeds 0.85, the system will immediately trigger a manual review process, suspending the transaction and notifying a risk control specialist for manual review. In extreme cases where the risk index exceeds 0.95, the system will automatically perform a temporary account freeze and simultaneously send SMS and email notifications to the account's linked user.

[0061] The visual interactive platform provides system users with data display and query functions based on a web graphical interface.

[0062] The platform's web graphical interface is developed based on a component-based architecture, with all charts, tables, and controls encapsulated as independently configurable and reusable front-end components. Users can freely combine these components through drag-and-drop operations to customize personalized monitoring dashboards and data analysis views. The platform supports three preset theme styles and fully customizable style configurations to meet the aesthetic preferences of different users.

[0063] The data drill-down feature allows users to drill down from a highly summarized view to the finest granular details. Drill-down paths are automatically constructed based on a robust hierarchical structure, such as drilling down from annual sales to quarterly, monthly, daily, and even individual transaction records. The system includes 10 pre-defined drill-down paths, covering time, geographic, product category, and user segmentation dimensions.

[0064] The real-time dashboard maintains bidirectional communication with the backend data service via a WebSocket persistent connection. When the backend data service detects any relevant data updates, it immediately pushes the incremental data to the frontend via the WebSocket channel. Upon receiving the data, the frontend smoothly refreshes the charts within 2 seconds, ensuring users always see the latest data status.

[0065] The interactive report generator offers over 20 chart templates and rich filtering options. Users simply select the desired dimensions and metrics, and the system can generate a complete, customized report within 5 seconds. Reports support advanced features such as online preview, export to multiple formats, and scheduled email notifications.

[0066] To ensure front-end performance, the system employs two key technologies:

[0067] Asynchronous loading technology automatically divides large datasets into multiple data blocks, each with a fixed size of 100 records. When a user scrolls through a table or chart, the system only loads the data blocks corresponding to the currently visible area and the pre-read area, greatly reducing the browser's memory usage and rendering burden.

[0068] Data sharding technology is optimized for ultra-large-scale query results. By default, the system divides the query results into 100 records per page, and users can browse all the data by turning the page. For queries with more than 10,000 records, the system will enable the background data sharding preloading mechanism, which loads the data of the next three pages into the local cache in advance while the user is browsing the current page, so as to achieve a seamless page turning experience.

[0069] The entire system is deployed on a distributed cluster consisting of more than 50 physical servers, and external requests are distributed to various computing nodes through a load balancer. The system adopts a microservice architecture, with each core module running as an independent microservice. Services communicate with each other through remote procedure calls and message queues. All microservices have horizontal scaling capabilities and the number of instances can be dynamically adjusted according to business load.

[0070] The system monitoring framework covers all aspects from hardware resources to application performance, including key metrics such as CPU utilization, memory usage, disk I / O, network bandwidth, service response time, and error rate. When any metric exceeds a preset threshold, the monitoring system will immediately trigger an alarm and notify the operations and maintenance team.

[0071] In terms of data security, the system implements end-to-end encrypted data transmission and storage, and all sensitive data is encrypted before being stored. The access control system is based on a role-based access control model to ensure that different users can only access data within their authorized scope; the system performs a full data backup every 24 hours, retains the backup data for 30 days, and supports data recovery operations at any point in time.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A shopping mall data analysis system, characterized in that, include: The data access and preprocessing module is used to connect to multi-source heterogeneous data input, perform data cleaning, format standardization and preliminary feature extraction operations, and internally deploys a data quality verification unit and a real-time data distribution channel. The integrated stream and batch computing engine adopts a unified architecture that supports both real-time streaming data processing and offline batch computing tasks. The engine includes a stream processing sub-engine and a batch processing sub-engine, both of which share the same set of memory resource management and task scheduling strategies, and maintain the consistency of the computing context through a state synchronization mechanism. The dynamic pattern recognition and prediction module, built on an incremental learning algorithm, continuously receives output data from the integrated streaming and batch computing engine, automatically detects sudden changes in user behavior patterns and abnormal sales trends, and generates short-term predictive indicators. This module integrates a pattern drift detection unit and an adaptive model update unit, which can dynamically adjust the parameters and structure of the recognition model without interrupting the service. The intelligent decision support module receives the analysis results from the dynamic pattern recognition and prediction module, and combines them with preset business rules and optimization goals to generate inventory allocation suggestions, marketing strategy adjustment plans, and risk warning signals. This module has a built-in multi-objective optimization solver for solving optimal decision combinations under resource constraints; The visualization and interactive platform provides data display and query functions based on a web graphical interface, supports multi-dimensional data drill-down, real-time dashboard refresh and interactive report generation, and its front-end rendering engine adopts asynchronous loading and data sharding technology.

2. The e-commerce data analysis system according to claim 1, characterized in that, The data access and preprocessing module receives input data streams from user terminals, transaction servers, logistics databases, and third-party data interfaces through a distributed message middleware. The data cleaning unit uses a rule engine and outlier detection algorithm to automatically identify and filter missing fields, duplicate records, and data with format errors; the format standardization unit converts heterogeneous data into a unified JSON-LD structured format and adds a timestamp and data source identifier to each data record; the preliminary feature extraction unit extracts key attributes from the raw data based on preset feature templates, including user session duration, product click sequence, transaction amount distribution, and geographic location information.

3. The e-commerce data analysis system according to claim 2, characterized in that, The data quality verification unit dynamically evaluates the quality of input data by calculating data integrity rate, consistency index and timeliness score, and triggers data retransmission or alarm process; the real-time data distribution channel adopts the publish-subscribe mode to push the preprocessed data to the input buffer of the stream batch computing engine in parallel.

4. The e-commerce data analysis system according to claim 1, characterized in that, The stream processing sub-engine processes real-time data streams based on event-time semantics. Internally, it implements a sliding window aggregation operator and a state snapshot mechanism, enabling it to complete user behavior funnel analysis and real-time transaction statistics within any time window. The batch processing sub-engine periodically executes full data calculation tasks, including historical sales trend modeling, user profile updates, and inventory turnover rate analysis.

5. The e-commerce data analysis system according to claim 4, characterized in that, The memory resource management unit adopts a dynamic partitioning allocation strategy, which automatically adjusts the allocation ratio and reclamation threshold of memory blocks according to the real-time load of stream processing and batch processing tasks. The task scheduling strategy is based on priority queues and resource prediction models, which allocate more computing resources to high real-time tasks while ensuring that the progress of batch tasks is not affected.

6. The e-commerce data analysis system according to claim 5, characterized in that, The state synchronization mechanism records the intermediate state of the stream processing sub-engine through a distributed transaction log and restores a consistent view when the batch processing sub-engine starts.

7. The e-commerce data analysis system according to claim 1, characterized in that, The incremental learning algorithm employs an online gradient descent optimization method, updating the model weights with each new batch of data received, without needing to retrain on all historical data. The pattern drift detection unit automatically identifies pattern shifts in user behavior or sales trends by monitoring changes in the statistical distribution of the prediction error sequence. When a significant drift is detected, this unit sends a model reconstruction instruction to the adaptive model update unit.

8. The e-commerce data analysis system according to claim 7, characterized in that, The adaptive model update unit selects a local parameter fine-tuning or global structure reconstruction strategy based on the magnitude and direction of the mode drift. For slight drift, only the weight parameters of the model output layer are adjusted. For severe drift, feature selection and recombination and hidden layer node addition and deletion operations are triggered. The short-term prediction indicators include the sales forecast for the next 24 hours, the probability of changes in the ranking of popular products, and the abnormal transaction risk index. The short-term prediction indicators are refreshed every 5 minutes and pushed to the intelligent decision support module.

9. The e-commerce data analysis system according to claim 1, characterized in that, The multi-objective optimization solver constructs a constrained mathematical programming model with the parallel objectives of maximizing sales, minimizing inventory backlog, and optimizing customer satisfaction. The solver employs a hybrid solution strategy combining genetic algorithms and linear programming, outputting a Pareto optimal solution set within milliseconds. The inventory allocation suggestion is based on real-time sales forecasts and current inventory levels, calculating the transfer quantity and replenishment timing between warehouses. The marketing strategy adjustment plan dynamically generates personalized coupon distribution strategies and ad content update instructions based on changes in user behavior patterns. The risk warning signal triggers a manual review process or automatically freezes suspicious accounts for detected abnormal transaction patterns.

10. The e-commerce data analysis system according to claim 1, characterized in that, The visualization and interactive platform's web graphical interface is developed based on a component-based architecture, supporting drag-and-drop layout customization and theme style switching; the data drill-down function allows users to drill down from the summary view to detailed data layer by layer, with the drill-down path automatically constructed based on the dimensional hierarchy; the real-time dashboard maintains synchronization with the backend data service via a WebSocket long connection; the interactive report generator provides a variety of chart templates and filtering conditions, allowing users to quickly generate customized reports by selecting dimensions and indicators.

Images