Big-data-based real-time analysis system

Abstract

The present invention relates to the technical field of big data. Disclosed is a big-data-based real-time analysis system. The system comprises: a data collection module, which is deployed on a server of a shopping website and is responsible for collecting in real-time log data of the shopping website, wherein the log data comprises a user behavior log and a user transaction log; a data transmission module, which is configured to transmit the collected log data to a data preprocessing module by means of a distributed real-time data transmission channel so as to perform processing; the data preprocessing module, which is configured to preprocess the received log data, and perform cleaning and format conversion on the log data for subsequent data analysis; a data analysis module, which is configured to perform real-time analysis on the preprocessed log data by using a big data analysis method, so as to obtain a transaction trend of a user; and a data display module, which is configured to display an analysis result to relevant personnel in a visualized manner. The system can accurately obtain a transaction trend of users, thereby enabling accurate commodity recommendations.

Description

Real-time analysis system based on big data TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a real-time analysis system for shopping website log based on big data. BACKGROUND

[0002] Electronic marketing is a new marketing model that uses the Internet as a media platform, uses new ways, methods and business philosophy, and through a series of attractive network marketing plans, formulates and implements marketing activities of enterprises, and more effectively promotes the realization of personal and organizational transaction activities. It is an important part of the overall marketing strategy of an enterprise, and is a variety of activities that create an online business environment using the Internet as a basic means to achieve the overall or partial business objectives of an enterprise. The existing e-commerce marketing system only promotes similar or similar products to customers in a cramming manner when promoting products to customers, and cannot deeply understand where the product points of customer attention are, so as to more deeply grasp the purchasing desire of customers.

[0003] SUMMARY

[0004] The technical problem to be solved by the present application is how to provide a real-time analysis system based on big data that can accurately obtain the transaction trend of users and thus push goods.

[0005] To solve the above technical problems, the technical solution adopted by the present application is: a real-time analysis system based on big data, comprising:

[0006] A data collection module is deployed on a server of a shopping website and is used to collect log data of the shopping website in real time, wherein the log data includes user behavior log and user transaction log;

[0007] A data transmission module is used to transmit the collected log data to a data preprocessing module for processing through a distributed real-time data transmission channel;

[0008] A data preprocessing module is used to preprocess the received log data, clean and format the log data, and is used for subsequent data analysis;

[0009] A data analysis module is used to analyze the preprocessed log data in real time by using a big data analysis method to obtain the transaction trend of users;

[0010] A data display module is used to display the analysis results to relevant personnel in a visual manner so that they can make corresponding decisions according to the analysis results.

[0011] A further technical solution is that the data collection module obtains the log data of the shopping website by a page embedding method, including the following steps:

[0012] Determine the type of data needed to collect, according to the specific business logic and user flow of the website, determine the location and trigger event of page tracking;

[0013] Select the appropriate tracking tool, ensure that the selected tool can support the required data collection and analysis functions;

[0014] Add tracking code on key pages and events, for user behavior logs, capture user click, scroll and dwell time events, for transaction logs, record various transaction information when the user completes the purchase process, including single number, transaction amount and purchase commodity list.

[0015] Further technical solutions are that the method for pre-processing the log data received by the data preprocessing module specifically includes the following steps:

[0016] Data cleaning: check if there are missing values in the data, and choose to delete or fill according to the situation; identify and process data points deviating from the normal range; detect and delete or merge duplicate records to ensure the uniqueness of the data;

[0017] Data conversion: select features related to analysis tasks, remove irrelevant or redundant features; extract key features through dimension reduction methods; standardize and normalize the data to eliminate the influence of dimension and value range;

[0018] Data integration: through splicing, merging and connecting operations, integrate log data from multiple sources into a unified data set;

[0019] Data normalization: use normalization methods to scale the data to a unified scale for subsequent comparison and analysis;

[0020] Data discretization: for continuous data, perform discretization processing for analysis and modeling;

[0021] Feature construction: according to business requirements and domain knowledge, construct new feature variables through feature combination and feature cross;

[0022] Data verification and evaluation: during and after preprocessing, verify and evaluate the data in processing and the processed data to ensure the effectiveness and correctness of preprocessing.

[0023] Further technical solutions are that the preprocessed log data is connected to a big data real-time analysis system;

[0024] Use the big data real-time analysis system to analyze the preprocessed log data in real time, extract key indicators and features;

[0025] Predict a transaction trend of the user by the extracted key indicators and features.

[0026] The system can collect, transmit and process the log data of the shopping website in real time, ensure the real-time of the analysis result, efficiently process the large-scale log data by using the big data analysis method, improve the analysis efficiency, remove the invalid and error data by preprocessing and cleaning the log data, ensure the accuracy of the analysis result, analyze the preprocessed log data in real time by using the big data analysis method, predict the transaction trend of the user by the extracted key indicators and features, improve the accuracy of the prediction, and display the analysis result in an intuitive visual way to the related personnel, facilitate the decision and optimization of the operation strategy. BRIEF DESCRIPTION OF DRAWINGS

[0027] The application will be further described below in combination with the drawings and specific embodiments.

[0028] Fig. 1 is a principle block diagram of the system according to the embodiment of the application. DETAILED DESCRIPTION

[0029] In the following description, many specific details are set forth in order to provide a thorough understanding of the application. However, the application can be practiced without the specific details, other than in the examples, and it will be apparent to those skilled in the art that the application can be practiced in other ways, and that the application is not limited to the examples disclosed below.

[0030] As shown in Fig. 1, the embodiment of the application discloses a real-time analysis system based on big data, comprising:

[0031] The data collection module 101 is arranged on the server of the shopping website, and is used for collecting the log data of the shopping website in real time, wherein the log data comprises user behavior log and user transaction log.

[0032] The data transmission module 102 is used for transmitting the collected log data to the data preprocessing module through a distributed real-time data transmission channel for processing.

[0033] The data preprocessing module 103 is used for preprocessing the received log data, cleaning and format converting the log data, and is used for subsequent data analysis.

[0034] The data analysis module 104 is used for analyzing the preprocessed log data in real time by using the big data analysis method, and obtaining the transaction trend of the user.

[0035] The data display module 105 is used to display the analysis results in a visual manner to relevant personnel, so that they can make corresponding decisions based on the analysis results.

[0036] Further, the data collection module 101 obtains the log data of the shopping website through the page embedding method, specifically including the following steps:

[0037] Determine the type of data that needs to be collected, such as page views, button clicks, transaction volumes, etc. These are key indicators for analyzing user behavior and transaction conditions. According to the specific business logic and user flow of the website, determine the location and triggering event of the page embedding; for example, set up embedding points on key pages such as product detail pages, shopping cart pages, and checkout pages, as well as key buttons such as adding to cart and immediate purchase.

[0038] Select appropriate embedding tools to ensure that the selected tools can support the required data collection and analysis functions. These tools usually provide rich APIs and interfaces for easy integration and customization.

[0039] Add embedding codes on key pages and events. For user behavior logs, capture user click, scroll, and dwell time events, such as when the user clicks the Add to Cart button, trigger an event and record relevant data (such as user ID, product ID, and click time, etc.). For transaction logs, record various transaction information when the user completes the purchase process, such as order number, transaction amount, and purchased product list.

[0040] The data preprocessing module 103 receives the log data and preprocesses it in the following steps:

[0041] Data cleaning: check if there are missing values in the data, and choose to delete or fill according to the situation. The filling method can be to use mean, median, mode, etc. statistical quantities, or to predict missing values through algorithms. Identify and process data points that deviate from the normal range; detect and delete or merge duplicate records to ensure the uniqueness of the data;

[0042] Data conversion: select features related to the analysis task and remove irrelevant or redundant features; extract key features through dimension reduction methods to reduce the complexity of the data set; standardize and normalize the data to eliminate the influence of dimension and value range;

[0043] Data integration: integrate log data from multiple sources into a unified data set through splicing, merging, and connection operations;

[0044] Data normalization: use min-max normalization or z-score normalization methods to scale the data to a unified scale for subsequent comparison and analysis;

[0045] Data discretization: For continuous data, discretization is performed for analysis and modeling;

[0046] Feature construction: According to business requirements and domain knowledge, new feature variables are constructed through feature combination and feature intersection methods;

[0047] Data verification and evaluation: During and after preprocessing, the data in processing and the processed data are verified and evaluated to ensure the effectiveness and correctness of preprocessing.

[0048] Through the above steps, the data preprocessing module can comprehensively preprocess the received log data, thereby improving the data quality and laying a solid foundation for subsequent data analysis and modeling.

[0049] Further, the preprocessed log data is connected to the big data real-time analysis system, and the big data real-time analysis system is used to analyze the preprocessed log data in real time, extract key indicators and features; through the extracted key indicators and features, the transaction trend of the user is predicted.

[0050] The specific method of using the big data real-time analysis system to analyze the preprocessed log data in real time and extract key indicators and features includes the following steps:

[0051] 1) Constructing a big data real-time analysis system: according to the data volume, processing speed and accuracy requirements, select a suitable big data processing framework, select a suitable data storage and query system for efficient storage and query of real-time data; configure and optimize the big data analysis system, adjust the parameters and settings of the system according to the data characteristics and analysis requirements;

[0052] 2) Real-time data analysis and key indicator extraction: through the data stream processing framework, realize the real-time flow and processing of data, window processing of data stream for incremental analysis and real-time statistics, extract and analyze the features related to the target from the data stream, reduce and select the features, complete the feature extraction; according to the business requirements and analysis target, define and calculate the key indicators, use the real-time data stream to dynamically calculate and update the indicators.

Claims

1. A real-time analytics system based on big data, characterized by Comprise: Data acquisition module: deployed on the server of the shopping website, responsible for real-time collection of log data of the shopping website, including user behavior log and user transaction log; Data transmission module: for transmitting the collected log data to the data preprocessing module through a distributed real-time data transmission channel for processing; Data preprocessing module: for preprocessing the received log data, including log data cleaning and format conversion, for subsequent data analysis; Data analysis module: for real-time analysis of preprocessed log data using big data analysis methods to obtain user transaction trends; Data display module: for displaying the analysis results in a visual manner to relevant personnel for making corresponding decisions based on the analysis results.

2. The big data based real time analytics system as claimed in claim 1, wherein, The data acquisition module obtains the log data of the shopping website through page tracking, including the following steps: Determine the type of data to be collected, determine the location of page tracking and trigger events according to the specific business logic and user flow of the website; Select the appropriate tracking tool to ensure that the selected tool can support the required data collection and analysis functions; Add tracking code on key pages and events, for user behavior log, capture user click, scroll and dwell time events, for transaction log, record various transaction information when the user completes the purchase process, including single number, transaction amount and purchase commodity list.

3. The big data based real time analytics system as claimed in claim 1, wherein, The data preprocessing module receives log data and preprocesses it according to the following steps: Data cleaning: check if there are missing values in the data, and choose to delete or fill according to the situation; identify and process data points deviating from the normal range; detect and delete or merge duplicate records to ensure data uniqueness; Data conversion: select features related to analysis tasks, remove irrelevant or redundant features; extract key features through dimension reduction methods; standardize and normalize the data to eliminate the influence of dimension and value range; Data integration: integrate log data from multiple sources into a unified data set through splicing, merging and connection operations; Data normalization: use normalization methods to scale data to a unified scale for subsequent comparison and analysis; Data discretization: for continuous data, perform discretization processing for analysis and modeling; Feature construction: according to business requirements and domain knowledge, construct new feature variables through feature combination and feature cross method; Data verification and evaluation: verify and evaluate the data during and after preprocessing to ensure the effectiveness and correctness of preprocessing.

4. The real-time analysis system based on big data of claim 1, wherein: The preprocessed log data is connected to the big data real-time analysis system; Use the big data real-time analysis system to analyze the preprocessed log data in real time, extract key indicators and features; Predict user transaction trends through extracted key indicators and features. ​ 5. The big data based real time analytics system as claimed in claim 4, wherein, The specific method for real-time analysis of the preprocessed log data by the big data real-time analysis system and extraction of key indicators and features comprises the following steps: 1) Constructing a big data real-time analysis system: according to the data volume, processing speed and accuracy requirements, selecting a suitable big data processing framework, and selecting a suitable data storage and query system for efficient storage and query of real-time data; configuring and optimizing the big data analysis system, adjusting the parameters and settings of the system according to the data characteristics and analysis requirements; 2) Real-time data analysis and key indicator extraction: through a data stream processing framework, realize the real-time flow and processing of data, window processing of data stream for incremental analysis and real-time statistics, extract and analyze the features related to the target from the data stream, reduce and select the features, and complete the extraction of the features; according to the business requirements and analysis target, define and calculate the key indicators, and use the real-time data stream to dynamically calculate and update the indicators. ​

Images