Sales data statistical method and system based on cloud platform
By using a cloud-based sales data statistics method and leveraging the Apache Spark distributed computing framework and Spark SQL, the problems of slow data processing speed and limited storage space in traditional methods are solved. This enables fast, secure, and flexible data processing and analysis, supporting real-time business decision-making and sales optimization for enterprises.
Patent Information
- Application Number
- CN202410921792.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional sales data statistics methods suffer from slow processing speed, limited storage space, and difficulties in data backup and recovery when dealing with massive amounts of data. Furthermore, enterprises find it difficult to effectively utilize the advantages of cloud computing after migrating to a cloud platform.
We adopt a cloud-based sales data statistics method, using the Apache Spark distributed computing framework for data collection, storage, cleaning and analysis, combined with Spark SQL and ML libraries for feature extraction and data analysis, and leveraging the elastic scaling and automated operation and maintenance capabilities of the cloud platform to achieve rapid data collection, storage and processing.
It enables the rapid collection, storage, and processing of massive sales data, provides real-time analysis results, accurately extracts user behavior data, reduces system maintenance and management costs, supports business decisions and optimizes sales strategies, and also has data security and flexible resource adjustment capabilities.
Smart Images

Figure CN119027149A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of sales data statistics, and in particular to a sales data statistics method and system based on a cloud platform. Background Art
[0002] With the rapid development of information technology and the popularization of the Internet, e-commerce and online sales have risen rapidly and become one of the important sales channels for enterprises. However, with the continuous growth and complexity of sales data, how to effectively collect, store, process and analyze this data to obtain valuable business insights has become a major challenge facing enterprises.
[0003] Traditional sales data statistics methods often rely on local servers or single-machine processing. This method has many limitations when processing massive data, such as slow processing speed, limited storage space, and difficulty in data backup and recovery. In addition, with the maturity and widespread application of cloud computing technology, more and more companies are beginning to migrate their businesses to cloud platforms to take full advantage of cloud computing's elastic scalability, high availability, and low cost. Summary of the invention
[0004] In view of this, the present invention proposes a sales data statistics method and system based on a cloud platform, which can solve the defects of the prior art such as slow processing speed, limited storage space, and difficult data backup and recovery.
[0005] The technical solution of the present invention is achieved in this way:
[0006] A sales data statistics method based on a cloud platform, specifically comprising:
[0007] Collect and store sales data in the cloud;
[0008] Read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data;
[0009] Based on the Apache Spark distributed computing framework, the pre-processed sales data is analyzed to obtain user behavior data;
[0010] Commodity sales statistics are performed based on user behavior data to obtain sales statistics results.
[0011] As a further optional solution of the sales data statistics method based on the cloud platform, the reading of the sales data stored in the cloud specifically includes:
[0012] Interact with the cloud based on Spark’s DataFrame API;
[0013] The spark.read method based on Spark SQL reads the structured data in the cloud sales data;
[0014] Read unstructured data in cloud sales data based on Spark's RDD API.
[0015] As a further optional solution of the sales data statistics method based on the cloud platform, the data cleaning preprocessing of the sales data specifically includes:
[0016] Use the dropDuplicates method of DataFrame to remove duplicate data from the sales data;
[0017] Use the fill na method to fill missing values in sales data;
[0018] Use Spark cluster functions to identify and process anomalous data.
[0019] As a further optional solution of the sales data statistics method based on the cloud platform, the pre-processed sales data is analyzed based on the Apache Spark distributed computing framework to obtain user behavior data, which specifically includes:
[0020] Use Spark SQL in the Apache Spark distributed computing framework to perform feature extraction on the preprocessed sales data to obtain feature information related to user behavior;
[0021] Using the Spark ML lib machine learning library, clustering algorithms, classification algorithms, and association rule mining algorithms are applied to conduct in-depth analysis of feature information related to user behavior to obtain user behavior data.
[0022] As a further optional solution of the sales data statistics method based on the cloud platform, the Spark SQL in the Apache Spark distributed computing framework is used to perform feature extraction operations on the pre-processed sales data to obtain feature information related to user behavior, specifically including:
[0023] Extract basic statistical features based on Spark SQL to obtain the user's behavior frequency, activity, and preferred product categories;
[0024] Analyze user behavior paths based on user behavior frequency, activity, and preferred product categories;
[0025] Based on the user's behavior path, characteristic information related to the user's behavior is obtained.
[0026] As a further optional solution of the sales data statistics method based on the cloud platform, the sales statistics of commodities are performed based on the user behavior data to obtain the sales statistics results, which specifically includes:
[0027] Use the groupBy function of DataFrame to group by product ID to get the grouping results;
[0028] Use the agg addition function of DataFrame to calculate the total sales quantity of each group and obtain the product sales statistics.
[0029] As a further optional solution of the sales data statistics method based on the cloud platform, the method further includes:
[0030] Send product sales statistics back to cloud storage;
[0031] Use the visualization tools provided by the Spark cluster to export the product sales statistics results into visualization charts.
[0032] A sales data statistics system based on a cloud platform, comprising:
[0033] A collection module is used to collect and store sales data in the cloud;
[0034] The reading module is used to read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data;
[0035] The data analysis module is used to analyze the pre-processed sales data based on the Apache Spark distributed computing framework to obtain user behavior data;
[0036] The statistics module is used to perform commodity sales statistics based on user behavior data and obtain sales statistics results.
[0037] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the above-mentioned cloud platform-based sales data statistics method when executing the computer program.
[0038] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the above-mentioned cloud platform-based sales data statistics methods.
[0039] The beneficial effects of the present invention are as follows: by utilizing the powerful computing and storage capabilities of the cloud platform and combining it with the Apache Spark distributed computing framework, it is possible to quickly collect, store and process massive sales data. This efficient data processing capability enables enterprises to obtain real-time analysis results of sales data more quickly, providing strong support for business decisions. At the same time, based on Apache Spark, in-depth analysis of the pre-processed sales data can accurately extract user behavior data, which includes purchasing preferences, browsing habits, search keywords, etc., which helps enterprises better understand user needs and optimize sales strategies. In addition, deploying the sales data statistics system on the cloud platform can make full use of the cloud platform's automated operation and maintenance functions to reduce the system's maintenance and management costs. At the same time, the cloud platform's pay-as-you-go model also enables enterprises to flexibly adjust resource investment according to actual usage and reduce operating costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flow chart of a sales data statistics method based on a cloud platform according to the present invention;
[0042] Figure 2 The present invention is a schematic diagram of the composition of a sales data statistics system based on a cloud platform. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] refer to Figure 1-2 , a sales data statistics method based on a cloud platform, specifically comprising:
[0045] Collect and store sales data in the cloud;
[0046] Read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data;
[0047] Based on the Apache Spark distributed computing framework, the pre-processed sales data is analyzed to obtain user behavior data;
[0048] Commodity sales statistics are performed based on user behavior data to obtain sales statistics results.
[0049] In this embodiment, by utilizing the powerful computing and storage capabilities of the cloud platform and combining it with the Apache Spark distributed computing framework, it is possible to quickly collect, store and process massive sales data. This efficient data processing capability enables enterprises to obtain real-time analysis results of sales data more quickly, providing strong support for business decisions. At the same time, based on Apache Spark, in-depth analysis of the pre-processed sales data can accurately extract user behavior data. These user behavior data include purchasing preferences, browsing habits, search keywords, etc., which help enterprises better understand user needs and optimize sales strategies. In addition, deploying the sales data statistics system on the cloud platform can make full use of the cloud platform's automated operation and maintenance functions to reduce the system's maintenance and management costs. At the same time, the cloud platform's pay-as-you-go model also enables enterprises to flexibly adjust resource investment according to actual usage and reduce operating costs.
[0050] It should be noted that the elastic scaling capability provided by the cloud platform enables the sales data statistics system to automatically adjust computing and storage resources according to business needs to cope with the growth and changes in data volume. This scalability ensures the stability and reliability of the system, and it can maintain efficient operation even when the data volume surges. By pre-processing sales data through data cleaning, noise, outliers and duplicate records in the data can be eliminated to improve data quality, which helps to ensure the accuracy and reliability of subsequent data analysis, thereby deriving more valuable business insights. By collecting and analyzing sales data in real time, companies can quickly obtain sales statistics results. This real-time nature enables companies to adjust sales strategies in a timely manner, seize market opportunities, and improve sales efficiency. Cloud platforms usually provide multi-level security protection mechanisms, including data encryption, access control, security auditing, etc. These mechanisms can ensure the security of sales data during storage, transmission and processing, and prevent data leakage and illegal access.
[0051] Preferably, the reading of the sales data stored in the cloud specifically includes:
[0052] Interact with the cloud based on Spark's DataFrame API;
[0053] The spark.read method based on Spark SQL reads the structured data in the cloud sales data;
[0054] Read unstructured data in cloud sales data based on Spark's RDD API.
[0055] In this embodiment, Spark's DataFrame API is used to interact with the cloud, which can make full use of Spark's distributed computing capabilities to achieve parallel reading of cloud data, thereby greatly improving the efficiency of data reading. SparkSQL's spark.read method provides an optimized reading strategy for structured data, which can quickly parse and load data into Spark SQL's DataFrame, providing an efficient data structure for subsequent data processing and analysis. At the same time, DataFrame API and Spark SQL provide rich query and conversion operations for structured data, support SQL syntax and DataFrame API functions, so that various complex data processing and analysis can be performed directly after the data is read. RDD API (Resilient Distributed Datasets) provides native support for unstructured data, and can read unstructured data in various formats, such as text files, pictures, videos, etc., and perform customized data processing logic. In addition, through Spark SQL's spark.read method, structured data on the cloud can be directly read as DataFrame without additional data format conversion, simplifying the data processing process. The API provides a wealth of transformation and action operations, which can directly process unstructured data without converting the data into other formats, further simplifying the data processing process; in addition, based on Spark's distributed computing framework, the solution has good scalability and maintainability. As the business develops and data grows, computing resources can be easily increased to cope with greater data processing needs. Spark's ecosystem provides a wealth of tools and libraries that can easily integrate other data processing and analysis tools to improve the overall performance of the system.
[0056] Preferably, the data cleaning preprocessing of the sales data specifically includes:
[0057] Use the dropDuplicates method of DataFrame to remove duplicate data from the sales data;
[0058] Use the fill na method to fill missing values in sales data;
[0059] Use Spark cluster functions to identify and process anomalous data.
[0060] In this embodiment, the dropDuplicates method of DataFrame can be used to efficiently remove duplicate records in sales data. In a big data environment, duplicate data may take up a lot of storage space and affect the accuracy of data analysis. Through deduplication processing, data redundancy can be significantly reduced and data processing efficiency can be improved. At the same time, using file The lna method fills the missing values of sales data to ensure the integrity of the data. Missing values may cause information loss or mislead analysis results in data analysis. Through reasonable filling strategies (such as using mean, median, mode or predicted value, etc.), the impact of missing values on data integrity can be compensated and the accuracy of data analysis can be improved. In addition, combining Spark cluster functions to identify and process abnormal data can effectively improve data quality. Abnormal data may be caused by system errors, input errors or special events, which may have an adverse effect on data analysis results. Through cluster functions, abnormal data can be quickly found and processed (such as deletion, correction or marking), thereby improving data accuracy and reliability. In addition, since Spark is a distributed computing framework, the above-mentioned data cleaning and preprocessing operations can be executed in parallel on the Spark cluster, thereby greatly improving processing efficiency. This means that even in the face of massive sales data, data cleaning and preprocessing can be completed in a short time, providing timely and accurate data support for subsequent data analysis.
[0061] Preferably, the data analysis of the pre-processed sales data based on the Apache Spark distributed computing framework to obtain user behavior data specifically includes:
[0062] Use Spark SQL in the Apache Spark distributed computing framework to perform feature extraction on the preprocessed sales data to obtain feature information related to user behavior;
[0063] Using the Spark ML lib machine learning library, clustering algorithms, classification algorithms, and association rule mining algorithms are applied to conduct in-depth analysis of feature information related to user behavior to obtain user behavior data.
[0064] In this embodiment, Apache Spark, as an efficient distributed computing framework, can process large-scale data sets. Using Spark SQL for feature extraction operations can make full use of Spark's parallel computing capabilities to quickly and efficiently process preprocessed sales data, thereby shortening the data analysis cycle. At the same time, Spark SQL supports SQL query language, allowing users to extract features from preprocessed sales data through simple SQL statements. This method is both intuitive and flexible, and feature information related to user behavior can be extracted without complex methods. In addition, using the SparkML lib machine learning library, various algorithms can be applied to conduct in-depth analysis of feature information related to user behavior. Clustering algorithms can help discover the similarity of user groups, classification algorithms can predict user behavior or preferences, and association rule mining algorithms can discover the correlation between different user behaviors. These in-depth analyses can provide enterprises with valuable user behavior data, thereby supporting more accurate marketing strategies and decisions. In addition, Spark supports stream processing (SparkStreaming), which can be used in combination with Spark SQL and ML lib to achieve real-time data analysis, which means that enterprises can monitor user behavior in real time and immediately obtain analysis results, so as to respond and adjust in time.
[0065] Preferably, the use of Spark SQL in the Apache Spark distributed computing framework to perform feature extraction operations on the preprocessed sales data to obtain feature information related to user behavior specifically includes:
[0066] Extract basic statistical features based on Spark SQL to obtain the user's behavior frequency, activity, and preferred product categories;
[0067] Analyze user behavior paths based on user behavior frequency, activity, and preferred product categories;
[0068] Based on the user's behavior path, characteristic information related to the user's behavior is obtained.
[0069] In this embodiment, Spark SQL allows users to extract features from large-scale data sets through SQL query statements. This declarative approach greatly simplifies the complexity of data processing while maintaining high processing efficiency. It can quickly count basic information such as user behavior frequency, activity, and preferred product categories, providing a basis for subsequent in-depth analysis. Based on the user's behavior frequency, activity, and preferred product categories, Spark SQL can further analyze the user's behavior path, which helps companies understand the user's behavioral habits and trends during the shopping process, such as the user's shopping frequency, shopping time distribution, and preferred product types. The user's behavior path analysis can provide companies with rich user behavior feature information, which can be used to build user portraits and help companies better understand user needs and market trends. For example, companies can recommend related products based on users' shopping preferences to improve user satisfaction and purchase conversion rates. Spark SQL's distributed computing capabilities enable the solution to easily cope with the processing needs of large-scale data sets. At the same time, Spark SQL supports a variety of data sources and data formats, making the data extraction process highly flexible and scalable. Spark SQL can be used with other Spark components (such as Spark Streaming) to achieve real-time data analysis, which means that enterprises can monitor user behavior in real time and immediately obtain relevant feature information, so as to quickly respond to market changes and user needs. By deeply understanding the feature information related to user behavior, enterprises can formulate more accurate marketing strategies. For example, enterprises can optimize the product recommendation system according to the user's shopping preferences and behavior paths, improve user satisfaction and purchase conversion rate. In addition, enterprises can also formulate personalized promotional activities and coupon strategies based on user behavior data to improve marketing effectiveness.
[0070] Preferably, the performing of commodity sales statistics based on user behavior data to obtain sales statistics results specifically includes:
[0071] Use the groupBy function of DataFrame to group by product ID to get the grouping results;
[0072] Use the agg addition function of DataFrame to calculate the total sales quantity of each group and obtain the product sales statistics.
[0073] In this embodiment, through the groupBy function, DataFrame can efficiently group data by product ID, which is particularly important for processing large-scale data sets because it can reduce the complexity and time of data processing. The agg function allows aggregation operations on grouped data, such as calculating the total sales quantity, which can quickly provide key business indicators.
[0074] Preferably, the method further comprises:
[0075] Send product sales statistics back to cloud storage;
[0076] Use the visualization tools provided by the Spark cluster to export the product sales statistics results into visualization charts.
[0077] In this embodiment, the commodity sales statistics are sent back to cloud storage, such as HDFS, S3, etc., to ensure the persistent storage of data and avoid the risk of data loss. Cloud storage is usually highly scalable and can automatically adjust the storage space according to the growth of data to meet the growing data storage needs. Cloud storage allows data to be accessed at any time and any place, facilitating data sharing and collaboration among team members. At the same time, through cloud storage, users can set data access permissions as needed to ensure data security. Cloud storage supports multiple data access methods, such as API calls, Web interfaces, etc., making data access more flexible and convenient. In addition, Use the visualization tools provided by the Spark cluster to export the sales statistics results into visualization charts, which can intuitively display the changing trends and distribution of sales data. Visualization charts can simplify the process of understanding data, so that non-technical personnel can also quickly understand the meaning behind the data. Visualization charts also support a variety of chart types (such as bar charts, line charts, pie charts, etc.), and you can choose the appropriate chart type to display data as needed. In addition, through visualization charts, companies can obtain key information of sales data more quickly and provide strong support for business decisions. The data and chart types in visualization charts can be customized as needed to ensure the accuracy and reliability of decision analysis.
[0078] A sales data statistics system based on a cloud platform, comprising:
[0079] A collection module is used to collect and store sales data in the cloud;
[0080] The reading module is used to read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data;
[0081] The data analysis module is used to analyze the pre-processed sales data based on the Apache Spark distributed computing framework to obtain user behavior data;
[0082] The statistics module is used to perform commodity sales statistics based on user behavior data and obtain sales statistics results.
[0083] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the above-mentioned cloud platform-based sales data statistics method when executing the computer program.
[0084] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the above-mentioned cloud platform-based sales data statistics methods.
[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A sales data statistics method based on a cloud platform, characterized in that: Specifically include: Collect and store sales data in the cloud; Read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data; Based on the Apache Spark distributed computing framework, the pre-processed sales data is analyzed to obtain user behavior data; Commodity sales statistics are performed based on user behavior data to obtain sales statistics results.
2. The sales data statistics method based on the cloud platform according to claim 1, characterized in that: The reading of the sales data stored in the cloud specifically includes: Interact with the cloud based on Spark's DataFrame API; The spark.read method based on Spark SQL reads the structured data in the cloud sales data; Read unstructured data in cloud sales data based on Spark's RDD API.
3. A sales data statistics method based on a cloud platform according to claim 2, characterized in that: The data cleaning preprocessing of the sales data specifically includes: Use the dropDuplicates method of DataFrame to remove duplicate data from the sales data; Use the fillna method to fill missing values in sales data; Use Spark cluster functions to identify and process anomalous data.
4. A cloud platform-based sales data statistics method according to claim 3, characterized in that: The pre-processed sales data is analyzed based on the Apache Spark distributed computing framework to obtain user behavior data, specifically including: Use Spark SQL in the Apache Spark distributed computing framework to perform feature extraction on the preprocessed sales data to obtain feature information related to user behavior; Using the Spark MLlib machine learning library, clustering algorithms, classification algorithms, and association rule mining algorithms are applied to conduct in-depth analysis of feature information related to user behavior to obtain user behavior data.
5. The sales data statistics method based on the cloud platform according to claim 4 is characterized in that: The Spark SQL in the Apache Spark distributed computing framework is used to perform feature extraction operations on the preprocessed sales data to obtain feature information related to user behavior, specifically including: Extract basic statistical features based on Spark SQL to obtain the user's behavior frequency, activity, and preferred product categories; Analyze user behavior paths based on user behavior frequency, activity, and preferred product categories; Based on the user's behavior path, characteristic information related to the user's behavior is obtained.
6. A cloud platform-based sales data statistics method according to claim 5, characterized in that: The commodity sales statistics are performed based on the user behavior data to obtain the sales statistics results, specifically including: Use the groupBy function of DataFrame to group by product ID to get the grouping results; Use the agg addition function of DataFrame to calculate the total sales quantity of each group and obtain the product sales statistics.
7. A cloud platform-based sales data statistics method according to claim 6, characterized in that: The method further comprises: Send product sales statistics back to cloud storage; Use the visualization tools provided by the Spark cluster to export the product sales statistics results into visualization charts.
8. A sales data statistics system based on a cloud platform, characterized in that: include: A collection module is used to collect and store sales data in the cloud; The reading module is used to read the sales data stored in the cloud and perform data cleaning and preprocessing on the sales data; The data analysis module is used to analyze the pre-processed sales data based on the Apache Spark distributed computing framework to obtain user behavior data; The statistics module is used to perform commodity sales statistics based on user behavior data and obtain sales statistics results.
9. A computing device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the cloud platform-based sales data statistics methods in claims 1-7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the cloud platform-based sales data statistics methods in claims 1-7.
