E-commerce data processing method and system based on big data

Abstract

The invention relates to the technical field of data processing, in particular to an e-commerce data processing method and system based on big data, and the method comprises the steps: constructing a multi-source data collection interface, developing a corresponding data collection plug-in or interface program for different systems and modules of an e-commerce platform, and building a unified data model to integrate data; performing data preprocessing on the integrated data, selecting a proper storage scheme according to the type and access frequency of the data, and establishing a data management mechanism; performing multi-dimensional analysis on the preprocessed data based on big data analysis, and performing data mining on the data by adopting a machine learning algorithm; and applying the data analysis and data mining results to each business link of the e-commerce platform, and establishing a data application feedback mechanism to evaluate the effect of data application. According to the method, the scattered e-commerce data are effectively integrated, processed and mined in combination with big data analysis, and the e-commerce data processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method and system for e-commerce data processing based on big data. Background Technology

[0002] The booming e-commerce industry has generated massive amounts of user data, product data, transaction data, and marketing data. However, this data is often scattered across different systems and modules of e-commerce platforms, such as user registration systems, product management systems, order payment systems, and marketing promotion systems. The data formats are diverse, including structured data (such as order information and user IDs), semi-structured data (such as product descriptions and user reviews), and unstructured data (such as product images and user-uploaded videos).

[0003] Existing e-commerce data processing methods suffer from several problems. First, data integration is difficult; data from different sources and in different formats are hard to merge effectively, forming data silos and making it impossible to fully grasp the overall situation of e-commerce operations. Second, data processing efficiency is low; faced with massive amounts of data, traditional data processing technologies are far from sufficient in terms of processing speed and analytical depth, failing to provide timely support for the operational decisions of e-commerce platforms. Third, data value mining is insufficient; valuable information, such as user consumption preferences and market demand trends for products, cannot be accurately extracted from massive amounts of data, resulting in poor performance in personalized recommendations and precision marketing, affecting user experience and the platform's competitiveness. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, this invention provides a big data-based e-commerce data processing method and system, which solves the problems of data silos, low processing efficiency, and insufficient value mining in existing e-commerce data processing.

[0005] This invention is achieved through the following technical solution: A big data-based e-commerce data processing method is provided, the method comprising the following steps: Step S10: Construct a multi-source e-commerce data collection interface. For different data types of e-commerce platforms, develop corresponding data collection plugins or interface programs to collect e-commerce data, and establish a unified e-commerce data model to integrate e-commerce data, defining data standards and formats. Step S20: Perform data preprocessing on the integrated e-commerce data, select a suitable storage scheme based on the type and access frequency of the e-commerce data, and establish an e-commerce data management mechanism; Step S30: Based on big data analysis, perform multi-dimensional analysis on the preprocessed e-commerce data, and use machine learning algorithms to perform data mining on the e-commerce data; Step S40: Apply the results of data analysis and data mining to various business processes of the e-commerce platform, and establish a data application feedback mechanism to evaluate the effectiveness of data application.

[0006] Preferably, step S10 includes: Multi-source data acquisition: For different data types on e-commerce platforms, develop corresponding data acquisition plugins or interface programs to collect e-commerce data. For structured data, collect directly from relational databases through database connection technology. For semi-structured data, use web crawling technology to collect from the web pages and APP interfaces of e-commerce platforms. During collection, parse data in formats such as JSON or XML. For unstructured data, collect product images and user-uploaded videos through file transfer protocols and add tag information to them for subsequent processing. Establish an e-commerce data model: Establish a unified e-commerce data model, define data standards and formats, and transform and map the collected e-commerce data according to unified field names, data types and encoding rules. At the same time, design e-commerce data association rules to link and integrate e-commerce data from different sources through common key fields to obtain a complete e-commerce database and eliminate data silos.

[0007] Preferably, the step of preprocessing the integrated e-commerce data in step S20 includes: E-commerce data cleaning: Clean the integrated e-commerce data, remove duplicate data, handle missing values, and use mean imputation, median imputation, or predictive imputation methods based on machine learning algorithms for missing values ​​of important fields. For missing values ​​of non-important fields, the record can be directly deleted. Outlier detection and handling: For confirmed outliers, they are corrected or deleted according to the actual situation. Data transformation: Transform the cleaned e-commerce data, including data standardization, data discretization, that is, dividing continuous user age data into different age groups, and data normalization, that is, making data of different magnitudes comparable; Data compression: Due to the massive amount of e-commerce data, data compression technologies are used, including wavelet transform-based compression algorithms or a combination of lossy and lossless compression methods, to compress the pre-processed e-commerce data, reduce data storage capacity, and improve data transmission and processing efficiency.

[0008] Preferably, step S20, which involves selecting a suitable storage scheme and establishing an e-commerce data management mechanism based on the type and access frequency of the e-commerce data, includes: E-commerce data storage: Select an appropriate storage solution based on the type and access frequency of e-commerce data. For structured data and data that requires high-frequency access, store it in a relational database or a distributed relational database. For semi-structured, unstructured data and massive amounts of historical data, store it in a distributed file system. Data Management Mechanism: The constructed e-commerce data management mechanism includes e-commerce data backup and recovery strategies, data access control, and data lifecycle management. The e-commerce data backup and recovery strategy refers to performing full and incremental backups regularly to ensure the security and recoverability of e-commerce data. Data access control refers to assigning different data access permissions according to user roles and responsibilities to prevent data leakage. Data lifecycle management refers to defining the storage period of data and archiving or deleting expired and no longer needed e-commerce data to optimize storage resources.

[0009] Preferably, step S30, which involves performing multi-dimensional analysis of the preprocessed e-commerce data based on big data analytics and using machine learning algorithms to perform data mining on the e-commerce data, includes: Multi-dimensional analysis: Based on a big data analytics engine, preprocessed e-commerce data is analyzed from multiple dimensions. Descriptive analysis methods are used to statistically analyze basic indicators such as total transaction volume, number of orders, number of users, and sales volume of products on the e-commerce platform, generating operational reports. Diagnostic analysis is used to understand the reasons for data changes. User behavior data is analyzed in real time. Stream processing technology is used to monitor users' browsing, searching, and adding items to their shopping carts on the e-commerce platform in real time. When users trigger specific behaviors, relevant promotional information or product recommendations are pushed in real time to improve user experience. Data Mining: Employing machine learning and data mining algorithms to deeply mine e-commerce data, including using collaborative filtering and content-based recommendation algorithms to analyze users' historical purchase records, browsing behavior, and favorites lists to build user profiles and product feature vectors for personalized product recommendations; using clustering algorithms to segment users and identify user groups with different consumption preferences to provide a basis for precision marketing; and using time series analysis algorithms to predict market demand trends and sales of products to assist e-commerce platforms in inventory management and procurement decisions.

[0010] Preferably, step S40, which involves applying the data analysis and data mining results to various business processes of the e-commerce platform and establishing a data application feedback mechanism to evaluate the effectiveness of the data application, includes: Data Application: Apply the results of data analysis and mining to various business processes of the e-commerce platform. In terms of personalized recommendations, based on user profiles and product recommendations, display personalized product lists to users on the homepage and product detail pages of the e-commerce platform. In terms of precision marketing, develop differentiated marketing strategies for different user groups. In terms of inventory management, adjust inventory levels in a timely manner based on product sales forecasts to avoid inventory backlog or stockouts. Data application feedback mechanism: Establish a data application feedback mechanism to evaluate the effectiveness of data application by collecting user metrics such as click-through rate of personalized recommendations, purchase conversion rate, and participation and sales of marketing activities. Based on the evaluation results, optimize the data collection, preprocessing, analysis and mining process and algorithm model to form a closed-loop optimization system and continuously improve the accuracy and effectiveness of data processing.

[0011] Furthermore, to achieve the above objectives, the present invention also proposes a big data-based e-commerce data processing system, which includes: E-commerce multi-source data acquisition and integration module: used to build multi-source e-commerce data acquisition interfaces, develop corresponding data acquisition plugins or interface programs to collect e-commerce data for different data types of e-commerce platforms, establish a unified e-commerce data model to integrate e-commerce data, and define data standards and formats; E-commerce data preprocessing, storage and management module: used to preprocess the integrated e-commerce data, select a suitable storage scheme according to the type and access frequency of the e-commerce data and establish an e-commerce data management mechanism; E-commerce data analysis and mining module: used to perform multi-dimensional analysis of preprocessed e-commerce data based on big data analysis, and to perform data mining on e-commerce data using machine learning algorithms; E-commerce data application and feedback optimization module: This module is used to apply the results of data analysis and data mining to various business processes of the e-commerce platform, and to establish a data application feedback mechanism to evaluate the effectiveness of data application.

[0012] Furthermore, to achieve the above objectives, the present invention also proposes a big data-based e-commerce data processing device, the device comprising: a memory, a processor, and programs such as a big data-based e-commerce data processing algorithm stored in the memory and executable on the processor, wherein the big data-based e-commerce data processing algorithm and other programs are steps for implementing the big data-based e-commerce data processing method described above.

[0013] In addition, to achieve the above objectives, the present invention also provides a computer program product, which includes programs such as a big data-based e-commerce data processing algorithm, wherein when the big data-based e-commerce data processing algorithm is executed by a processor, it implements the big data-based e-commerce data processing method described above.

[0014] The advantages and effects of this invention are: This invention proposes a big data-based e-commerce data processing method and system. Through multi-source data collection and integration, it effectively eliminates e-commerce data silos, achieving unified data management and sharing. Simultaneously, the preprocessing step improves data quality, laying the foundation for subsequent analysis and mining. A reasonable data storage and management scheme ensures efficient storage and access to massive amounts of e-commerce data. Furthermore, through in-depth data analysis and mining, the value of e-commerce data is fully explored, accurately extracting information such as user consumption preferences and product market trends. Through data application and feedback optimization mechanisms, the data processing results can be effectively applied to e-commerce operations and continuously optimized, improving the operational efficiency, personalized service level, and market competitiveness of the e-commerce platform, resulting in significant economic and social benefits. Attached Figure Description

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a flowchart of an e-commerce data processing method based on big data according to the present invention.

[0017] Figure 2 This is a schematic diagram of the structure of an e-commerce data processing system based on big data according to the present invention.

[0018] Figure 3 This is a schematic block diagram of an electronic device for e-commerce data processing based on big data, according to the present invention. Detailed Implementation

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, in one embodiment of the present invention, an e-commerce data processing method based on big data includes the following steps: Step S10: Construct a multi-source e-commerce data collection interface. For different data types of e-commerce platforms, such as user registration data, product management data, order payment data, marketing promotion data, logistics data, and user behavior tracking data, develop corresponding data collection plugins or interface programs to collect e-commerce data, and establish a unified e-commerce data model to integrate e-commerce data, defining data standards and formats.

[0021] Specifically, step S10 includes: Multi-source data acquisition: For different data types on e-commerce platforms, develop corresponding multi-source data acquisition plugins or interface programs to collect e-commerce data. For structured data, use database connection technologies such as JDBC to directly collect data from relational databases such as MySQL and Oracle. For semi-structured data, use web crawling technologies such as Scrapy to collect data from the web pages and APP interfaces of e-commerce platforms. During collection, parse data in formats such as JSON or XML. For unstructured data, use file transfer protocols such as FTP to collect product images and user-uploaded videos, and add tag information such as product category and user ID to them for subsequent processing. Establish an e-commerce data model: Develop a unified e-commerce data model, define data standards and formats, and convert and map the collected e-commerce data according to unified field names, data types, and encoding rules. For example, unify the field representing "User ID" in different data types as "user_id", and unify the date format as "year-month-day, hour:minute:second". Simultaneously, design e-commerce data association rules to link and integrate e-commerce data from different sources through common key fields such as user_id, order_id, and product_id, resulting in a complete e-commerce database and eliminating data silos.

[0022] For example, through multi-source data collection interfaces, basic user information such as age, gender, and address from the e-commerce platform's user registration system, product information such as name, category, price, and description from the product management system, order data such as order number, user_id, product_id, quantity purchased, payment amount, and order time from the order payment system, and user behavior data such as browsed product ID, browsing duration, search keywords, and added-to-cart product ID are collected from the user behavior tracking system. Then, according to a unified data model, this data is transformed and mapped, and integrated through key fields such as user_id and product_id to obtain the e-commerce database.

[0023] Step S20: Perform data preprocessing on the integrated e-commerce data, select a suitable storage scheme based on the type and access frequency of the e-commerce data, and establish an e-commerce data management mechanism.

[0024] Specifically, step S20, which involves preprocessing the integrated e-commerce data, includes: E-commerce data cleaning: Cleaning the integrated e-commerce data, removing duplicate data, such as duplicate order records of the same user, handling missing values, using mean imputation, median imputation, or predictive imputation methods based on machine learning algorithms for missing values ​​of important fields, and directly deleting records with missing values ​​of unimportant fields. Outlier detection and handling, such as detecting outliers using box plots, Z-scores, etc., and correcting or deleting confirmed outliers according to the actual situation. For example, removing duplicate order records and user behavior records, using mean imputation for missing values ​​of fields such as user age and product price, and detecting and deleting abnormal payment amount records using the Z-score method. Data transformation: The cleaned e-commerce data is transformed and processed, including data standardization, such as converting numerical data such as user spending amount and product price into standardized values ​​in the range of [0,1], data discretization, dividing continuous user age data into different age groups, such as 0-18 years old, 19-30 years old, etc., and data normalization, so that data of different magnitudes are comparable. Data compression: Due to the massive amount of e-commerce data, data compression technologies are used, including wavelet transform-based compression algorithms or a combination of lossy and lossless compression methods, to compress the pre-processed e-commerce data, reduce data storage capacity, and improve data transmission and processing efficiency.

[0025] Specifically, step S20, which involves selecting a suitable storage solution and establishing an e-commerce data management mechanism based on the type and access frequency of e-commerce data, includes: E-commerce data storage: Select an appropriate storage solution based on the type and access frequency of e-commerce data. For structured data and data that requires high-frequency access, store it in a relational database or distributed relational database, such as Greenplum. For semi-structured, unstructured data and massive historical data, store it in a distributed file system, such as HDFS, or NoSQL database, such as MongoDB and HBase. Data Management Mechanism: The constructed e-commerce data management mechanism includes e-commerce data backup and recovery strategies, data access control, and data lifecycle management. The e-commerce data backup and recovery strategy refers to performing full and incremental backups regularly to ensure the security and recoverability of e-commerce data. Data access control refers to assigning different data access permissions according to user roles and responsibilities to prevent data leakage. Data lifecycle management refers to defining the storage period of data and archiving or deleting expired and no longer needed e-commerce data to optimize storage resources.

[0026] Step S30: Based on big data analysis, perform multi-dimensional analysis on the preprocessed e-commerce data, and use machine learning algorithms to perform data mining on the e-commerce data.

[0027] Specifically, step S30, which involves multi-dimensional analysis of the preprocessed e-commerce data based on big data analytics and data mining of the e-commerce data using machine learning algorithms, includes: Multi-dimensional analysis: Based on big data analytics engines such as Spark and Flink, preprocessed e-commerce data is analyzed from multiple dimensions. Descriptive analysis methods are used to statistically analyze basic indicators such as total transaction volume, number of orders, number of users, and product sales on the e-commerce platform, generating operational reports. Diagnostic analysis is used to understand the reasons for data changes, such as whether the decline in sales of a certain type of product is due to price increases or the introduction of substitutes by competitors. User behavior data is analyzed in real time. Through streaming processing technologies such as Flink Streaming, user browsing, searching, and adding items to the shopping cart on the e-commerce platform are monitored in real time. When users trigger specific behaviors, such as browsing a certain type of product for a long time, relevant promotional information or product recommendations are pushed in real time to improve user experience. Data Mining: Employing machine learning and data mining algorithms for in-depth analysis of e-commerce data, including using collaborative filtering and content-based recommendation algorithms to analyze users' historical purchase records, browsing behavior, and favorites lists to build user profiles and product feature vectors for personalized product recommendations; using clustering algorithms, such as K-means clustering, to segment users and identify user groups with different consumption preferences, providing a basis for precision marketing; and using time series analysis algorithms, such as the ARIMA model, to predict market demand trends and sales volume of products, assisting e-commerce platforms in inventory management and procurement decisions.

[0028] Step S40: Apply the results of data analysis and data mining to various business processes of the e-commerce platform, and establish a data application feedback mechanism to evaluate the effectiveness of data application.

[0029] Specifically, step S40, which involves applying the data analysis and data mining results to various business processes of the e-commerce platform and establishing a data application feedback mechanism to evaluate the effectiveness of the data application, includes: Data Application: Apply the results of data analysis and mining to various business processes of the e-commerce platform. In terms of personalized recommendations, based on user profiles and product recommendations, display personalized product lists to users on the homepage and product detail pages of the e-commerce platform. In terms of precision marketing, develop differentiated marketing strategies for different user groups, such as pushing coupons to price-sensitive users and recommending luxury goods to high-end users. In terms of inventory management, adjust inventory levels in a timely manner based on product sales forecasts to avoid inventory backlog or stockouts. Data application feedback mechanism: Establish a data application feedback mechanism to evaluate the effectiveness of data application by collecting user metrics such as click-through rate of personalized recommendations, purchase conversion rate, and participation and sales of marketing activities. Based on the evaluation results, optimize the data collection, preprocessing, analysis and mining process and algorithm model to form a closed-loop optimization system and continuously improve the accuracy and effectiveness of data processing.

[0030] In addition, such as Figure 2 As shown, in one embodiment of the present invention, a big data-based e-commerce data processing system is proposed, the system comprising: E-commerce multi-source data collection and integration module: Used to build multi-source e-commerce data collection interfaces. For different data types of e-commerce platforms, such as user registration data, product management data, order payment data, marketing promotion data, logistics data, and user behavior tracking data, corresponding data collection plugins or interface programs are developed to collect e-commerce data, and a unified e-commerce data model is established to integrate e-commerce data, defining data standards and formats. E-commerce data preprocessing, storage and management module: used to preprocess the integrated e-commerce data, select a suitable storage scheme according to the type and access frequency of the e-commerce data and establish an e-commerce data management mechanism; E-commerce data analysis and mining module: used to perform multi-dimensional analysis of preprocessed e-commerce data based on big data analysis, and to perform data mining on e-commerce data using machine learning algorithms; E-commerce data application and feedback optimization module: This module is used to apply the results of data analysis and data mining to various business processes of the e-commerce platform, and to establish a data application feedback mechanism to evaluate the effectiveness of data application.

[0031] This application provides a big data-based e-commerce data processing system, employing a big data-based e-commerce data processing method as described in the above embodiments, which can solve the technical problem of low efficiency in traditional e-commerce data processing methods. Compared with the prior art, the beneficial effects of the big data-based e-commerce data processing system provided in this application are the same as those of the big data-based e-commerce data processing method provided in the above embodiments, and other technical features of the big data-based e-commerce data processing system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0032] This application provides a big data-based e-commerce data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the big data-based e-commerce data processing method in the above embodiment 1.

[0033] like Figure 3 As shown in the illustration, in one embodiment of the present invention, a structural schematic diagram of a big data-based e-commerce data processing device suitable for implementing the embodiments of this application is presented. The big data-based e-commerce data processing device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The e-commerce data processing device based on big data shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0034] Figure 3The illustrated e-commerce data processing device based on big data may include a processing system 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1002 or programs loaded from storage system 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the e-commerce data processing device based on big data. The processing system 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input systems 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output systems 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage systems 1003 including, for example, magnetic tapes, hard disks, etc.; and communication systems 1009. Communication system 1009 allows a big data-based e-commerce data processing device to exchange data with other devices wirelessly or via wired communication. Although the figure shows a big data-based e-commerce data processing device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0035] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication system, or installed from storage system 1003, or installed from ROM 1002. When the computer program is executed by processing system 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0036] This application provides a big data-based e-commerce data processing device, which employs a big data-based e-commerce data processing method as described in the above embodiments, and can solve the technical problem of low efficiency in traditional e-commerce data processing methods. Compared with the prior art, the beneficial effects of the big data-based e-commerce data processing device provided in this application are the same as those of the big data-based e-commerce data processing method provided in the above embodiments, and other technical features of this big data-based e-commerce data processing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0037] The various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0038] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described e-commerce data processing method based on big data.

[0039] The computer program product provided in this application can solve the technical problem of low efficiency in traditional e-commerce data processing methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the big data-based e-commerce data processing method provided in the above embodiments, and will not be repeated here.

[0040] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing e-commerce data based on big data, characterized in that, The method includes the following steps: Step S10: Construct a multi-source e-commerce data collection interface. For different data types of e-commerce platforms, develop corresponding data collection plugins or interface programs to collect e-commerce data, and establish a unified e-commerce data model to integrate the e-commerce data. Step S20: Perform data preprocessing on the integrated e-commerce data, select a suitable storage scheme based on the type and access frequency of the e-commerce data, and establish an e-commerce data management mechanism; Step S30: Based on big data analysis, perform multi-dimensional analysis on the preprocessed e-commerce data, and use machine learning algorithms to perform data mining on the e-commerce data; Step S40: Apply the results of data analysis and data mining to various business processes of the e-commerce platform, and establish a data application feedback mechanism to evaluate the effectiveness of data application.

2. The e-commerce data processing method based on big data according to claim 1, characterized in that, Step S10 includes: Multi-source data acquisition: For different data types on e-commerce platforms, develop corresponding data acquisition plugins or interface programs to collect e-commerce data. For structured data, collect directly from relational databases through database connection technology. For semi-structured data, use web crawling technology to collect from the web pages and APP interfaces of e-commerce platforms. During the collection process, parse the data in JSON or XML format. For unstructured data, collect product images and user-uploaded videos through file transfer protocols and add tag information to them. Establish an e-commerce data model: Establish a unified e-commerce data model, define data standards and formats, and transform and map the collected e-commerce data according to unified field names, data types and encoding rules. At the same time, design e-commerce data association rules to link and integrate e-commerce data from different sources through common fields to obtain a complete e-commerce database.

3. The e-commerce data processing method based on big data according to claim 1, characterized in that, The step of preprocessing the integrated e-commerce data in step S20 includes: E-commerce data cleaning: Clean the integrated e-commerce data to remove duplicate data, and handle missing values ​​and outlier detection and processing. For confirmed outliers, correct or delete them according to the actual situation. Data transformation: Transform the cleaned e-commerce data, including data standardization, data discretization, dividing continuous user age data into different age groups, and data normalization; Data compression: Data compression techniques, including wavelet transform-based compression algorithms or a combination of lossy and lossless compression, are used to compress preprocessed e-commerce data.

4. The e-commerce data processing method based on big data according to claim 1, characterized in that, The steps in step S20, which involve selecting a suitable storage solution and establishing an e-commerce data management mechanism based on the type and access frequency of e-commerce data, include: E-commerce data storage: Select an appropriate storage solution based on the type and access frequency of e-commerce data. For structured data and data that requires high-frequency access, store it in a relational database or a distributed relational database. For semi-structured, unstructured, and historical data, store it in a distributed file system. Data Management Mechanism: The constructed e-commerce data management mechanism includes e-commerce data backup and recovery strategy, data access permission control, and data lifecycle management. The e-commerce data backup and recovery strategy refers to performing full backups and incremental backups regularly. Data access permission control refers to assigning different data access permissions according to user roles and responsibilities. Data lifecycle management refers to defining the storage period of data and archiving or deleting e-commerce data that has expired and is no longer needed.

5. The e-commerce data processing method based on big data according to claim 1, characterized in that, The steps in step S30, which involve multi-dimensional analysis of the preprocessed e-commerce data based on big data analytics and data mining of the e-commerce data using machine learning algorithms, include: Multi-dimensional analysis: Based on the big data analysis engine, the preprocessed e-commerce data is analyzed from multiple dimensions. Descriptive analysis methods are used to statistically analyze the basic indicators of e-commerce platform, such as total transaction volume, number of orders, number of users, and sales volume of goods, and to generate operation reports. Diagnostic analysis is used to understand the reasons for data changes. User behavior data is analyzed in real time. Stream processing technology is used to monitor users' browsing, searching, and adding to cart behaviors on the e-commerce platform in real time. When users trigger specific behaviors, relevant promotional information or product recommendations are pushed in real time. Data Mining: Employing machine learning and data mining algorithms to mine e-commerce data, including using collaborative filtering and content-based recommendation algorithms to analyze users' historical purchase records, browsing behavior, and favorites lists to build user profiles and product feature vectors, enabling personalized product recommendations; using clustering algorithms to segment users and identify user groups with different consumption preferences; and using time series analysis algorithms to predict market demand trends and sales of products, assisting e-commerce platforms in inventory management and procurement decisions.

6. The e-commerce data processing method based on big data according to claim 1, characterized in that, The steps in S40, which involve applying the data analysis and data mining results to various business processes of the e-commerce platform and establishing a data application feedback mechanism to evaluate the effectiveness of the data application, include: Data Application: Apply the results of data analysis and mining to various business processes of the e-commerce platform. In terms of personalized recommendations, based on user profiles and product recommendations, display personalized product lists to users on the homepage and product detail pages of the e-commerce platform. In terms of precision marketing, develop differentiated marketing strategies for different user groups. In terms of inventory management, adjust inventory levels in a timely manner based on product sales forecasts. Data application feedback mechanism: Establish a data application feedback mechanism to evaluate the effectiveness of data application by collecting user click-through rates, purchase conversion rates, marketing campaign participation and sales metrics. Based on the evaluation results, re-optimize the data collection, preprocessing, analysis and mining processes and algorithm models to form a closed-loop optimization system.

7. A big data-based e-commerce data processing system, characterized in that, The aforementioned big data-based e-commerce data processing system includes: E-commerce multi-source data acquisition and integration module: used to build multi-source e-commerce data acquisition interfaces, develop corresponding data acquisition plugins or interface programs for different data types of e-commerce platforms to collect e-commerce data, and establish a unified e-commerce data model to integrate e-commerce data; E-commerce data preprocessing, storage and management module: used to preprocess the integrated e-commerce data, select a suitable storage scheme according to the type and access frequency of the e-commerce data and establish an e-commerce data management mechanism; E-commerce data analysis and mining module: used to perform multi-dimensional analysis of preprocessed e-commerce data based on big data analysis, and to perform data mining on e-commerce data using machine learning algorithms; E-commerce data application and feedback optimization module: This module is used to apply the results of data analysis and data mining to various business processes of the e-commerce platform, and to establish a data application feedback mechanism to evaluate the effectiveness of data application.

8. A big data-based e-commerce data processing device, characterized in that, The aforementioned big data-based e-commerce data processing device includes: The present invention includes a memory, a processor, and a big data-based e-commerce data processing program stored on the memory and executable on the processor, wherein the big data-based e-commerce data processing program, when executed by the processor, implements a big data-based e-commerce data processing method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a big data-based e-commerce data processing program, which, when executed by a processor, implements a big data-based e-commerce data processing method as described in any one of claims 1 to 6.

Images