Shopping webpage design platform and method based on big data collection

Abstract

The invention discloses a shopping webpage design platform and method based on big data acquisition, and belongs to the technical field of internet big data, the platform comprises a data acquisition module, a data storage module, a data processing and analysis module and an information generation module, and the method comprises the steps of data acquisition; storing and processing data; feature extraction and analysis; and generating design information. According to the method, the preference and the behavior mode of the user can be comprehensively understood, so that commodities which better meet personalized requirements of the user can be recommended to the user.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet big data, and in particular relates to a shopping webpage design platform and method based on big data collection. Background Art

[0002] Nowadays, shopping websites generally use recommendation algorithms to recommend items on shopping websites. Although recommendation algorithms have significant advantages in the function of recommending items on shopping websites, the current recommendation system still has some shortcomings. These shortcomings mainly include data sparsity, cold start problems, and lack of real-time performance.

[0003] Data sparsity means that the interaction data between users and items is often very sparse, and many users only interact with a small number of items. This results in the lack of sufficient data support for collaborative filtering algorithms when calculating similarities, making it difficult to give accurate recommendation results. In order to deal with data sparsity, matrix decomposition, missing value filling and other technologies are usually used, but these methods may introduce noise data and affect the accuracy of recommendations.

[0004] The cold start problem includes user cold start and item cold start, that is, for new users and new items, the system lacks sufficient historical data and cannot make effective recommendations. Common coping methods include using content recommendations and guiding users to make initial ratings, but these methods often require additional user operations and may reduce user experience.

[0005] The problem of insufficient real-time performance is that the recommendation system needs to process and analyze large-scale user behavior data. Traditional batch processing methods cannot meet the needs of real-time recommendation, resulting in delayed recommendation results. Although technologies such as stream processing and online learning have been introduced to improve real-time performance, these methods usually require high computing resources and complex system architecture, increasing maintenance costs. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a shopping webpage design platform and method based on big data collection.

[0007] The object of the present invention is achieved through the following technical solutions:

[0008] A shopping webpage design platform based on big data collection, the platform comprising:

[0009] A data collection module, which is used to obtain web page data and user behavior data;

[0010] A data storage module, the data storage module is used to store the web page data and the user behavior data;

[0011] A data processing and analysis module, which is used to extract web design features from web page data, extract user preferences and needs from user behavior data, and output data analysis results to the information generation module;

[0012] The information generation module stores design rules described in a domain-specific language, and automatically generates a shopping web page template according to a preset recommendation algorithm in combination with data analysis results and the design rules.

[0013] Furthermore, the recommendation algorithm includes a hybrid recommendation algorithm, which includes using a weighted average method to calculate the average score of user-based collaborative filtering predictions, item-based collaborative filtering predictions, and content recommendation algorithm predictions, with weight coefficients pre-set and the sum of the weight coefficients being 1.

[0014] Furthermore, the design platform also includes a display and feedback module for real-time preview and adjustment of the generated web page design, and optimization of the design rules based on user feedback.

[0015] Furthermore, the web page data includes text, pictures and layout data, and the user behavior data includes user click, scroll and dwell time data.

[0016] Furthermore, the data processing and analysis module also includes using statistical analysis and machine learning methods to analyze user behavior data and generate user preference and demand portraits, and the information generation module generates personalized design solutions based on user preferences and demand portraits.

[0017] Further, the data storage module includes a distributed file storage unit and an unstructured database;

[0018] Among them, the distributed file storage unit is used to store web page data and user behavior data, and the unstructured database is used to store unstructured data, which includes text and images.

[0019] Furthermore, the data processing and analysis module also includes a data cleaning and preprocessing unit for data deduplication, format conversion and noise filtering.

[0020] On the other hand, the present invention further provides a shopping webpage design method based on big data collection, the method is implemented based on any of the aforementioned webpage design platforms, and the method comprises:

[0021] Start the data collection module to crawl web page data on the Internet and collect user behavior data on different web pages;

[0022] Storing the acquired web page data and user behavior data in a data storage module;

[0023] The data processing and analysis module extracts web design features from web page data and analyzes user behavior data to extract user preferences and needs;

[0024] The information generation module is based on design rules and combines the feature extraction results and behavior analysis results output by the data processing and analysis module to generate a web page design template according to a preset hybrid recommendation algorithm.

[0025] The beneficial effects of the present invention are:

[0026] (1) The present invention can more comprehensively understand the user's preferences and behavior patterns, thereby designing web pages that can recommend products that better meet the user's personalized needs, thereby enhancing the user experience and the accuracy of purchasing decisions.

[0027] (2) The hybrid recommendation algorithm adopted by the present invention can effectively overcome the limitations of a single algorithm. By combining content information and user behavior data, it can more accurately recommend new users and new items, thereby improving the coverage of the system and the participation of new users. The hybrid recommendation algorithm can dynamically adjust and optimize the recommendation results through flexible algorithm combination and trade-offs, and can quickly adjust to user preferences and market changes.

[0028] (3) The present invention can improve the diversity and personalization level of recommendation results, avoid the "filter bubble" problem in traditional recommendation algorithms, and enable users to be exposed to a wider and more diverse range of product choices. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural block diagram of a shopping webpage design platform based on big data collection according to an embodiment of the present invention;

[0030] Figure 2 It is a flowchart of a shopping webpage design method based on big data collection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0032] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0033] Nowadays, shopping websites generally use recommendation algorithms to recommend items on shopping websites. Although recommendation algorithms have significant advantages in the function of recommending items on shopping websites, the current recommendation system still has some shortcomings. These shortcomings mainly include data sparsity, cold start problems, and lack of real-time performance.

[0034] In order to solve the above technical problems, the following embodiments of a shopping webpage design platform and method based on big data collection are proposed in the present invention.

[0035] Example 1

[0036] Reference Figure 1 ,like Figure 1 The structure diagram of the shopping webpage design platform based on big data collection in this embodiment is shown. The platform specifically includes a data collection module, a data storage module, a data processing and analysis module, and an information generation module.

[0037] The data acquisition module includes:

[0038] Web data crawler: Use crawler technology to widely crawl web page data from the Internet, including text, pictures, and layout information.

[0039] User behavior collector: Through the tracking code embedded in the web page, the user's behavior data on the web page is collected in real time, including click, scroll, and dwell time information.

[0040] The data storage module includes:

[0041] Data warehouse: uses a distributed storage system to store captured web page data and user behavior data to ensure efficient access to data.

[0042] NoSQL databases: Used to store unstructured data, including text and images.

[0043] Data processing and analysis modules include:

[0044] Data cleaning and preprocessor: including data deduplication, data format conversion, and noise filtering functions.

[0045] Feature Extraction and Labeler: Extract key features of web design, including color matching, layout structure, font style information, and perform labeling. Labeling can structure the extracted features, making it easier for computers to understand and process. For example, representing features such as color matching, layout structure, and font style with specific labels can facilitate subsequent analysis and operations. Through labeling, data classification, clustering, and retrieval can be performed quickly. Labeled data can be used to train machine learning models to improve the accuracy and efficiency of the models.

[0046] Behavior Analyzer: Analyze user preferences and needs based on user behavior data to form a user profile.

[0047] The information generation module includes:

[0048] Design rule library: used to guide the generation of web design information.

[0049] Design template generator: Combines data analysis results and design rules to automatically generate a variety of web design templates.

[0050] Personalized customization engine: recommend or generate personalized web design solutions based on user portraits and real-time needs;

[0051] The display and feedback module includes:

[0052] Live Previewer: Users can preview the generated web design in real time and make fine adjustments.

[0053] User Feedback Collector: Further optimize design rules and generate models through user feedback.

[0054] As an implementation mode, in the data acquisition module of this embodiment:

[0055] Web data scrapers include:

[0056] Crawler scheduling unit: controls the operation of the crawler and starts crawling tasks regularly.

[0057] Web page parsing unit: parses web page content and extracts required text, images, and layout data.

[0058] Anti-crawler strategy response unit: includes random UA, IP rotation, and simulated click technology to circumvent the website's anti-crawler mechanism.

[0059] User behavior collectors include:

[0060] Tracking script unit: embedded in the web page to collect user click, scroll, and dwell time behavior data.

[0061] Event processor unit: processes user behavior data in real time, records and sends it to the backend server. As an implementation method, in the data storage module of this embodiment:

[0062] The data warehouse includes:

[0063] Distributed file unit: HDFS, used to store large-scale web crawling data.

[0064] Data management tool unit: Apache Hive, used for data query and management.

[0065] NoSQL databases include:

[0066] Database unit: MongoDB, which stores unstructured data and supports high concurrent reading and writing. Index mechanism unit: Improves data retrieval speed.

[0067] As an implementation mode, in the data processing and analysis module of this embodiment:

[0068] Data cleaning and preprocessing removes duplicate data through deduplication algorithms, including:

[0069] Format conversion unit: Apache Spark, used for data format conversion and batch processing; feature extraction and labeling units include:

[0070] Natural Language Processing Unit: Extract text features for NLTK.

[0071] Image processing tool: OpenCV, extract image features.

[0072] Labeling system unit: Labels design features based on machine learning algorithms.

[0073] Behavior Analyzer includes:

[0074] User behavior modeling: Use statistical analysis and machine learning methods to analyze user behavior data. User portrait generation: Combine multi-dimensional data to generate user preference and demand portraits.

[0075] As an implementation mode, in the information generation module of this embodiment:

[0076] The design rule library includes:

[0077] Rule definition language unit: Use DSL to describe design rules.

[0078] Rule Management System Unit: Maintain and update design rules.

[0079] The design template generator includes:

[0080] Template engine single: such as Jinja2, generates HTML and CSS templates according to design rules.

[0081] Style library: contains commonly used CSS frameworks and style components.

[0082] Personalized customization engine:

[0083] Recommendation algorithm unit: Generates personalized design solutions based on collaborative algorithms.

[0084] Real-time customization: Adjust and generate designs in real time based on user feedback.

[0085] As an implementation mode, the recommendation algorithm unit in this embodiment includes a collaborative filtering algorithm, a content recommendation algorithm, and a hybrid recommendation algorithm.

[0086] The collaborative filtering algorithms include user-based collaborative filtering algorithms and item-based collaborative filtering algorithms.

[0087] Specifically, the user-based collaborative filtering algorithm is as follows:

[0088] User similarity calculation usually uses cosine similarity:

[0089]

[0090] Among them, u and v are users, r ui is the rating of item i by user u, I uv is the set of items rated by both users u and v, I u is the set of items rated by user u.

[0091] Predict user u’s rating for item i:

[0092]

[0093] in, is the average rating of user u, and N(u) is the set of users most similar to user u.

[0094] Item-based collaborative filtering is as follows:

[0095] Item similarity calculation:

[0096]

[0097] Among them, i and j are items.

[0098] r ui is the rating of item i by user u, U ij is the set of users who have rated items i and j, U i is the set of users who have rated item i.

[0099] Predict user u’s rating for item i:

[0100]

[0101] Among them, N(i) is the set of items most similar to item i.

[0102] The content recommendation algorithm is as follows:

[0103] Content similarity calculation:

[0104]

[0105] Among them, w ik is the weight of item i on feature k.

[0106] Predict user u’s rating for item i:

[0107]

[0108] where p uk is the preference weight of user u on feature k.

[0109] The hybrid recommendation algorithm combines the above methods and uses weighted average to make predictions:

[0110]

[0111] Among them, α, β, γ are weight coefficients, satisfying α+β+γ=1.

[0112] For example: Assume the following data:

[0113] User 1 rated items A, B, and C as 4, 5, and 3 respectively.

[0114] User 2 rated items A, B, and C as 5, 4, and 4 respectively.

[0115] User 3 rated items A, B, and D as 3, 5, and 4 respectively.

[0116] User 1's content preference weights are 0.8 (color) and 0.6 (layout).

[0117] The feature weights of items A, B, and C are 0.9 (color), 0.7 (color), 0.6 (color); 0.5 (layout), 0.6 (layout), and 0.8 (layout).

[0118] We need to predict the rating of user 1 for item D.

[0119] 1. User-based collaborative filtering prediction

[0120] Calculate the similarity between user 1 and user 2:

[0121]

[0122] Prediction score:

[0123]

[0124] 2. Item-based collaborative filtering prediction

[0125] Calculate the similarity between items A and D:

[0126]

[0127] Prediction score:

[0128]

[0129] 3. Content recommendation algorithm prediction

[0130] Calculate the similarity between item D and user 1:

[0131]

[0132] 4. Hybrid recommendation prediction

[0133] Assume α = 0.4, β = 0.4, γ = 0.2;

[0134]

[0135] Through the hybrid recommendation algorithm, we combine the advantages of collaborative filtering, item collaborative filtering and content recommendation to generate a personalized rating prediction for item D from user 1. This method can comprehensively consider the characteristics of different recommendation technologies and provide more accurate and diverse recommendation results, which are convenient for display in the display and feedback module.

[0136] As an implementation method, in this embodiment, the shopping webpage design platform based on big data collection also includes a display and feedback module. In the display and feedback module:

[0137] The Live Previewer includes:

[0138] Preview interface unit: instantly displays the generated web page design and supports user interaction.

[0139] Visual editing units: such as Froala Editor, which allows users to make drag-and-drop adjustments.

[0140] User feedback collectors include:

[0141] Feedback system unit: collect users’ evaluation and suggestions on design solutions.

[0142] Data analysis tool unit: Analyze user feedback and optimize the design generation process.

[0143] Example 2

[0144] This embodiment provides a shopping webpage design method based on big data collection. Figure 2 ,like Figure 2 The figure is a flowchart of a shopping webpage design method based on big data collection in this embodiment, and the method includes the following steps:

[0145] Step 1, Data Collection

[0146] Start a web page data crawler to crawl web page data on the Internet regularly or in real time;

[0147] Deploy user behavior collectors to collect user interaction behaviors on different web pages in real time;

[0148] Step 2: Data storage and processing

[0149] Store the captured web page data and user behavior data in data warehouses and NoSQL databases;

[0150] Perform data cleaning and preprocessing to remove redundant and noisy data;

[0151] Step 3: Feature extraction and analysis

[0152] Extract design features from preprocessed web page data, including color, layout, and font information;

[0153] Analyze user behavior data, extract user preferences and demand information, and generate user portraits;

[0154] Step 4: Design information generation

[0155] Based on the design rule library, combined with feature extraction and behavior analysis results, web page design templates are automatically generated;

[0156] Use a personalized customization engine to generate personalized design solutions based on user portraits and needs;

[0157] Step 5: Display and feedback

[0158] Provides a real-time preview feature so users can view and fine-tune the generated web design;

[0159] Collect user feedback, optimize design rules and generate models, and continuously improve design quality.

[0160] As an implementation method, the user behavior data in this embodiment includes clicks, scrolling, dwell time, etc.

[0161] As an implementation method, the user preference data in this embodiment includes browsing history, search history, purchase history, etc.

[0162] As an implementation method, the web page design data of this embodiment includes web page layout, color matching, font selection, pictures, etc.

[0163] As an implementation method, data preprocessing in this embodiment includes removing noise and outliers, standardizing different features, ensuring data consistency, and extracting key features such as color, layout, font, etc. from web page design and user behavior.

[0164] As an implementation method, the user portrait generation in this embodiment includes:

[0165] Basic information: basic attributes of the user, such as age, gender, region, etc.

[0166] Behavioral characteristics: user browsing habits, click preferences, length of stay, etc.

[0167] Interest preferences: the user’s favorite content types, style preferences, etc.

[0168] Use clustering algorithms (such as K-means, DBSCAN) to cluster users and generate different user group portraits.

[0169] As an implementation method, the personalized recommendation algorithm of this embodiment includes:

[0170] ① User-based collaborative filtering, process:

[0171] Calculate the similarity between the target user and other users.

[0172] Find out the preferences of similar user groups.

[0173] Recommend design solutions based on the preferences of similar users.

[0174] ② Item-based collaborative filtering, process:

[0175] Calculate the similarity between the current web page design and other web page designs.

[0176] Find designs that are similar to your current design.

[0177] Optimize based on user feedback of similar designs.

[0178] ③Content recommendation algorithm, process:

[0179] Analyze the content of web pages viewed by users.

[0180] Extract content features (such as color, layout, font, etc.).

[0181] Recommend design solutions that are similar to the user’s historical browsing content.

[0182] ④ Hybrid recommendation algorithm, process:

[0183] Combining collaborative filtering and content recommendation, comprehensively considering the similarities between users and items, and providing more accurate and diverse recommendation results.

[0184] As an implementation method, the generation of a personalized design solution in this embodiment includes:

[0185] Rule base application:

[0186] The design features generated by the recommendation algorithm are combined with the design rule base to generate a design solution that meets user preferences.

[0187] The design rule library contains industry standards, design principles, etc. to ensure the quality of the generated solutions.

[0188] Solution optimization:

[0189] Feedback collection: Collect user feedback on the recommended design solutions.

[0190] Algorithm adjustment: Based on user feedback, adjust the weights and parameters of the recommendation algorithm to optimize the recommendation results.

[0191] Iterative improvement: Improve the accuracy and satisfaction of personalized recommendations through continuous iteration.

[0192] As an implementation mode, the real-time preview and user feedback in this embodiment specifically include:

[0193] Live Preview:

[0194] Users can preview recommended design solutions in real time on the platform.

[0195] Provides interactive tools that allow users to fine-tune design details.

[0196] User feedback:

[0197] Collect users' comments and modification suggestions on design solutions.

[0198] Analyze user feedback to further optimize recommendation algorithms and design rule bases.

[0199] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A shopping webpage design platform based on big data collection, characterized in that: The platform includes: A data collection module, which is used to obtain web page data and user behavior data; A data storage module, the data storage module is used to store the web page data and the user behavior data; A data processing and analysis module, which is used to extract web design features from web page data, extract user preferences and needs from user behavior data, and output data analysis results to the information generation module; The information generation module stores design rules described in a domain-specific language, and automatically generates a shopping web page template according to a preset recommendation algorithm in combination with data analysis results and the design rules.

2. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The recommendation algorithm includes a hybrid recommendation algorithm, which includes using a weighted average method to calculate the average score of user-based collaborative filtering predictions, item-based collaborative filtering predictions and content recommendation algorithm predictions, with weight coefficients pre-set and the sum of the weight coefficients being 1.

3. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The design platform also includes a display and feedback module for real-time preview and adjustment of the generated web page design, and optimization of the design rules based on user feedback.

4. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The web page data includes text, pictures and layout data, and the user behavior data includes user click, scroll and dwell time data.

5. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The data processing and analysis module also includes using statistical analysis and machine learning methods to analyze user behavior data and generate user preference and demand portraits. The information generation module generates personalized design solutions based on user preferences and demand portraits.

6. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The data storage module includes a distributed file storage unit and an unstructured database; Among them, the distributed file storage unit is used to store web page data and user behavior data, and the unstructured database is used to store unstructured data, which includes text and images.

7. The shopping webpage design platform based on big data collection as claimed in claim 1, characterized in that: The data processing and analysis module also includes a data cleaning and preprocessing unit for data deduplication, format conversion and noise filtering.

8. A shopping webpage design method based on big data collection, characterized in that: The method is implemented based on the webpage design platform described in any one of claims 1 to 7, and the method comprises: Start the data collection module to crawl web page data on the Internet and collect user behavior data on different web pages; Storing the acquired web page data and user behavior data in a data storage module; The data processing and analysis module extracts web design features from web page data and analyzes user behavior data to extract user preferences and needs; The information generation module is based on design rules and combines the feature extraction results and behavior analysis results output by the data processing and analysis module to generate a web page design template according to a preset hybrid recommendation algorithm.

Images