An e-commerce data strategy system

Abstract

The application relates to the technical field of big data analysis, and discloses an e-commerce data strategy system, data acquisition tools: Python, UIBut and shadow knives, wherein Python is a full code, and the tool is developed from 0; UIBut and shadow knives are graphical process development tools; step 1, a part of seed URLs selected carefully is selected in mainstream e-commerce; data of each e-commerce platform is acquired through a module developed by using a UiBot tool and Python; through an Ali cloud platform, a special database is built in cooperation with MySQL, and model data is classified, stored and deeply combed; a multi-dimensional data analysis model of the system is realized by using BI software, multi-dimensional queries of different posts, dates and platforms are met, an architecture is designed by oneself, a closed loop of data collection, arrangement and analysis is formed, visual display is realized, and enterprise digital decision-making is promoted.

Description

An e-commerce data strategy system Technical Field

[0001] This invention relates to the field of big data analytics, specifically to an e-commerce data strategy system. Background Technology

[0002] In the internet age, a user-centric approach is essential; only by fully understanding users can products be created that meet their expectations. E-commerce platforms use data to monitor and improve. By analyzing users, they can see where users come from, how to organize products to achieve good conversion rates, the efficiency of advertising, and so on. E-commerce operations require analyzing a wide range of data, including external data such as market share in the industry; page views (PV) reflecting website traffic; and metrics reflecting e-commerce website sales such as traffic conversion rate, order conversion rate, and average order value. Every small change based on data analysis translates into a gradual improvement in profitability.

[0003] However, many newly established or small operating companies are unable to effectively utilize mature and effective data analysis models and tools.

[0004] Therefore, this research aims to build an enterprise platform data analysis system to address the challenges encountered in enterprise digital transformation, such as complex and unintuitive data, high professional requirements for personnel, and incomplete data analysis. This research integrates the advantages of various platforms, establishing a novel architecture platform. It utilizes big data analytics tools to access multiple data sources, builds multi-dimensional data models, and performs data processing and analysis on massive datasets. The results are then visualized, ultimately achieving integrated data processing and driving the upgrade of enterprise digital decision-making. Summary of the Invention

[0005] The purpose of this invention is to provide an e-commerce data strategy system that enables integrated data processing and promotes the upgrading of enterprise digital decision-making.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An e-commerce data strategy system includes the following steps:

[0008] Data acquisition tools: Python, UIBut, and Shadow Knife. Python is a fully coded tool developed from scratch; UIBut and Shadow Knife are graphical workflow development tools.

[0009] import …class inSYCM(object): # Set the login page as SYCM url_sycm='https: / / sycm.taobao.com / custom / login.htm' # Webpage address for each business section url_qn_kh='https: / / voc.taobao.com / home.htm# / newTask / tmall-beacon-shop' url_qn_ztc='https: / / subway.simba.taobao.com / index.jsp' url_qn_ylmf='https: / / tuijian.taobao.com / indexbp.html' url_qn_cjtj='https: / / tuijian.taobao.com / indexbp-feedflow.html' # url_qn_zz='https: / / zuanshi.taobao.com / indexbp.html' url_qn_zz='https: / / zuanshi.taobao.com / ' url_pxb='https: / / branding.taobao.com' #Solidate account password user_pw={…} web=None page_log=None def __init__(self):…

[0010] After locating the login iframe, find the username / password input field. The code snippet `def login_iframe(self,url,user,pw):... # Enter username / password to log in` is followed by `def login_UserPw(self,user,pw):... # Specific account login`, `def logSYCM_RS(self):... # Specific account login`, `def logSYCM_ASKO(self):... # Specific page login`, `def logQN_RS(self):... # Specific page login`, `def logQN_CT(self):... # Specific page login`, `def logQN_KH(self):... # Specific page login`, `def logQN_ztc(self):... # Specific page login`, `def logQN_ylmf(self):... # Specific page login`, `def logQN_cjtj(self):... # Specific page login`, `def logQN_zz(self):... # Specific page login`, and `def logPXB_RS(self):... # Specific page login`. logTM(self, url='https: / / www.taobao.com / '):... # Close the page, close the browser def close(self):...if __name__=='__main__':...

[0011] E-commerce platforms: including but not limited to: multiple e-commerce platforms;

[0012] Step 1: First, select a group of carefully chosen seed URLs from multiple e-commerce platforms and arrange the selected seed URLs in order.

[0013] Step 2: Add these URLs to the URL queue to be crawled.

[0014] Step 3: Retrieve the URLs to be crawled from the URL queue, resolve the DNS, obtain the host's IP address, download the web pages corresponding to the URLs, and store them in the downloaded web page library. In addition, add these URLs to the crawled URL queue.

[0015] Step 4: Analyze the URLs in the crawled URL queue, analyze other URLs in it, and put the URLs into the URL queue to be crawled, thus entering the next loop.

[0016] In principle, the process of a web crawler scraping data involves three steps:

[0017] Download

[0018] Extract

[0019] Save = save

[0020] Specific steps for data crawling:

[0021] import …class DLfiles(object): # Main directory for data retrieval path_root=r'D:\

[00]

Metadata

Backend Data

'+str_ymonth+'

'+str_ydate+'

[0022] Download 'Shop 1 - Market - Market Ranking' def dl_scph2(self,datelist,file_date):… # Log in to 'Shop 2' homepage def dl_qn(self): … # Download customer service data from 'Shop 2 - Chitu' def dl_qn_ct(self): … # Download customer service data from 'Shop 2 - Customers' def dl_qn_kh(self): … # Download data from 'Shop 2 - [Direct Train]' def dl_qn_ztc(self): … # Download data from 'Shop 2 - Diamond Display' def dl_qn_zz(self): … # Download data from 'Shop 2 - Super Recommendation' def dl_qn_cjtj(self):… # Download data from 'Shop 2 - Gravity Cube' def dl_qn_ylmf(self): … # Download data from 'Pinxiaobao' def dl_pxb(self): … # File download and transfer commands def `movefile(self,date_start=yesterday,date_end=yesterday): ... # File upload` `UP_sql(self): ... # Used to directly read the directory and upload files after manually downloading / organizing them` `read_folder(self,path): ... # Read and process customer service data` `read_xls_kf(self,filepath): ... # Read and process 'Transaction Analysis - Category Composition' data` `read_xls_jygc(self,filepath): ... # Read and process 'Products - All Terminals' data` `read_xls_cp(self,filepath): ...`

[0023] Step 4.1: Obtain the webpage link

[0024] Taking Python as an example, using Python scripts and mounting the Selenium toolkit, code scripts can be written to access e-commerce platforms using the Chrome browser, store multiple webpage links obtained into a dictionary to act as a temporary database, and retrieve them directly through function calls when needed; the scripts can also simulate manual operations to locate, click, and fill in content for specific target elements on the page.

[0025] The system identifies and extracts valid information from the page by reading the HTML content.

[0026] Step 4.2, Data Storage

[0027] The web pages crawled by the web crawler are stored in the original page database. The page data and the HTML obtained by the user's browser are used to identify and extract valid data. When crawling the page, the engine will perform duplicate content detection. The data can be stored in a local database, a temporary mobile database, or a TXT or CSV file. In short, there are many different formats. We mainly use the JSON data format.

[0028] Step 4.3: Preprocessing (Data Cleaning)

[0029] Remove unwanted numbers or symbols from the data.

[0030] And use software to visualize the model data.

[0031] The script can be used to merge and organize the data, and it can be saved as a local archive as needed.

[0032] Step 4.4, Data Upload

[0033] After a batch of data is completed, the data can be uploaded to the database in a pre-set format for long-term storage and data analysis.

[0034] Step 5: Database Management

[0035] All crawled data is stored in an Alibaba Cloud database, and multiple security measures, including account management and database backup, are employed to ensure the security of the source data.

[0036] def dl_qn_zz(self): … # Download data from 'Shop 2 - Super Recommendation' def dl_qn_cjtj(self): … # Download data from 'Shop 2 - Gravity Cube' def dl_qn_ylmf(self): … # Download data from 'Product Sales Treasure' def dl_pxb(self): … # File download and transfer commands def movefile(self,date_start=yesterday,date_end=yesterday): … # File upload def UP_sql(self): … # Used to directly read the directory for file upload after manually downloading / organizing files def read_folder(self,path):… # Read and process data from

Customer Service

[0037] Read and process 'Drill Display' data: def read_xls_zz(self,filepath): … # Read and process 'Gravity Cube' data: def read_xls_almm(self,filepath): … # Read data that does not require special processing: def read_xls(self,filepath,sheetname): … # Convert the amount in the 'Direct Train' table: Yuan -> Fen: def ztc_100(self,filepath): … # Check if the date is unique and is yesterday: def check_date(self,date_list): … # Sum and merge for missing or duplicate brands read from downloaded files: def brands_sum(self,df,col_sum,col_check,brand2): … # Fill with 0 for general purposes, multiple tables need to use… def fill_null(self,df,cols): … # Convert 'Yuan -> Fen': def y2f(self,df,cols2): … def close(self): …if __name__=='__main__': start = time.time() dl = DLfiles() # dl.read_folder(r'D:\

[00] [Metadata]\[Backend Data]\[2022-06]\[2022-06-20]') dl.movefile() dl.UP_sql() end = time.time() print(end - start)

[0038] Step 6: Data ETL Preparation

[0039] For each project's data table, ETL data processing is performed according to business logic. Using SQL, queries, views, and stored events are written to optimize effective data for the visualization data analysis system, thus reducing some of the data processing pressure on the database server.

[0040] Step 7: Data Loading

[0041] In a visual data analysis tool, the corresponding data table in the database is read, and the valid data is loaded according to business requirements.

[0042] Step 8: Build the data model

[0043] For the loaded data tables, logical relationships are established between them according to business logic and data analysis requirements to form a large logical table. All related records can be retrieved by searching specific field information.

[0044] import …class MonTM(object): today=datetime.date.today() str_date=datetime.date.strftime(today,'%Y-%m-%d') str_month=datetime.date.strftime(today,'%Y-%m') path_root=r'D:\

[00]

Metadata

Single Product Page Data

[0045] import …# Select page sections and tags, compatible with 1 / 2 level jumps def selTarget(driver,level1,l1_num,level2=None,l2_num=0):# Set basic options on the page, including sections & tags, shops, and dates def setBasic(driver,target,shop,date_range,day_num=3):# Set the date on the page def setDate(driver,element,date):# Set drop-down menu options def setOption(driver,task,option,num,options):# Set single-select items, applicable to 'Diamond Display', 'Super Recommendation' (old version), and 'Direct Train' def selSingle1(driver,task):# Set single-select items def selSingle2(driver,task):# Enter category / brand information in the input box def setInStr(driver,task,InStr1,InStr2=None):# The operation part of entering category / brand information in the input box is a separate function for easy reuse def setInStr_input(driver, task, InStr, el_inputs): # Check if pages 2 through num exist. def nextpage(driver, num): # Manage page element location for each task when fetching data. def getEl(task, InStr1, brand1_full):

[0046] Step 9: Generate a single data report

[0047] Depending on business needs, different presentation methods and different data information can be selected.

[0048] # Based on the read multi-category brand list df, select the corresponding category's brand list def getBrandlist(df,category): # Fixed 'data grabbing task - category - brand list' def readBrandlist(task,category): # Fixed predefined 'brand abbreviation: brand full name' dictionary def Brand_dict(brand): # Fixed core brands for each category (ranked by top 15 in 2021 sales), used to perform a second check for 'no data' defCheckData_dict(task,category): # Check page ads and close them def ad_close(driver): # Check if "insufficient balance, account recharge required" def recharge(driver): # Check: whether there is a new / old version, use the old version def return_old(driver,el_old): import …

[0049] # Copy single-page data def scrap(driver, task, cols, isfirst): # If the scraped data is missing or duplicated, a flag needs to be added when re-scraping def re_scrap(nodata_list, nofind_list, brands_list, brand): # Copy 3 pages of data def scrap_brand(driver, task, cols): # Copy data for all date ranges def data_all_dates(web_opts, filepath, datelist, categories, brands_opts): # Copy data for all categories for the current date def data_all_categorys(driver, categories, brands_opts): # Copy data for all brands in the current category def data_all_brands(driver, category, brands_opts): # Scrape data for a single product page def data_product(driver, url, ID, model, brand, attr, info_product, tmp_str): # Activate the one-click conversion interface of the Amin tool and copy the data def am1k(driver,task,InStrs,title,cols,num=1): # Perform deduplication / merging of 'brand name' values ​​on the copied 'Market Ranking' data. def drop_dup(df,cols_dup,cols_sum): # Convert string percentages to decimals and replace any ',' characters. def f2i(df,cols):

[0050] Step 10: Link multiple data reports together

[0051] Based on business needs, individual data reports with different data sources and different data information can be linked and linked through the same data field. When filtering data in any of the data reports, the filtering results can be synchronized to the data reports, so that all reports with the same related fields can be filtered and presented at the same time.

[0052] import …class SQLData(object): query=SQLQuery.SQLQuery() db_info=[…] def __init__(self): # General data reading of tables / views / stored procedures, etc., and save the query results to csv def db2csv(self,TVFlist): # Read all tables in the database def getTdata(self): # Read all views in the database def getVdata(self): # Read all stored procedures in the database def getFdata(self): # To ensure that the date column is in the correct format, one or more columns need to be converted to date format def str2date(self,df_name,*cols): # Delete data by time range for date / time columns def delrows_date(self,db,table,col,date_start,date_end): # Upload data def upTdata(self,db,TVF,data,cols_f): if __name__=='__main__': start=time.time() d=SQLData() d.getTdata() # d.getVdata() # d.getFdata() # d.upTdata() end=time.time()

[0053] import pymysql, pandas class SQLQuery(object): login inf={…} def __init__(self): # Connection detection part, using private methods def __GetConn(self,db='…'): # Database query def QueryDBslist(self): # Database list query def QueryTableslist(self): # View list query def QueryViewslist(self): # Function list query def QueryFuncslist(self): # Read data from specified tables and views in specified databases def QueryTVFdata(self,db,TVF): # Command execution def RunSQL(self,db,sql): self.__GetConn(db) cursor=self.conn.cursor() result=cursor.execute(sql) self.conn.commit() self.close() return result

[0054] # Exit handling def close(self): if __name__=='__main__': # Test

[0055] helper=SQLQuery() # DBs_list=helper.QueryDBslist() # print(DBs_list)Tables_list=helper.QueryTableslist() # print(Tables_list) # Views_list=helper.QueryViewslist() # print(Views_list) # Funcs_list=helper.Queryfuncslist() # print(Funcs_list)

[0056] if __name__=='__main__': # Internal test command for this module

[0057] helper=SQLQuery() # DBs_list=helper.QueryDBslist() # print(DBs_list)Tables_list=helper.QueryTableslist() # print(Tables_list) # Views_list=helper.QueryViewslist() # print(Views_list) # Funcs_list=helper.Queryfuncslist() # print(Funcs_list)

[0058] Step 11, Web Publishing

[0059] After debugging, multi-page, multi-report systems can be published via the web, allowing authorized personnel to view them anytime, anywhere. The number of reports and the granularity of report summaries can vary depending on the user's permissions.

[0060] Preferably, the data is reorganized based on the business logic and data model of the data analysis in step 6.

[0061] Preferably, the presentation method according to step 9 includes, but is not limited to: bar chart, line chart, area chart, and table.

[0062] Preferably, in step 4.3, garbled characters, abnormal and inconsistent values ​​in the data are removed and cleaned, and an abnormal data record table is set up. The staff repeatedly reads the preprocessed data by setting a timed reading process, and stores the read abnormal data in the abnormal data record table.

[0063] Preferably, the selection of page elements in step 4.1 is set according to different business projects and manual operation logic, with each script corresponding to different projects and / or different operation processes.

[0064] Preferably, the intelligent data crawling method used in step 4 can significantly shorten the manual operation time. As long as the script is maintained regularly, there will be no data errors or damage. Furthermore, some problems with the data itself can be identified and alerted by the script.

[0065] Preferably, the presentation method in step 9 includes, but is not limited to: bar chart, line chart, area chart, and table, and the different data information presented includes, but is not limited to: date, type, product name, quantity, and amount.

[0066] Preferably, the dimensions of multi-table linkage in step 10 include, but are not limited to: date, category, product name, and channel name.

[0067] Preferably, the authorized personnel in step 11 include personnel's email information, account information, password information, and permission information. The granularity of the report summary includes, but is not limited to, manager level, supervisor level, and ordinary employee level.

[0068] This invention provides an e-commerce data strategy system, which has the following beneficial effects:

[0069] This invention acquires data from various e-commerce platforms using the UiBot tool and a custom-developed Python module; it utilizes the Alibaba Cloud platform and MySQL to build a dedicated database for the categorized storage and in-depth analysis of device model data; the system's multi-dimensional data analysis model is implemented using BI software to meet the multi-dimensional query requirements of different positions, dates, and platforms; and its self-designed architecture forms a closed loop for data collection, organization, and analysis, enabling visualized presentation and driving enterprise digital decision-making. Attached Figure Description

[0070] Figure 1 is a schematic diagram of the overall process breakdown of the present invention;

[0071] Figure 2 is a schematic diagram of the promotion platform process of the present invention;

[0072] Figure 3 is a schematic diagram of the marketing platform process of the present invention;

[0073] Figure 4 is a schematic diagram of the overall process of seed URL data collection according to the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] As shown in Figures 1-4, the present invention provides a technical solution:

[0076] An e-commerce data strategy system includes the following steps:

[0077] Data acquisition tools: Python, UIBut, and Shadow Knife. Python is a fully coded tool developed from scratch; UIBut and Shadow Knife are graphical workflow development tools.

[0078] import …class inSYCM(object): # Set the login page as SYCM url_sycm='https: / / sycm.taobao.com / custom / login.htm' # Webpage address for each business section url_qn_kh='https: / / voc.taobao.com / home.htm# / newTask / tmall-beacon-shop' url_qn_ztc='https: / / subway.simba.taobao.com / index.jsp' url_qn_ylmf='https: / / tuijian.taobao.com / indexbp.html' url_qn_cjtj='https: / / tuijian.taobao.com / indexbp-feedflow.html' # url_qn_zz='https: / / zuanshi.taobao.com / indexbp.html' url_qn_zz='https: / / zuanshi.taobao.com / ' url_pxb='https: / / branding.taobao.com' #Solidate account password user_pw={…} web=None page_log=None def __init__(self):…

[0079] After locating the login iframe, find the username / password input field. The code snippet `def login_iframe(self,url,user,pw):... # Enter username / password to log in` is followed by `def login_UserPw(self,user,pw):... # Specific account login`, `def logSYCM_RS(self):... # Specific account login`, `def logSYCM_ASKO(self):... # Specific page login`, `def logQN_RS(self):... # Specific page login`, `def logQN_CT(self):... # Specific page login`, `def logQN_KH(self):... # Specific page login`, `def logQN_ztc(self):... # Specific page login`, `def logQN_ylmf(self):... # Specific page login`, `def logQN_cjtj(self):... # Specific page login`, `def logQN_zz(self):... # Specific page login`, and `def logPXB_RS(self):... # Specific page login`. logTM(self, url='https: / / www.taobao.com / '):... # Close the page, close the browser def close(self):...if __name__=='__main__':...

[0080] E-commerce platforms: including but not limited to: multiple e-commerce platforms;

[0081] Step 1: First, select a group of carefully chosen seed URLs from multiple e-commerce platforms and arrange the selected seed URLs in order.

[0082] Step 2: Add these URLs to the URL queue to be crawled.

[0083] Step 3: Retrieve the URLs to be crawled from the URL queue, resolve the DNS, obtain the host's IP address, download the web pages corresponding to the URLs, and store them in the downloaded web page library. In addition, add these URLs to the crawled URL queue.

[0084] Step 4: Analyze the URLs in the crawled URL queue, analyze other URLs in it, and put the URLs into the URL queue to be crawled, thus entering the next loop.

[0085] In principle, the process of a web crawler scraping data involves three steps:

[0086] Download

[0087] Extract

[0088] Save = save

[0089] Specific steps for data crawling:

[0090] import …class DLfiles(object): # Main directory for data retrieval path_root=r'D:\

[00]

Metadata

Backend Data

'+str_ymonth+'

'+str_ydate+'

[0091] The system identifies and extracts valid information from the page by reading the HTML content.

[0092] Step 4.2, Data Storage

[0093] The web pages crawled by the web crawler are stored in the original page database. The page data and the HTML obtained by the user's browser are used to identify and extract valid data. For all extracted valid data, the engine will perform duplicate content detection when crawling the page. The data can be stored in a local database, a temporary mobile database, or a TXT or CSV file. We mainly use JSON data format.

[0094] Step 4.3: Preprocessing (Data Cleaning)

[0095] Remove unwanted numbers or symbols from the data.

[0096] And use software to visualize the model data.

[0097] The script can be used to merge and organize the data, and it can be saved as a local archive as needed.

[0098] Step 4.4, Data Upload

[0099] After a batch of data is completed, the data can be uploaded to the database in a pre-set format for long-term storage and data analysis.

[0100] Step 5: Database Management

[0101] All crawled data is stored in an Alibaba Cloud database, and multiple security measures, including account management and database backup, are employed to ensure the security of the source data.

[0102] def dl_qn_zz(self): … # Download data from 'Shop 2 - Super Recommendation' def dl_qn_cjtj(self): … # Download data from 'Shop 2 - Gravity Cube' def dl_qn_ylmf(self): … # Download data from 'Product Sales Treasure' def dl_pxb(self): … # File download and transfer commands def movefile(self,date_start=yesterday,date_end=yesterday): … # File upload def UP_sql(self): … # Used to directly read the directory for file upload after manually downloading / organizing files def read_folder(self,path):… # Read and process data from

Customer Service

[00]

Metadata

Backend Data

[0103] Step 6: Data ETL Preparation

[0104] For each project's data table, ETL data processing is performed according to business logic. Using SQL, queries, views, and stored events are written to optimize effective data for the visualization data analysis system, thus reducing some of the data processing pressure on the database server.

[0105] Step 7: Data Loading

[0106] In a visual data analysis tool, the corresponding data table in the database is read, and the valid data is loaded according to business requirements.

[0107] Step 8: Build the data model

[0108] For multiple loaded data tables, logical relationships are established between them based on business logic and data analysis requirements to form a single logical large table. All related records can be retrieved by searching specific field information.

[0109] import …class MonTM(object): today=datetime.date.today() str_date=datetime.date.strftime(today,'%Y-%m-%d') str_month=datetime.date.strftime(today,'%Y-%m') path_root=r'D:\

[00]

Metadata

Single Product Page Data

[0110] import …# Select page sections and tags, compatible with 1 / 2 level jumps def selTarget(driver,level1,l1_num,level2=None,l2_num=0):# Set basic options on the page, including sections & tags, shops, and dates def setBasic(driver,target,shop,date_range,day_num=3):# Set the date on the page def setDate(driver,element,date):# Set drop-down menu options def setOption(driver,task,option,num,options):# Set single-select items, applicable to 'Diamond Display', 'Super Recommendation' (old version), and 'Direct Train' def selSingle1(driver,task):# Set single-select items def selSingle2(driver,task):# Enter category / brand information in the input box def setInStr(driver,task,InStr1,InStr2=None):# The operation part of entering category / brand information in the input box is a separate function for easy reuse def setInStr_input(driver, task, InStr, el_inputs): # Check if pages 2 through num exist. def nextpage(driver, num): # Manage page element location for each task when fetching data. def getEl(task, InStr1, brand1_full):

[0111] Step 9: Generate a single data report

[0112] Depending on business needs, different presentation methods and different data information can be selected.

[0113] # Based on the read multi-category brand list df, select the corresponding category's brand list def getBrandlist(df,category): # Fixed 'data grabbing task - category - brand list' def readBrandlist(task,category): # Fixed predefined 'brand abbreviation: brand full name' dictionary def Brand_dict(brand): # Fixed core brands for each category (ranked by top 15 in 2021 sales), used to perform a second check for 'no data' defCheckData_dict(task,category): # Check page ads and close them def ad_close(driver): # Check if "insufficient balance, account recharge required" def recharge(driver): # Check: whether there is a new / old version, use the old version def return_old(driver,el_old): import …

[0114] # Copy single-page data def scrap(driver, task, cols, isfirst): # If the scraped data is missing or duplicated, a flag needs to be added when re-scraping def re_scrap(nodata_list, nofind_list, brands_list, brand): # Copy 3 pages of data def scrap_brand(driver, task, cols): # Copy data for all date ranges def data_all_dates(web_opts, filepath, datelist, categories, brands_opts): # Copy data for all categories for the current date def data_all_categorys(driver, categories, brands_opts): # Copy data for all brands in the current category def data_all_brands(driver, category, brands_opts): # Scrape data for a single product page def data_product(driver, url, ID, model, brand, attr, info_product, tmp_str): # Activate the one-click conversion interface of the Amin tool and copy the data def am1k(driver,task,InStrs,title,cols,num=1): # Perform deduplication / merging of 'brand name' values ​​on the copied 'Market Ranking' data. def drop_dup(df,cols_dup,cols_sum): # Convert string percentages to decimals and replace any ',' characters. def f2i(df,cols):

[0115] Step 10: Link multiple data reports together

[0116] Based on business needs, individual data reports with different data sources and different data information can be linked and linked through the same data field. When filtering data in any one of the data reports, the filtering results can be synchronized to multiple data reports, so that all reports with the same related field can filter data and present data at the same time.

[0117] import …class SQLData(object): query=SQLQuery.SQLQuery() db_info=[…] def __init__(self): # General data reading of tables / views / stored procedures, etc., and save the query results to csv def db2csv(self,TVFlist): # Read all tables in the database def getTdata(self): # Read all views in the database def getVdata(self): # Read all stored procedures in the database def getFdata(self): # To ensure that the date column is in the correct format, one or more columns need to be converted to date format def str2date(self,df_name,*cols): # Delete data by time range for date / time columns def delrows_date(self,db,table,col,date_start,date_end): # Upload data def upTdata(self,db,TVF,data,cols_f): if __name__=='__main__': start=time.time() d=SQLData() d.getTdata() # d.getVdata() # d.getFdata() # d.upTdata() end=time.time()

[0118] import pymysql, pandas class SQLQuery(object): login inf={…} def __init__(self): # Connection detection part, using private methods def __GetConn(self,db='…'): # Database query def QueryDBslist(self): # Database list query def QueryTableslist(self): # View list query def QueryViewslist(self): # Function list query def QueryFuncslist(self): # Read data from specified tables and views in specified databases def QueryTVFdata(self,db,TVF): # Command execution def RunSQL(self,db,sql): self.__GetConn(db) cursor=self.conn.cursor() result=cursor.execute(sql) self.conn.commit() self.close() return result

[0119] # Exit handling def close(self): if __name__=='__main__': # Test

[0120] helper=SQLQuery() # DBs_list=helper.QueryDBslist() # print(DBs_list)Tables_list=helper.QueryTableslist() # print(Tables_list) # Views_list=helper.QueryViewslist() # print(Views_list) # Funcs_list=helper.Queryfuncslist() # print(Funcs_list)

[0121] if __name__=='__main__': # Internal test command for this module

[0122] helper=SQLQuery() # DBs_list=helper.QueryDBslist() # print(DBs_list)Tables_list=helper.QueryTableslist() # print(Tables_list) # Views_list=helper.QueryViewslist() # print(Views_list) # Funcs_list=helper.Queryfuncslist() # print(Funcs_list)

[0123] Step 11, Web Publishing

[0124] After debugging, multi-page, multi-report systems can be published via the web, allowing authorized personnel to view them anytime, anywhere. The number of reports and the granularity of report summaries can vary depending on the user's permissions.

[0125] In summary, this invention acquires data from various e-commerce platforms using the UiBot tool and a custom-developed Python module; it utilizes the Alibaba Cloud platform and a dedicated database built with MySQL for the categorized storage and in-depth analysis of device model data; its multi-dimensional data analysis model, implemented using BI software, satisfies multi-dimensional queries based on different positions, dates, and platforms; its self-designed architecture forms a closed loop of data collection, organization, and analysis, enabling visualization and driving enterprise digital decision-making; it collects and organizes relevant data obtained after transactions through the marketing platform, and determines the overall trend of transactions based on transaction categories, product types, and traffic; the promotion platform can build multi-dimensional data in the form of reports, organize and analyze massive amounts of data, and visualize the data analysis results, ultimately achieving integrated data processing.

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

Claims

1. An e-commerce data strategy system, characterized in that, The process includes the following steps: Step 1: First, select a group of carefully chosen seed URLs from multiple e-commerce platforms and arrange them in order; Step 2: Add these URLs to the URL queue to be crawled; Step 3: Retrieve the URLs to be crawled from the URL queue, resolve the DNS, obtain the host IP, download the corresponding webpages, and store them in the downloaded webpage library. Additionally, add these URLs to the crawled URL queue; Step 4: Analyze the URLs in the crawled URL queue, analyze other URLs within them, and add these URLs to the URL queue to be crawled, thus entering the next loop. In principle, the process of a web crawler crawling data involves three steps: download = download, extract = extract, save = save. Specific data crawling steps: Step 4.1: Obtain webpage links using a Python script, mounting the Selenium toolkit, and writing code to achieve Chrome... The browser accesses the e-commerce platform, stores the obtained webpage links in a dictionary, acting as a temporary database, which can be retrieved directly through function calls when needed; a script is used to simulate manual operation, locating, clicking, and filling in content for specific target elements on the page; valid information on the page is identified and extracted by reading HTML content; Step 4.2, Data Storage: The webpages crawled by the web crawler are stored in the original page database. The page data in the database is compared with the HTML obtained by the user's browser for valid data identification and extraction. All extracted valid data is processed; the engine performs duplicate content detection when crawling pages. Data can be stored in a local database, a temporary mobile database, or a txt or csv file; Step 4.3, Preprocessing: Unnecessary numbers or symbols are removed from the data; and the data is visualized using software; the data is merged and organized using scripts, and can be saved locally as needed; Step 4.

4. Data Upload: After completing a batch of data, the data can be synchronously uploaded to the database according to a pre-defined format for long-term storage and data analysis.

5. Database Management: All crawled data is stored in an Alibaba Cloud database, employing account management, database backup, and other security measures to ensure source data security.

6. Data ETL Processing: For each project's data tables, ETL data processing is performed according to business logic. Using SQL, queries, views, and stored events are written to optimize effective data for the visual data analysis system, allowing the database server to share some of the data processing load.

7. Data Loading: The visual data analysis tool reads the corresponding data tables from the database and loads effective data according to business needs.

8. Data Model Building: For the loaded data tables, based on business logic and data... Based on the analysis requirements, logical relationships are established between various data tables to form a single logical table. All related records are retrieved by searching specific field information. Step 9: Generate individual data reports. Different presentation methods and data information are selected based on business needs. Step 10: Link data reports. Based on business needs, individual data reports with different data sources and different data information are linked together using the same data field. When filtering data in one data report, the filtering results can be synchronized to the other data report, allowing all reports with the same related fields to perform data filtering and presentation simultaneously. Step 11: Web publishing. After debugging, multi-page, multi-report systems are published via the web, allowing authorized personnel to view them anytime, anywhere. The number of reports and the granularity of report summaries displayed vary depending on the user's permissions.

2. The e-commerce data strategy system according to claim 1, characterized in that: Reorganize the data based on the business logic and data model of the data analysis in step 6.

3. The e-commerce data strategy system according to claim 1, characterized in that: The presentation methods of step 9 include, but are not limited to: bar charts, line charts, area charts, and tables.

4. The e-commerce data strategy system according to claim 1, characterized in that: In step 4.3, garbled characters, abnormal and inconsistent values ​​in the data are removed and cleaned, and an abnormal data record table is set up. Staff members repeatedly read the preprocessed data by setting up a timed reading process, and store the read abnormal data in the abnormal data record table.

5. The e-commerce data strategy system according to claim 1, characterized in that: The selection of page elements in step 4.1 is set according to different business projects and manual operation logic, and the scripts correspond to different projects and / or different operation processes.

6. The e-commerce data strategy system according to claim 1, characterized in that: The intelligent data crawling method used in step 4 can significantly shorten the manual operation time. As long as the script is maintained regularly, there will be no data errors or damage. Furthermore, the script can also identify and alert you to some problems with the data itself.

7. The e-commerce data strategy system according to claim 1, characterized in that: The presentation methods in step 9 include, but are not limited to: bar charts, line charts, area charts, and tables, and the presentation of different data information includes, but is not limited to: date, type, product name, quantity, and amount.

8. The e-commerce data strategy system according to claim 1, characterized in that: The dimensions involved in the multi-table linkage in step 10 include, but are not limited to: date, category, product name, and channel name.

9. The e-commerce data strategy system according to claim 1, characterized in that: The authorized personnel in step 11 include personnel's email information, account information, password information, and permission information. The granularity of the report summary includes, but is not limited to, manager level, supervisor level, and ordinary employee level.

Images