Consumer industry data set construction method and device and medium

Abstract

The invention discloses a consumer industry data set construction method and device and a medium. The method comprises the steps of obtaining multi-source data related to a consumer industry; the multi-source data comprises e-commerce platform public data, policy data, information data, public opinion data, macroeconomic data, industry data, consumption complaint data and consumption infrastructure data; establishing an association relationship among the data of the multi-source data on the basis of association elements; wherein the associated elements comprise a time dimension, a consumption subject and a consumption object; carrying out fusion processing on the multi-source data with the incidence relation; and based on the fused multi-source data, storing the fused multi-source data to a data warehouse by adopting a star model and / or a snowflake model. According to the method, an accurate consumer industry high-quality data set with clear data association is constructed.

Description

Technical Field

[0001] This application relates to the field of big data collection and analysis technology, and in particular to a method, device and medium for constructing consumer industry datasets. Background Technology

[0002] With the digital transformation of the consumer industry, data has become a key driver of innovation and development. In the consumer sector, this encompasses diverse and heterogeneous data sources, including publicly available data from e-commerce platforms, policy data, news data, public opinion data, publicly available macroeconomic data from the government, data from industry association websites, consumer complaint data, and consumer infrastructure data. This data is characterized by its massive volume, complexity, and dynamic changes.

[0003] Traditional data processing methods struggle to effectively integrate and utilize these multi-source heterogeneous data, resulting in datasets with incomplete data, inconsistent quality, and unclear data relationships. This fails to meet the high-quality data needs of the consumer industry for precise analysis, market forecasting, and consumer insights. Summary of the Invention

[0004] This application provides a method, device, and medium for constructing a consumer industry dataset, which addresses the problems of difficulty in effectively integrating multi-source heterogeneous data and low dataset quality in the consumer industry.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On the one hand, embodiments of this application provide a method for constructing a consumer industry dataset, the method comprising:

[0007] Acquire multi-source data related to the consumer industry; the multi-source data includes: publicly available data from e-commerce platforms, policy data, information data, public opinion data, macroeconomic data, industry data, consumer complaint data, and consumer infrastructure data;

[0008] Based on the correlation elements, the correlation relationships between the various data from the multi-source data are established; wherein, the correlation elements include: time dimension, consumer subject, and consumer object;

[0009] Fusion processing of multi-source data with correlation relationships;

[0010] Based on the fused multi-source data, a star schema and / or snowflake schema are used to store the fused multi-source data in a data warehouse.

[0011] In one example, acquiring multi-source data related to the consumer industry specifically includes:

[0012] Using distributed web crawler technology, crawling tasks are automatically triggered according to preset crawling rules and frequencies to obtain publicly available data from the e-commerce platform;

[0013] The automated scripts access designated sections of government department websites and policy release platforms at preset intervals to obtain the policy data.

[0014] A multi-source information aggregation and collection framework is adopted to connect with the public data interfaces or page resources of news information platforms and industry media websites, and to scan the information content of the public data interfaces or page resources of the news information platforms and industry media websites through keyword matching and semantic analysis algorithms to obtain the information data.

[0015] By using automated social media data collection tools, the platform's public data interfaces are called or page data is parsed to obtain the aforementioned public opinion information;

[0016] The macroeconomic data is obtained by accessing the government statistical database through an automated script that crawls the link.

[0017] Access the official website of the Consumer Industry Association through an automated web page parsing tool to automatically identify and download the industry data;

[0018] The consumer complaint data is obtained through standardized data interfaces connected to consumer associations and third-party complaint platforms;

[0019] The consumer infrastructure data is obtained through the geographic information open platform application interface and based on the automated passenger flow data collection tool.

[0020] In one example, prior to the fusion processing of multi-source data with correlation relationships, the method further includes:

[0021] By statistically analyzing the number and proportion of missing values ​​in each field of the multi-source data, the target field is determined, and the target field is supplemented or discarded.

[0022] Outliers in the multi-source data are identified using statistical methods and machine learning algorithms, and these outliers are corrected or deleted.

[0023] By calculating the hash value of each piece of data from the multi-source data or by using the deduplication function of the database, duplicate data records are identified, and the duplicate data records are deduplicated according to business requirements.

[0024] The numerical data from the multi-source data is standardized by unifying the units and value ranges, and the text data from the multi-source data is standardized by converting it into a machine-processable format.

[0025] In one example, the storage of the fused multi-source data in a data warehouse using a star schema and / or a snowflake schema specifically includes:

[0026] By using multi-dimensional analysis techniques, single-level dimension carrying capacity verification, coarse-grained fixed classification verification, and dimensionless inter-table transfer association verification are performed on the fused multi-source data to determine whether the fused multi-source data simultaneously meets the judgment conditions of the three dimensions: single-level dimension carrying capacity, coarse-grained fixed classification, and dimensionless inter-table transfer association.

[0027] If the conditions are met, the merged multi-source data will be stored in the data warehouse using the star schema.

[0028] If the conditions are not met, the merged multi-source data will be stored in the data warehouse using the snowflake model.

[0029] In one example, storing the merged multi-source data in a data warehouse using the star schema specifically includes:

[0030] Using the star schema, fields of consumption attribute data are extracted from the fused multi-source data according to data attributes to construct a fact table in the data warehouse. Fields of subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data are also extracted from the fused multi-source data to construct subject attribute dimension tables, object attribute dimension tables, comprehensive correlation dimension tables, and policy attribute dimension tables in the data warehouse, respectively.

[0031] The SQL statements are used to establish inter-table relationships between the fact table and the subject attribute dimension table, the object attribute dimension table, the comprehensive association dimension table and the policy attribute dimension table in the data warehouse. The foreign key fields of the fact table are associated with the primary key fields of the subject attribute dimension table, the object attribute dimension table, the comprehensive association dimension table and the policy attribute dimension table to form a star schema relationship structure.

[0032] The subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data from the fused multi-source data are loaded using an ETL tool and written into the subject attribute dimension table, the object attribute dimension table, the comprehensive correlation dimension table, and the policy attribute dimension table, respectively.

[0033] The consumption attribute data from the merged multi-source data is loaded using an ETL tool, and the consumption attribute data is written into the fact table.

[0034] In one example, storing the fused multi-source data in a data warehouse using the snowflake model specifically includes:

[0035] Using the snowflake model, the fused multi-source data is processed in layers according to data attributes;

[0036] By extracting core data fields of consumer infrastructure from the merged multi-source data, a facility master table is constructed in the data warehouse;

[0037] By extracting core data fields of industry associations, public opinion, and consumer complaints from the merged multi-source data, an industry master table, a public opinion master table, and a complaint master table are constructed in the data warehouse. By extracting core fields of consumer attribute data from the merged multi-source data, a consumer fact table is constructed in the data warehouse.

[0038] Fields from consumer infrastructure ancillary data are extracted from the merged multi-source data to construct facility location sub-tables and facility operation sub-tables in the data warehouse; fields from industry association ancillary data are extracted from the merged multi-source data to construct industry category sub-tables in the data warehouse; fields from public opinion ancillary data are extracted from the merged multi-source data to construct public opinion sentiment sub-tables in the data warehouse; fields from consumer complaint ancillary data are extracted from the merged multi-source data to construct complaint issue sub-tables in the data warehouse.

[0039] Using SQL statements, establish inter-table relationships between the consumption fact table and the facility master table, industry master table, public opinion master table, and complaint master table in the data warehouse, respectively, so as to associate the foreign key fields of the consumption fact table with the primary key fields of the facility master table, industry master table, public opinion master table, and complaint master table;

[0040] Using SQL statements, establish the following master tables in the data warehouse: facility master table, industry master table, public opinion master table, and complaint master table. These tables are then linked to the following sub-tables: facility location sub-table, facility operation sub-table, industry category sub-table, public opinion sentiment sub-table, and complaint issue sub-table. Foreign key fields in the facility location sub-table and facility operation sub-table are linked to the primary key fields in the facility master table; foreign key fields in the industry category sub-table are linked to the primary key fields in the industry master table; foreign key fields in the public opinion sentiment sub-table are linked to the primary key fields in the public opinion master table; and foreign key fields in the complaint issue sub-table are linked to the primary key fields in the complaint master table, thus forming a snowflake model association structure.

[0041] The consumer attribute data, core data of consumer infrastructure, core data of industry associations, core data of public opinion, and core data of consumer complaints from the merged multi-source data are loaded using an ETL tool and written into the consumer fact table, the facility master table, the industry master table, the public opinion master table, and the complaint master table, respectively.

[0042] An ETL tool is used to load the consumer infrastructure ancillary data, industry association ancillary data, public opinion ancillary data, and consumer complaint ancillary data from the merged multi-source data, and write them into the construction facility location sub-table, the facility operation sub-table, the industry category sub-table, the public opinion sentiment sub-table, and the complaint issue sub-table, respectively.

[0043] In one example, after storing the merged multi-source data in a data warehouse, the method further includes:

[0044] The quality of multi-source data in the data warehouse is assessed based on a pre-defined multi-dimensional indicator evaluation system.

[0045] The data in the data warehouse is evaluated and corrected based on the evaluation results.

[0046] In one example, the step of evaluating the quality of multi-source data in the data warehouse according to a preset multi-dimensional indicator evaluation system, and then evaluating and correcting the data in the data warehouse based on the evaluation results, specifically includes:

[0047] Obtain the data to be assessed from the data warehouse;

[0048] The data to be evaluated for quality is subjected to multi-dimensional indicator evaluation calculations to obtain multi-dimensional indicator evaluation results; wherein, the multi-dimensional indicators include: accuracy indicators, completeness indicators, consistency indicators, timeliness indicators and relevance indicators;

[0049] The evaluation results of the multi-dimensional indicators are compared with the preset thresholds in the preset multi-dimensional indicator evaluation system to determine whether the evaluation results of the multi-dimensional indicators meet the standards.

[0050] If not, the substandard data will be corrected and subjected to structured evaluation.

[0051] On the other hand, embodiments of this application provide a consumer industry dataset construction device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a consumer industry dataset construction method as described above.

[0052] On the other hand, embodiments of this application provide a non-volatile computer storage medium for constructing consumer industry datasets, storing computer-executable instructions that can execute any of the above-described methods for constructing consumer industry datasets.

[0053] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0054] By acquiring multi-source data covering various types, including publicly available data from e-commerce platforms, policy data, news data, public opinion data, macroeconomic data, industry data, consumer complaint data, and consumer infrastructure data, the data can reflect the full picture of the consumer industry, solve the problem of incomplete data in existing datasets, and provide rich information for subsequent analysis.

[0055] Establishing data relationships enhances the logical coherence between data points, resolving the issue of unclear data relationships in existing datasets and making previously scattered data more organized. Multi-source data fusion processing removes redundant and contradictory information, improving data accuracy and completeness. Star and snowflake schemas provide structured data organization methods for data warehouses, facilitating data storage, retrieval, and management. This makes the data structure clearer, enabling rapid location and retrieval of required data and improving data management efficiency. Attached Figure Description

[0056] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which:

[0057] Figure 1 A flowchart illustrating the method for constructing a consumer industry dataset provided in this application embodiment;

[0058] Figure 2 A schematic diagram of the structure of the consumer industry dataset construction device provided in this application embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0061] Figure 1 This is a flowchart illustrating a method for constructing a consumer industry dataset, provided in an embodiment of this application. This method can be applied to different business domains, such as e-commerce and instant messaging. The process can be executed by computing devices specific to the domain, and certain input parameters or intermediate results can be manually adjusted to improve accuracy.

[0062] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0063] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0064] Figure 1 The process includes the following steps:

[0065] S101. Obtain multi-source data related to the consumer industry. Multi-source data includes: publicly available data from e-commerce platforms, policy data, news data, public opinion data, macroeconomic data, industry data, consumer complaint data, and consumer infrastructure data.

[0066] Among them, publicly available data from e-commerce platforms includes product and store information disclosed by major e-commerce platforms, which is core data reflecting the dynamics of consumer market transactions. For example, on the Tmall platform, a product in the home appliance category is listed as a 55-inch 4K smart TV of a certain brand, priced at 2,999 yuan, with monthly sales of 12,000 units and a 98% positive review rate. The core reviews are for its clear picture quality and smooth system.

[0067] Policy data refers to textual data such as policy documents, regulations, and policy interpretations related to the consumer industry. It is a key basis for analyzing the impact of policies on consumer behavior.

[0068] Information data refers to news articles related to the consumer industry, which are data used to grasp industry dynamics and market trends.

[0069] Public opinion data refers to data such as discussions on topics related to the consumer industry, user comments, and video content on social media platforms. It is the core data for understanding consumer attitudes.

[0070] Macroeconomic data reflects the state of macroeconomic operations and is a key basis for analyzing the impact of the economic situation on consumption capacity and willingness to consume.

[0071] The industry data consists of publicly available industry statistics, industry research reports, and industry standards and specifications from the official website of the Consumer Industry Association. It is authoritative data for understanding the overall scale, competitive landscape, and technical standards of the industry.

[0072] Consumer complaint data consists of records of consumer complaints about products or services, and is direct data for identifying consumer pain points and product / service problems.

[0073] Consumer infrastructure data refers to infrastructure information related to consumption scenarios, and it is data that analyzes the relationship between the spatial distribution of consumer infrastructure and consumer behavior.

[0074] Specifically, multi-source data can be obtained in the following ways: for example, distributed web crawling technology can be used to automatically trigger crawling tasks according to preset crawling rules and frequencies to obtain publicly available data from e-commerce platforms.

[0075] Among them, distributed web crawling technology is a crawling system in which multiple crawler nodes work together. Unlike single-node crawlers, it can distribute the crawling pressure and improve efficiency.

[0076] For example, an automated script can be used to access designated sections of government department websites and policy release platforms at preset intervals to obtain policy data.

[0077] Among them, the automation script is a program written in a preset language that can automatically perform access and data extraction operations according to a preset cycle.

[0078] For example, a multi-source information aggregation and collection framework is adopted to connect with the public data interfaces or page resources of news information platforms and industry media websites, and to scan the information content of the public data interfaces or page resources of news information platforms and industry media websites through keyword matching and semantic analysis algorithms to obtain information data.

[0079] Among them, the data collection tools from multiple information sources support integration with platform application programming interfaces and parsing of web page resources. They combine keyword matching (e.g., matching consumer industry with retail) with semantic analysis (e.g., using the BERT algorithm) to filter relevant information and avoid interference from irrelevant data.

[0080] For example, social media data automated collection tools can be used to call the platform's public data interfaces or parse page data to obtain public opinion information.

[0081] For example, a script that automates the crawling of links can access a government statistical database to obtain macroeconomic data.

[0082] For example, an automated webpage parsing tool can be used to access the official website of the Consumer Industry Association, automatically identify and download industry data.

[0083] For example, consumer complaint data can be obtained through standardized data interfaces that connect with consumer associations and third-party complaint platforms.

[0084] For example, consumer infrastructure data can be obtained through the geographic information open platform application programming interface and based on an automated passenger flow data collection tool.

[0085] The geographic information open platform application programming interface (API) is an interface provided by a map platform that can acquire geographic location data, such as the latitude and longitude of a shopping mall, and combine it with customer flow collection tools, such as infrared counters, to form consumer infrastructure data that combines location and flow.

[0086] S102. Based on the correlation elements, establish the correlation relationships between the data from multiple sources. These correlation elements include: time dimension, consumer subject, and consumer object.

[0087] Among them, the related elements are the key dimensions used to establish logical connections between multi-source data, including the time dimension, such as the year / month / day the data was generated, the consumer subject, such as consumer ID, company name, and the consumer object, such as product ID and service type. They are the core link to break down data silos.

[0088] The relationships between data are logical links formed by connecting scattered multi-source data through related elements, such as the association between consumer ID, product ID and time.

[0089] S103. Perform fusion processing on multi-source data with correlation relationships.

[0090] For example, relationships between data from different data sources can be established based on factors such as time dimension, consumer subject, and consumer object. Algorithms such as weighted average fusion and Dempster-Shafer evidence theory fusion can be used to fuse data describing the same consumption event or object.

[0091] Furthermore, to improve the accuracy of dataset construction, the method also includes the following steps before fusing multi-source data with correlations:

[0092] By statistically analyzing the number and proportion of missing values ​​in each field of multi-source data, target fields are identified and then supplemented or discarded. Outliers in the multi-source data are identified using statistical methods and machine learning algorithms, and then corrected or deleted. Duplicate data records are identified by calculating the hash value of each data entry from the multi-source data or by utilizing the database's deduplication function, and then deduplicated according to business requirements. Numerical data from the multi-source data is standardized to have unified units and value ranges, and textual data from the multi-source data is standardized to a machine-processable format.

[0093] The data preprocessing methods described above improve data integrity by handling missing values, preventing biases in dataset creation. Outlier handling ensures data accuracy. Deduplication optimizes data consistency and reduces redundant data interference. Data standardization enhances data usability and ensures consistency and correlation among data points.

[0094] S104. Based on the fused multi-source data, a star schema and / or snowflake schema are used to store the fused multi-source data in a data warehouse.

[0095] Specifically, through multi-dimensional analysis technology, single-level dimension carrying capacity verification, coarse-grained fixed classification verification, and dimensionless inter-table transfer association verification are performed on the fused multi-source data to determine whether the fused multi-source data simultaneously meets the judgment conditions of the three dimensions: single-level dimension carrying capacity, coarse-grained fixed classification, and dimensionless inter-table transfer association.

[0096] In this context, "single-level dimension carrying" refers to data warehouse modeling where each dimension table contains only one level of descriptive information, eliminating the need for further subdivision into sub-dimension tables. "Coarse-grained fixed classification" means the classification of merged multi-source data is at a macro-level, with stable classification criteria that are not frequently adjusted and do not require further refinement to finer sub-categories. "No intermediary links between dimension tables" means that dimension tables and the central fact table in the data warehouse can be directly linked via primary and foreign keys, without the need for indirect links through other dimension tables or intermediate tables. In other words, the link between the dimension table and the fact table is simply between the fact table and the dimension table, without any additional intermediary steps.

[0097] For example, field attribution analysis technology determines whether the merged multi-source data meets the requirement of a single-level dimension. Classification granularity verification technology determines whether the merged multi-source data meets the requirement of a coarse-grained fixed classification. Association link topology analysis technology determines whether the merged multi-source data has no intermediate associations between dimensional tables.

[0098] Based on this, if the conditions are met, the merged multi-source data is stored in the data warehouse using a star schema. If the conditions are not met, the merged multi-source data is stored in the data warehouse using a snowflake schema.

[0099] The star schema uses a fact table as its core, with multiple dimension tables directly related to it, resulting in a simple structure and high query efficiency. The fact table stores core metrics, while the dimension tables store descriptive information.

[0100] The snowflake schema is an extension of the star schema, which can be based on a dimension table and can be split into sub-tables.

[0101] Specifically, using a star schema, the merged multi-source data is stored in a data warehouse, including:

[0102] First, using a star schema, fields of consumption attribute data are extracted from the merged multi-source data according to data attributes to build a fact table in the data warehouse. Fields of subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data are also extracted from the merged multi-source data to build subject attribute dimension tables, object attribute dimension tables, comprehensive correlation dimension tables, and policy attribute dimension tables in the data warehouse, respectively.

[0103] Secondly, SQL statements are used to establish inter-table relationships between the fact table and the subject attribute dimension table, object attribute dimension table, comprehensive association dimension table, and policy attribute dimension table in the data warehouse. Foreign key fields of the fact table are linked to primary key fields of the subject attribute dimension table, object attribute dimension table, comprehensive association dimension table, and policy attribute dimension table to form a star schema relationship structure.

[0104] Subsequently, an ETL tool is used to load the subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data from the merged multi-source data, and write them into the subject attribute dimension table, object attribute dimension table, comprehensive correlation dimension table, and policy attribute dimension table, respectively.

[0105] ETL tools are responsible for data extraction, transformation, and loading, and can automatically complete the transmission and format adaptation of data from the fusion result to the data warehouse.

[0106] Finally, an ETL tool is used to load the consumption attribute data from the merged multi-source data and write the consumption attribute data into the fact table.

[0107] Specifically, the snowflake model is used to store the merged multi-source data in a data warehouse, which may include, for example:

[0108] First, the multi-source data after fusion is processed in layers according to data attributes using the snowflake model.

[0109] Secondly, by extracting core data fields of consumer infrastructure from the merged multi-source data, a facility master table is built in the data warehouse.

[0110] By extracting core data fields from industry associations, public opinion, and consumer complaints from the merged multi-source data, a master table for the industry, a master table for public opinion, and a master table for complaints can be built in the data warehouse. By extracting core fields for consumer attribute data from the merged multi-source data, a consumer fact table can be built in the data warehouse.

[0111] Fields from consumer infrastructure ancillary data are extracted from the merged multi-source data to construct facility location and facility operation sub-tables in the data warehouse. Fields from industry association ancillary data are extracted from the merged multi-source data to construct industry category sub-tables in the data warehouse. Fields from public opinion ancillary data are extracted from the merged multi-source data to construct public opinion sentiment sub-tables in the data warehouse. Fields from consumer complaint ancillary data are extracted from the merged multi-source data to construct complaint issue sub-tables in the data warehouse.

[0112] Next, SQL statements are used to establish inter-table relationships between the consumption fact table and the facility master table, industry master table, public opinion master table, and complaint master table in the data warehouse, so as to associate the foreign key fields of the consumption fact table with the primary key fields of the facility master table, industry master table, public opinion master table, and complaint master table.

[0113] Using SQL statements, establish a main table for facilities, an industry, public opinion, and complaints in the data warehouse. Then, establish inter-table relationships between these tables and sub-tables for facility location, facility operation, industry category, public opinion sentiment, and complaint issues. Finally, link foreign key fields in the facility location and facility operation sub-tables to the primary key fields in the facility main table; link foreign key fields in the industry category sub-table to the primary key fields in the industry main table; link foreign key fields in the public opinion sentiment sub-table to the primary key fields in the public opinion main table; and link foreign key fields in the complaint issues sub-table to the primary key fields in the complaint main table, thus forming a snowflake model relationship structure.

[0114] Subsequently, an ETL tool was used to load the consumer attribute data, core data of consumer infrastructure, core data of industry associations, core data of public opinion, and core data of consumer complaints from the merged multi-source data, and then wrote them into the consumer fact table, facility master table, industry master table, public opinion master table, and complaint master table, respectively.

[0115] Finally, an ETL tool was used to load the consumer infrastructure ancillary data, industry association ancillary data, public opinion ancillary data, and consumer complaint ancillary data from the merged multi-source data, and then wrote them into the construction location sub-table, facility operation sub-table, industry category sub-table, public opinion sentiment sub-table, and complaint issue sub-table, respectively.

[0116] The method provided in this application acquires multi-source data of various types, including publicly available data from e-commerce platforms, policy data, information data, public opinion data, macroeconomic data, industry data, consumer complaint data, and consumer infrastructure data, enabling the data to reflect the overall picture of the consumer industry and providing rich information for subsequent analysis.

[0117] Establishing data relationships enhances the logical coherence between data points, making previously scattered data more organized. Multi-source data fusion processing removes redundant and contradictory information, improving data accuracy and completeness. Star and snowflake schemas provide structured data organization methods for data warehouses, facilitating data storage, retrieval, and management. This makes the data structure clearer, enabling rapid location and retrieval of required data and improving data management efficiency.

[0118] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S104 will be described sequentially, but this does not mean that steps S101 to S104 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S104 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S104 can be appropriately adjusted according to actual needs.

[0119] based on Figure 1 In addition to the method described herein, this specification also provides some specific implementation schemes and extension schemes of this method, which will be further explained below.

[0120] Furthermore, to improve the accuracy of the dataset, after storing the fused multi-source data in a data warehouse, the method also includes:

[0121] Based on a pre-defined multi-dimensional indicator evaluation system, the quality of multi-source data in the data warehouse is assessed. The data in the data warehouse is then evaluated and corrected according to the assessment results.

[0122] The pre-set multi-dimensional indicator evaluation system includes indicators such as accuracy, completeness, consistency, timeliness, and relevance, and sets thresholds and weights for each indicator.

[0123] Specifically, the data to be assessed for quality is retrieved from the data warehouse. Multi-dimensional indicator evaluation calculations are performed on this data to obtain multi-dimensional indicator evaluation results. These multi-dimensional indicators include: accuracy, completeness, consistency, timeliness, and relevance. The multi-dimensional indicator evaluation results are compared with preset thresholds in the pre-defined multi-dimensional indicator evaluation system to determine whether the evaluation results meet the standards. If not, the substandard data is corrected and subjected to structured evaluation processing.

[0124] A concrete example is the regular, such as daily or weekly, monitoring and evaluation of data quality in a data warehouse. By writing data quality check scripts, the values ​​of various evaluation indicators are automatically calculated and compared with set thresholds. When an indicator value exceeds the threshold range, an alert is issued, indicating a data quality issue. Simultaneously, a data quality assessment report is generated, detailing the evaluation results of each indicator, the distribution of data quality issues, and possible root causes. Based on the data quality assessment results, corresponding optimization measures are taken to address the identified problems. For example, for accuracy issues, the data collection and cleaning processes are reviewed, and erroneous data processing logic is corrected. For completeness issues, the coverage of data collection is expanded or the method for handling missing values ​​is optimized. For consistency issues, data standards and specifications are established to unify the data formats and encoding methods of different data sources. Through continuous monitoring, evaluation, and optimization, the quality of the dataset is continuously improved.

[0125] Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0126] Figure 2 A schematic diagram of a consumer industry dataset construction device provided in this application embodiment includes:

[0127] At least one processor; and,

[0128] A memory communicatively connected to the at least one processor; wherein,

[0129] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the above-described consumer industry dataset construction methods.

[0130] Some embodiments of this application provide a non-volatile computer storage medium for constructing consumer industry datasets, storing computer-executable instructions capable of executing any of the above-described consumer industry dataset construction methods.

[0131] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0132] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0139] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for constructing a consumer industry dataset, characterized in that, The method includes: Acquire multi-source data related to the consumer industry; the multi-source data includes: publicly available data from e-commerce platforms, policy data, information data, public opinion data, macroeconomic data, industry data, consumer complaint data, and consumer infrastructure data; Based on the correlation elements, the correlation relationships between the various data from the multi-source data are established; wherein, the correlation elements include: time dimension, consumer subject, and consumer object; Fusion processing of multi-source data with correlation relationships; Based on the fused multi-source data, a star schema and / or snowflake schema are used to store the fused multi-source data in a data warehouse.

2. The method according to claim 1, characterized in that, The acquisition of multi-source data related to the consumer industry specifically includes: Using distributed web crawler technology, crawling tasks are automatically triggered according to preset crawling rules and frequencies to obtain publicly available data from the e-commerce platform; The automated scripts access designated sections of government department websites and policy release platforms at preset intervals to obtain the policy data. A multi-source information aggregation and collection framework is adopted to connect with the public data interfaces or page resources of news information platforms and industry media websites, and to scan the information content of the public data interfaces or page resources of the news information platforms and industry media websites through keyword matching and semantic analysis algorithms to obtain the information data. By using automated social media data collection tools, the platform's public data interfaces are called or page data is parsed to obtain the aforementioned public opinion information; The macroeconomic data is obtained by accessing the government statistical database through an automated script that crawls the link. Access the official website of the Consumer Industry Association through an automated web page parsing tool to automatically identify and download the industry data; The consumer complaint data is obtained through standardized data interfaces connected to consumer associations and third-party complaint platforms; The consumer infrastructure data is obtained through the geographic information open platform application interface and based on the automated passenger flow data collection tool.

3. The method according to claim 1, characterized in that, Before performing the fusion processing on the multi-source data with correlation relationships, the method further includes: By statistically analyzing the number and proportion of missing values ​​in each field of the multi-source data, the target field is determined, and the target field is supplemented or discarded. Outliers in the multi-source data are identified using statistical methods and machine learning algorithms, and these outliers are corrected or deleted. By calculating the hash value of each piece of data from the multi-source data or by using the deduplication function of the database, duplicate data records are identified, and the duplicate data records are deduplicated according to business requirements. The numerical data from the multi-source data is standardized by unifying the units and value ranges, and the text data from the multi-source data is standardized by converting it into a machine-processable format.

4. The method according to claim 1, characterized in that, The process of storing the fused multi-source data in a data warehouse using a star schema and / or snowflake schema specifically includes: By using multi-dimensional analysis techniques, single-level dimension carrying capacity verification, coarse-grained fixed classification verification, and dimensionless inter-table transfer association verification are performed on the fused multi-source data to determine whether the fused multi-source data simultaneously meets the judgment conditions of the three dimensions: single-level dimension carrying capacity, coarse-grained fixed classification, and dimensionless inter-table transfer association. If the conditions are met, the merged multi-source data will be stored in the data warehouse using the star schema. If the conditions are not met, the merged multi-source data will be stored in the data warehouse using the snowflake model.

5. The method according to claim 4, characterized in that, The process of storing the fused multi-source data in a data warehouse using the star schema specifically includes: Using the star schema, fields of consumption attribute data are extracted from the fused multi-source data according to data attributes to construct a fact table in the data warehouse. Fields of subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data are also extracted from the fused multi-source data to construct subject attribute dimension tables, object attribute dimension tables, comprehensive correlation dimension tables, and policy attribute dimension tables in the data warehouse, respectively. The SQL statements are used to establish inter-table relationships between the fact table and the subject attribute dimension table, the object attribute dimension table, the comprehensive association dimension table and the policy attribute dimension table in the data warehouse. The foreign key fields of the fact table are associated with the primary key fields of the subject attribute dimension table, the object attribute dimension table, the comprehensive association dimension table and the policy attribute dimension table to form a star schema relationship structure. The subject attribute data, object attribute data, comprehensive correlation data, and policy attribute data from the fused multi-source data are loaded using an ETL tool and written into the subject attribute dimension table, the object attribute dimension table, the comprehensive correlation dimension table, and the policy attribute dimension table, respectively. The consumption attribute data from the merged multi-source data is loaded using an ETL tool, and the consumption attribute data is written into the fact table.

6. The method according to claim 4, characterized in that, The process of storing the fused multi-source data in a data warehouse using the snowflake model specifically includes: Using the snowflake model, the fused multi-source data is processed in layers according to data attributes; By extracting core data fields of consumer infrastructure from the merged multi-source data, a facility master table is constructed in the data warehouse; By extracting core data fields of industry associations, public opinion, and consumer complaints from the merged multi-source data, an industry master table, a public opinion master table, and a complaint master table are constructed in the data warehouse. By extracting core fields of consumer attribute data from the merged multi-source data, a consumer fact table is constructed in the data warehouse. Fields from consumer infrastructure ancillary data are extracted from the merged multi-source data to construct facility location sub-tables and facility operation sub-tables in the data warehouse; fields from industry association ancillary data are extracted from the merged multi-source data to construct industry category sub-tables in the data warehouse; fields from public opinion ancillary data are extracted from the merged multi-source data to construct public opinion sentiment sub-tables in the data warehouse; fields from consumer complaint ancillary data are extracted from the merged multi-source data to construct complaint issue sub-tables in the data warehouse. Using SQL statements, establish inter-table relationships between the consumption fact table and the facility master table, industry master table, public opinion master table, and complaint master table in the data warehouse, respectively, so as to associate the foreign key fields of the consumption fact table with the primary key fields of the facility master table, industry master table, public opinion master table, and complaint master table; Using SQL statements, establish the following master tables in the data warehouse: facility master table, industry master table, public opinion master table, and complaint master table. These tables are then linked to the following sub-tables: facility location sub-table, facility operation sub-table, industry category sub-table, public opinion sentiment sub-table, and complaint issue sub-table. Foreign key fields in the facility location sub-table and facility operation sub-table are linked to the primary key fields in the facility master table; foreign key fields in the industry category sub-table are linked to the primary key fields in the industry master table; foreign key fields in the public opinion sentiment sub-table are linked to the primary key fields in the public opinion master table; and foreign key fields in the complaint issue sub-table are linked to the primary key fields in the complaint master table, thus forming a snowflake model association structure. The consumer attribute data, core data of consumer infrastructure, core data of industry associations, core data of public opinion, and core data of consumer complaints from the merged multi-source data are loaded using an ETL tool and written into the consumer fact table, the facility master table, the industry master table, the public opinion master table, and the complaint master table, respectively. An ETL tool is used to load the consumer infrastructure ancillary data, industry association ancillary data, public opinion ancillary data, and consumer complaint ancillary data from the merged multi-source data, and write them into the construction facility location sub-table, the facility operation sub-table, the industry category sub-table, the public opinion sentiment sub-table, and the complaint issue sub-table, respectively.

7. The method according to claim 1, characterized in that, After storing the fused multi-source data in a data warehouse, the method further includes: The quality of multi-source data in the data warehouse is assessed based on a pre-defined multi-dimensional indicator evaluation system. The data in the data warehouse is evaluated and corrected based on the evaluation results.

8. The method according to claim 7, characterized in that, The process of evaluating the quality of multi-source data in the data warehouse according to a preset multi-dimensional indicator evaluation system, and evaluating and correcting the data in the data warehouse based on the evaluation results, specifically includes: Obtain the data to be assessed from the data warehouse; The data to be evaluated for quality is subjected to multi-dimensional indicator evaluation calculations to obtain multi-dimensional indicator evaluation results; wherein, the multi-dimensional indicators include: accuracy indicators, completeness indicators, consistency indicators, timeliness indicators and relevance indicators; The evaluation results of the multi-dimensional indicators are compared with the preset thresholds in the preset multi-dimensional indicator evaluation system to determine whether the evaluation results of the multi-dimensional indicators meet the standards. If not, the substandard data will be corrected and subjected to structured evaluation.

9. A consumer industry dataset construction device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a consumer industry dataset construction method as described in any one of claims 1-8.

10. A non-volatile computer storage medium for constructing consumer industry datasets, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the consumer industry dataset construction method described in any one of claims 1-8.

Images