E-commerce data dynamic acquisition and intelligent management system based on rule subscription

Abstract

The invention provides an e-commerce data dynamic acquisition and intelligent management system based on rule subscription, which relates to the technical field of e-commerce data acquisition and management, and comprises a rule subscription layer, a data acquisition layer and a data governance layer, the rule subscription layer is configured with a dynamic rule module and supports operation personnel to preset shop acquisition rules and implementation personnel to set emergency intervention rules; the data acquisition layer comprises an intelligent link generator which is internally provided with a multi-platform URL template library and is used for dynamically splicing standardized commodity links according to SKUIDs and issuing crawler tasks; the data governance layer executes a five-level cleaning mechanism which sequentially comprises platform compliance inspection, rule matching, aging verification, manual rechecking and data structured conversion. Through the three-layer architecture of the rule subscription layer, the data acquisition layer and the data governance layer, the whole-process automation from rule configuration to data storage is realized, and the operation manpower cost is saved.

Description

Technical Field

[0001] This invention relates to the field of e-commerce data collection and management technology, and more specifically, to a rule-based e-commerce data dynamic collection and intelligent management system. Background Technology

[0002] With the rapid development of e-commerce, third-party operation platforms need to manage product data from hundreds of stores and hundreds of thousands of SKUs simultaneously. According to industry data, leading third-party operation companies need to process more than 120,000 product information update requests per day, of which review data collection accounts for 35%. However, existing technologies have significant shortcomings: Manually dependent data collection mode: Operations staff need to manually export evaluation data, identify SKU codes, and concatenate product links. On average, each SKU configuration takes 3.2 minutes, with an error rate as high as 18%. Fixed-rule crawler system: The periodic polling mechanism based on a preset URL list cannot dynamically respond to changes in operational strategies, and changes in platform rules result in a link failure rate of up to 37%; Single-dimensional filtering mechanism: It only supports keyword blacklist filtering and cannot implement multi-condition combination strategies of "platform + store + category + spuiid + skuid". Invalid data feedback causes operations staff to waste an average of 2.3 hours per day for manual screening. Furthermore, a leading e-commerce data management service provider's ticketing system shows that 72% of customer service complaints stem from delayed product information updates, and 65% of operations staff spend overtime dealing with data integration issues. Existing technology is no longer sufficient to meet the demands of efficient operations. Therefore, this application addresses this issue by proposing a rule-based subscription-based dynamic data collection and intelligent management system for e-commerce. Summary of the Invention

[0003] This invention provides a rule-based e-commerce data dynamic collection and intelligent management system, including a rule subscription layer, a data collection layer, and a data governance layer; The rule subscription layer is configured with a dynamic rule module, which supports operators to preset store collection rules and implementers to set emergency intervention rules. The data acquisition layer includes an intelligent link generator with a built-in multi-platform URL template library, which is used to dynamically concatenate standardized product links based on SKUID and issue crawler tasks. The data governance layer implements a five-level cleaning mechanism, which consists of platform compliance checks, rule matching, timeliness verification, manual review, and data structuring transformation.

[0004] As a preferred technical solution of this application, the intelligent link generator extracts the unique identifier of the product through the SKUID intelligent parsing engine and dynamically generates cross-platform product links based on a preset URL template.

[0005] As a preferred technical solution of this application, the rule subscription layer supports a multi-condition combination filtering strategy of "platform + store + category + spuiid + skuid" to realize the custom configuration of data collection rules; The rule change response time of the rule subscription layer is in the millisecond range, meeting the needs of real-time monitoring and policy adjustment.

[0006] As a preferred technical solution of this application, the manual review module of the data governance layer intervenes in links that fail the automatic review, and manually adds them to the "all links" library after confirmation, thus forming a closed loop of data collection; After generating a product link, the data acquisition layer sends out a second crawler task to collect the specifications, category affiliation, and price fluctuation data of the product details page.

[0007] As a preferred technical solution of this application, it also includes a standardized data interface for connecting with ERP and CRM systems, and supports a hybrid architecture of private deployment and SaaS model.

[0008] As a preferred technical solution in this application, it also includes: data collection initiation phase, product link generation and details collection, multi-level rule filtering and review, and subscription rule execution and data closure.

[0009] As a preferred technical solution of this application, the data acquisition initiation phase includes: The implementers add target stores on the operations platform, triggering the system to issue a web crawler task; The web crawler collects product review data from the store and extracts the unique product identifier through the SKUID intelligent parsing engine.

[0010] As a preferred technical solution of this application, the product link generation and details collection include: The system dynamically concatenates standardized product links based on SKUID; The crawler task was issued a second time to collect specifications, category affiliation, and price fluctuations from the product details page; The generated product links are added to the collection link list in the e-commerce backend to form the initial data pool.

[0011] As a preferred technical solution of this application, the multi-level rule filtering and review includes: The system performs automated review based on preset rules: Automatic review: If a link meets the preset operational conditions, it will be directly added to the "All Links" library; Manual review: If automatic addition is not set, the link will not be automatically added to the full link database from the link list of the rules, and will wait for review by the implementers; The implementers manually review suspicious links and add them to the "All Links" database after confirmation.

[0012] As a preferred technical solution of this application, the subscription rule execution and data closure include: All links added in the system are crawled to accurately collect evaluation data (such as user reviews, ratings, and customer photos) corresponding to a specified SKUID. The collected results are sent to the message in a standardized message format for users to access and analyze.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: In the scheme of this application: 1. Fully automated: This application achieves full automation from rule configuration to data entry through a three-layer architecture of rule subscription layer, data collection layer and data governance layer, saving operational manpower costs; 2. Dynamic Response Capability: Rule changes take effect in real time, solving the problem that fixed-rule crawler systems cannot respond to changes in operational strategies and avoiding link failures caused by platform rule adjustments; 3. Multi-platform compatibility: Built-in multi-platform URL template library and intelligent link generator to eliminate data collection blind spots caused by differences in URL formats of e-commerce platforms; 4. Precise data governance: A five-level cleaning mechanism (platform compliance check, rule matching, timeliness verification, manual review, and data structuring transformation) is adopted, combined with a multi-condition filtering strategy to significantly reduce invalid data interference and improve data quality. Attached Figure Description

[0014] Figure 1 A schematic diagram of the five-level cleaning mechanism provided in this application; Figure 2 The implementation flowchart of the rule-based e-commerce data dynamic collection and intelligent management system provided for this application; Figure 3 The interface screenshots for adding a store provided in this application; Figure 4 The interface diagram for adding a data collection task provided in this application; Figure 5 Interface diagram for data synchronization provided in this application; Figure 6 An image of the e-commerce review link for this product provided in this application; Figure 7 An interface diagram for the rule subscription provided in this application; Figure 8 An interface diagram for adding rules provided for this application; Figure 9 The interface diagram of the e-commerce review provided for this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0016] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0017] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0018] Example 1, please refer to Figure 1-Figure 2 A rule-based e-commerce data dynamic collection and intelligent management system, comprising a rule subscription layer, a data collection layer, and a data governance layer; The rule subscription layer is configured with a dynamic rule module, supporting operators to preset store data collection rules and implementers to set emergency intervention rules. This design significantly improves the system's flexibility and response speed. Operators can preset store data collection rules according to business needs, such as customizing collection strategies for specific promotional activities to ensure the system accurately collects the required data. Implementers can set emergency intervention rules to quickly adjust the collection strategy in case of emergencies (such as abnormal data fluctuations), ensuring the accuracy and availability of data. This is thanks to the modular design of the rule subscription layer, which allows personnel in different roles to configure rules independently according to their respective responsibilities, avoiding mutual interference and achieving millisecond-level rule change response time, meeting the needs of real-time monitoring and strategy adjustment. The data acquisition layer includes an intelligent link generator with a built-in multi-platform URL template library. This generator dynamically constructs standardized product links based on SKUIDs and issues crawler tasks. The application of the intelligent link generator significantly improves the efficiency and accuracy of data acquisition. Through the built-in multi-platform URL template library, the system can dynamically generate standardized product links based on SKUIDs, eliminating the need for manual input and reducing error rates. Furthermore, this design supports cross-platform data acquisition, adapting to the URL structure differences of various e-commerce platforms and broadening the system's applicability. This is because the SKUID intelligent parsing engine can accurately extract the unique identifier of the product, and combined with preset URL templates, ensures that the generated links conform to the specifications of each platform, thereby improving the success rate of crawler tasks and reducing acquisition costs. The data governance layer implements a five-level cleansing mechanism, namely platform compliance check, rule matching, timeliness verification, manual review, and data structuring transformation. This five-level cleansing mechanism ensures high data quality and availability. The platform compliance check ensures that the collected data complies with the regulations of each e-commerce platform, avoiding legal risks caused by unauthorized collection. Rule matching filters data that meets business needs based on preset rules, improving data relevance. Timeliness verification ensures the timeliness of data, preventing outdated data from affecting decision-making. Manual review intervenes in data that fails automatic review, further improving data accuracy. Data structuring transformation converts unstructured data into structured data, facilitating subsequent analysis and application. This multi-layered cleansing mechanism, through layer-by-layer checks, gradually improves data quality, providing reliable data support for enterprise decision-making.

[0019] Furthermore, the intelligent link generator extracts unique product identifiers through the SKUID intelligent parsing engine and dynamically generates cross-platform product links based on preset URL templates. This function enhances the system's versatility and adaptability. The SKUID intelligent parsing engine can accurately identify the unique identifiers of products on different e-commerce platforms, achieving precise extraction even if the encoding rules of each platform differ. Dynamically generating cross-platform product links based on preset URL templates allows the system to seamlessly switch collection tasks between multiple e-commerce platforms without needing to develop separate collection modules for each platform. This not only reduces the system's development and maintenance costs but also increases the coverage of data collection, providing a guarantee for enterprises to obtain more comprehensive market data.

[0020] Furthermore, the rule subscription layer supports multi-condition combination filtering strategies of "platform + store + category + spuid + skuid," enabling customized configuration of data collection rules. The rule change response time of the rule subscription layer is in the millisecond range, allowing for real-time monitoring and strategy adjustment. The multi-condition combination filtering strategy provides enterprises with highly flexible data collection control capabilities. Operations personnel can accurately filter the data range to be collected based on specific business needs, avoiding the collection of irrelevant data and improving the efficiency and targeting of data collection. The millisecond-level rule change response time allows the system to respond in real-time to market changes and adjustments to business needs. For example, during e-commerce promotional activities, operations personnel can quickly adjust collection rules, strengthen data monitoring of promotional products, and obtain timely market feedback, providing strong support for adjustments to the enterprise's marketing strategies.

[0021] Furthermore, the manual review module of the data governance layer intervenes in links that fail the automatic review, and manually adds them to the "All Links" library after confirmation, forming a closed loop of data collection; After generating product links, the data acquisition layer sends out a second crawler task to collect specifications, category affiliation, and price fluctuation data from the product details page. The combination of a manual review module and the second collection mechanism significantly improves data integrity and accuracy. Manual review intervenes in links that fail automatic review, identifying and correcting potential misjudgments during the automatic review process, ensuring that all eligible data is collected. The second crawler task collects detailed information from the product details page, enriching the data content and providing enterprises with more comprehensive product information. These two mechanisms form a complete data acquisition closed loop, from link generation to data collection, and then to data review and supplementary collection, ensuring data quality and integrity and providing a more reliable basis for enterprise data analysis and decision-making.

[0022] Furthermore, it includes standardized data interfaces for integration with ERP and CRM systems, supporting both private deployment and a hybrid SaaS architecture. These standardized data interfaces enable seamless integration with existing enterprise business systems. By integrating with ERP and CRM systems, enterprises can combine collected e-commerce data with internal business data, achieving data sharing and collaborative applications. For example, combining e-commerce platform sales data with the enterprise's inventory management system enables real-time inventory monitoring and automatic replenishment. Support for a hybrid private deployment and SaaS architecture meets the diverse needs of different enterprises. Enterprises can choose the appropriate deployment method based on their own circumstances, ensuring data security and controllability while reducing system deployment and maintenance costs.

[0023] Furthermore, it also includes: the data collection initiation phase, product link generation and details collection, multi-level rule filtering and review, and subscription rule execution and data closure. Each phase has clear functions and objectives, which facilitates module design and debugging by developers. At the same time, the phased design also allows the system to be flexibly adjusted according to actual needs. For example, in the data collection initiation phase, different triggering methods can be selected according to different business scenarios. In the multi-level rule filtering and review phase, review steps can be added or reduced according to business needs, improving the system's adaptability and flexibility.

[0024] Furthermore, the data acquisition initiation phase includes: The implementers add target stores on the operations platform, triggering the system to issue a web crawler task; The web crawler collects product review data from stores and extracts unique product identifiers using the SKIID intelligent parsing engine. Implementers only need to add the target store to the operations platform, and the system will automatically issue crawler tasks without complex configuration. The crawler's collection of product review data and extraction of unique product identifiers provides the foundation for subsequent product link generation and data collection. The precise extraction by the SKIID intelligent parsing engine ensures the uniqueness and accuracy of product identifiers, providing a reliable basis for subsequent system processing.

[0025] Furthermore, the generation of product links and collection of product details include: The system dynamically concatenates standardized product links based on SKUID; The crawler task was issued a second time to collect specifications, category affiliation, and price fluctuations from the product details page; The generated product links are added to the collection link list in the e-commerce backend to form an initial data pool; dynamically splicing standardized product links improves the efficiency and accuracy of link generation and reduces manual intervention; a second crawler task is issued to collect detailed information, enriching the data content and providing enterprises with more comprehensive product information; the generated product links are added to the collection link list in the e-commerce backend to form an initial data pool, providing a data foundation for subsequent data processing and analysis; this design realizes an automated process from product identifier extraction to product information collection, improving the efficiency and quality of data collection.

[0026] Furthermore, the multi-level rule filtering and review includes: The system performs automated review based on preset rules: Automatic review: If a link meets the preset operational conditions, it will be directly added to the "All Links" library; Manual review: If automatic addition is not set, the link will not be automatically added to the full link database from the link list of the rules, and will wait for review by the implementers; The implementation team manually reviews suspicious links and adds them to the "All Links" database upon confirmation. A multi-level rule-based filtering and review mechanism ensures the quality of data entering the "All Links" database. Automated review quickly filters out data that meets preset conditions, improving review efficiency. Manual review provides opportunities for human intervention in suspicious links, avoiding potential misjudgments from automated review. This multi-layered review mechanism, combining automation and human intervention, ensures both the efficiency of data review and improves data accuracy and reliability, providing high-quality data support for enterprises.

[0027] Furthermore, the execution of the subscription rules and the data closure loop include: All links added in the system are crawled to accurately collect evaluation data (such as user reviews, ratings, and customer photos) corresponding to a specified SKUID. The collected results are sent to a message queue in a standardized message format for users to access and analyze. Precise collection of evaluation data corresponding to specific SKUIDs ensures the data's relevance and accuracy. The standardized message format facilitates user access and analysis by placing the collected results into a message queue. This design achieves a closed-loop process from data collection to data analysis, enabling enterprises to obtain timely market feedback and providing strong support for product optimization and marketing strategy adjustments. Simultaneously, the standardized message format improves the data's universality and exchangeability, facilitating integration and data sharing with other systems.

[0028] The front-end interactive interface of this system is as follows: Figures 3-9 As shown.

[0029] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.

Claims

1. A rule-based subscription-based e-commerce data dynamic collection and intelligent management system, characterized in that, It includes a rule subscription layer, a data collection layer, and a data governance layer; The rule subscription layer is configured with a dynamic rule module, which supports operators to preset store collection rules and implementers to set emergency intervention rules. The data acquisition layer includes an intelligent link generator with a built-in multi-platform URL template library, which is used to dynamically concatenate standardized product links based on SKUID and issue crawler tasks. The data governance layer implements a five-level cleaning mechanism, which consists of platform compliance checks, rule matching, timeliness verification, manual review, and data structuring transformation.

2. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 1, characterized in that, The intelligent link generator extracts the unique identifier of a product through the SKIID intelligent parsing engine and dynamically generates cross-platform product links based on a preset URL template.

3. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 1, characterized in that, The rule subscription layer supports multi-condition combination filtering strategies of "platform + store + category + spuiid + skuid", enabling custom configuration of data collection rules; The rule change response time of the rule subscription layer is in the millisecond range, meeting the needs of real-time monitoring and policy adjustment.

4. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 1, characterized in that, The manual review module of the data governance layer intervenes in links that fail the automatic review, and manually adds them to the "All Links" library after confirmation, thus forming a closed loop of data collection. After generating a product link, the data acquisition layer sends out a second crawler task to collect the specifications, category affiliation, and price fluctuation data of the product details page.

5. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 1, characterized in that, It also includes standardized data interfaces for integration with ERP and CRM systems, and supports hybrid architectures such as private deployment and SaaS models.

6. The e-commerce data dynamic collection and intelligent management system based on rule subscription according to any one of claims 1-5, characterized in that, Also includes: The data collection process includes the initial data collection phase, product link generation and details collection, multi-level rule filtering and review, and subscription rule execution and data closure.

7. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 6, characterized in that, The data acquisition initiation phase includes: The implementers add target stores on the operations platform, triggering the system to issue a web crawler task; The web crawler collects product review data from the store and extracts the unique product identifier through the SKUID intelligent parsing engine.

8. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 7, characterized in that, The generation of product links and collection of product details include: The system dynamically concatenates standardized product links based on SKUID; The crawler task was issued a second time to collect specifications, category affiliation, and price fluctuations from the product details page; The generated product links are added to the collection link list in the e-commerce backend to form the initial data pool.

9. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 8, characterized in that, The multi-level rule filtering and approval includes: The system performs automated review based on preset rules: Automatic review: If a link meets the preset operational conditions, it will be directly added to the "All Links" library; Manual review: If automatic addition is not set, the link will not be automatically added to the full link database from the link list of the rules, and will wait for review by the implementers; The implementers manually review suspicious links and add them to the "All Links" database after confirmation.

10. The e-commerce data dynamic collection and intelligent management system based on rule subscription as described in claim 9, characterized in that, The execution of subscription rules and data closure include: All links added in the system are crawled to accurately collect evaluation data corresponding to a specified SKUID; The collected results are sent to the message in a standardized message format for users to access and analyze.

Images