Real-time e-commerce data platform dynamic scheduling and distribution system

Abstract

The application discloses a real-time e-commerce data middle platform dynamic scheduling and distribution system and relates to the technical field of e-commerce time delay sensing, specifically comprising the following modules: an event driving module, a geographical time delay module and a distribution scheduling module; operation events of an e-commerce platform are collected in real time, event priorities are distributed according to the types and importance of the operation events, and a pushing strategy is dynamically adjusted; the geographical position of a user, the geographical position of an event source node and node time delay data are extracted, the comprehensive cost of multiple feasible paths reaching each operation event source node is dynamically calculated and sorted, the optimal and alternative paths are screened, a routing table is generated and dynamically updated; the heat is dynamically calculated based on the access frequency and user behavior mode of the operation event in real time and is marked as a heat event, the distribution priority is set, the cache strategy is optimized, the response speed and accuracy of risk control, recommendation and payment are improved, the redundancy and bandwidth occupation of cross-region transmission are reduced, and the overall data transmission cost and resource waste are reduced.

Description

Technical Field

[0001] This invention relates to the field of e-commerce latency perception technology, specifically to a real-time e-commerce data platform dynamic scheduling and distribution system. Background Technology

[0002] In the e-commerce industry, event data is a core resource driving transaction conversion, personalized recommendations, real-time risk control, and operational optimization. As business scale continues to expand, data sources become increasingly complex, user behavior becomes more diversified, and the demand for low latency, high availability, and strong consistency increases, traditional static, single-point-of-contact data processing and scheduling models are no longer sufficient to meet the timeliness and flexibility requirements of current scenarios.

[0003] The existing end-to-end approach combining geographic latency awareness with multi-path routing optimization is relatively fragmented, lacking a unified and dynamically adjustable collaborative scheme for routing table generation and cache distribution.

[0004] In traditional content delivery networks, routing optimization is mostly focused on static topology, single latency metrics, or fixed caching strategies. It lacks real-time optimization that incorporates factors such as latency weights, routing congestion, and path penalties to build a comprehensive cost, as well as dynamic cache copy management for trending events.

[0005] In e-commerce scenarios, the lack of end-to-end, real-time adjustable unified scheduling logic for distinguishing user behavior events, backend operation events, and external operation events, as well as dynamic push strategies, can easily lead to a contradiction between timeliness and resource utilization.

[0006] Delay-aware networks aim for single shortest paths or simple weights, and rarely dynamically combine multiple factors such as link delay, processing delay, queuing delay, congestion, and path length according to weights and regularly update the routing table.

[0007] Traditional data platforms emphasize multi-source data access but lack end-to-end capabilities for real-time scheduling and distribution.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] The purpose of this invention is to provide a real-time e-commerce data platform dynamic scheduling and distribution system to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a real-time e-commerce data platform dynamic scheduling and distribution system, specifically including the following modules: an event-driven module, a geographic latency module, and a distribution scheduling module;

[0011] Event-driven module: Collects operation events from the e-commerce platform in real time, assigns event priorities based on the type and importance of the operation events, and dynamically adjusts the push strategy;

[0012] Geographic latency module: Extracts the user's geographic location, the geographic location of the event source node, and node latency data; dynamically calculates and sorts the comprehensive cost of multiple feasible paths to each operation event source node; filters the optimal and alternative paths; and generates and dynamically updates the routing table.

[0013] Distribution and scheduling module: dynamically calculates popularity based on the access frequency of operation events and user behavior patterns in real time, marks them as popular events, sets distribution priorities, and optimizes caching strategies.

[0014] As a preferred embodiment of the real-time e-commerce data platform dynamic scheduling and distribution system described in this invention, wherein:

[0015] Real-time collection of operational events from e-commerce platforms and conversion into standardized structured data;

[0016] The operation events include user behavior events (users browse products, click on products, add products to shopping cart, and make payments), backend operation events (inventory changes, order status updates, and product price changes), and external operation events (payment success callbacks from third-party payment platforms and logistics information updates).

[0017] Transmitted via an event queue;

[0018] Assign event priorities based on the type and importance of the operation event;

[0019] The allocation of event priorities specifically includes,

[0020] User behavior events: low priority;

[0021] Background operation events: Medium priority;

[0022] External operation events: High priority;

[0023] The push strategy is dynamically adjusted based on event priority, specifically including...

[0024] High-priority operation events trigger instant push notifications;

[0025] Medium-priority operational events trigger queuing and parallel push;

[0026] Low-priority operation events are pushed out during off-peak hours based on the current load.

[0027] As a preferred embodiment of the real-time e-commerce data platform dynamic scheduling and distribution system described in this invention, wherein:

[0028] Extracting user geolocation based on user behavior;

[0029] Extract the geographical location and node latency data of the source node of the operation event based on the operation event IP;

[0030] Network latency from the user's geographic location to the geographic location of the operation event source node is calculated using latency awareness.

[0031] The optimal transmission path from the user's geographical location to the geographical location of the operation event source node is selected based on the topology results of the operation event source node;

[0032] The process of selecting the optimal transmission path from the user's geographical location to the geographical location of the operation event source node specifically includes,

[0033] For each operation event source node, list all feasible transmission paths from the user's geographical location to the geographical location of the operation event source node based on the topology information;

[0034] The cost of initializing the transmission path is the sum of the link delays;

[0035] The comprehensive cost of each transmission path is calculated by accumulating the link delay, processing delay, queuing delay, path congestion, path length penalty, and the product of their respective weight coefficients.

[0036] The weighting coefficients are dynamically adjusted based on latency priority or packet loss sensitivity.

[0037] Define topology constraints to suppress transmission paths through inefficient nodes and require an exit point to be specified via an operation event;

[0038] The overall cost is sorted, and the transmission path with the lowest cost is selected as the first priority transmission path. The transmission path with the second lowest cost is selected as the alternative priority transmission path under the same operation event source node.

[0039] The routing table is dynamically generated based on network latency, and the geographical location of the operation event source node closest to the user's geographical location is used as the priority routing target;

[0040] For each user request, the node with the lowest latency and the closest geographical location is selected as the optimal event source node through the routing table;

[0041] Update the routing table based on changes in network latency;

[0042] Whenever there are new user behaviors or changes in network conditions, the routing selection and transmission path of operation events are updated in real time through an event push mechanism.

[0043] After each request, the routing table is dynamically adjusted based on new latency data and user behavior data.

[0044] As a preferred embodiment of the real-time e-commerce data platform dynamic scheduling and distribution system described in this invention, wherein:

[0045] Based on the access frequency of operation events and user behavior patterns, the popularity of operation events is dynamically calculated and marked as popular events;

[0046] For content marked as trending events, copies are retained at multiple user locations and the transport node with the lowest latency.

[0047] The distribution priority is dynamically set based on the urgency of user behavior and the frequency of access to operation events;

[0048] The distribution priority specifically includes,

[0049] P1: Highest priority corresponds to high urgency and high access frequency;

[0050] P2: The second highest priority corresponds to medium urgency, medium to high access frequency, or high user preference;

[0051] P3: Medium priority corresponds to general urgency or low to medium access frequency and the trend is unclear;

[0052] P4: Low priority corresponds to low urgency, low access frequency, and a decreasing trend;

[0053] If the current user behavior corresponds to a trending event, the operation event should be obtained based on the optimal event source node first.

[0054] When a user action is triggered, it is determined whether the corresponding operation event already exists in the most recent cache node, specifically including:

[0055] If the operation event exists in the cache and is valid, then return the operation event directly.

[0056] If the operation event is not in the cache or the operation event in the cache has expired, retrieve the latest operation event from the operation event source node;

[0057] For high-profile events, select cache nodes based on the optimal storage strategy and set a longer cache validity period;

[0058] For cold events, the cache validity period is short, or no cache is used;

[0059] When an operation event changes, the operation events on each cache node are updated in a timely manner according to the set cache update strategy.

[0060] When an operation event changes, the cache invalidation mechanism is triggered, and the changed operation event is synchronized to each cache node;

[0061] Set a cache eviction policy to periodically evict unused operation events based on cache usage frequency and storage space.

[0062] For high-frequency access events, cached copies should be retained first; for low-frequency access events, the cache should be cleared promptly.

[0063] On the other hand, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements the steps of the real-time e-commerce data platform dynamic scheduling and distribution system as described above.

[0064] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements the steps of the real-time e-commerce data middle platform dynamic scheduling and distribution system as described above in the present invention.

[0065] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0066] (1) Through the collaboration of the event-driven module, the geographic latency module and the distribution scheduling module, millisecond-level to second-level event processing, routing decision and distribution execution can be achieved. Traditional systems are mostly static scheduling or batch processing modes, and lack latency awareness and geographic optimization mechanisms.

[0067] (2) For different types of events such as user behavior, medium, and external operations, clearly assign priorities and implement dynamic push strategies such as instant push / queuing parallel / timed sending. Traditional systems are often linear queues or simple distribution strategies, lacking fine-grained scheduling based on event type and importance.

[0068] (3) Event priority-driven instant push combined with latency-aware routing significantly shortens the downstream consumption latency of key events and improves the response speed and accuracy of key scenarios such as risk control, recommendation, and payment;

[0069] (4) Dynamic path selection, adaptive adjustment of routing tables and hot-driven replica distribution reduce redundancy and bandwidth occupation in cross-regional transmission, and reduce overall data transmission costs and resource waste. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0071] Figure 1This is a flowchart of the method for the real-time e-commerce data platform dynamic scheduling and distribution system of the present invention.

[0072] Figure 2 This is a schematic diagram of the modules of the real-time e-commerce data platform dynamic scheduling and distribution system of the present invention. Detailed Implementation

[0073] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0074] Example 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention. This embodiment provides a real-time e-commerce data platform dynamic scheduling and distribution system, which specifically includes the following modules: an event-driven module, a geographic latency module, and a distribution scheduling module.

[0075] Event-driven module: Collects operation events from the e-commerce platform in real time, assigns event priorities based on the type and importance of the operation events, and dynamically adjusts the push strategy;

[0076] Real-time collection of operational events from e-commerce platforms and conversion into standardized structured data;

[0077] The operation events include user behavior events (users browse products, click on products, add products to shopping cart, and make payments), backend operation events (inventory changes, order status updates, and product price changes), and external operation events (payment success callbacks from third-party payment platforms and logistics information updates).

[0078] Transmitted via an event queue;

[0079] Assign event priorities based on the type and importance of the operation event;

[0080] The allocation of event priorities specifically includes,

[0081] User behavior events: low priority;

[0082] Background operation events: Medium priority;

[0083] External operation events: High priority;

[0084] The push strategy is dynamically adjusted based on event priority, specifically including...

[0085] High-priority operation events trigger instant push notifications;

[0086] Medium-priority operational events trigger queuing and parallel push;

[0087] Low-priority operation events are pushed during off-peak hours based on the current load.

[0088] More specifically, the factors for dynamic adjustment include: real-time load, event urgency signals, business strategies, and event correlation. Event urgency signals include payment failures, inventory alerts, and price anomalies.

[0089] Geographic latency module: Extracts the user's geographic location, the geographic location of the event source node, and node latency data; dynamically calculates and sorts the comprehensive cost of multiple feasible paths to each operation event source node; filters the optimal and alternative paths; and generates and dynamically updates the routing table.

[0090] Extracting user geolocation based on user behavior;

[0091] Extract the geographical location and node latency data of the source node of the operation event based on the operation event IP;

[0092] Network latency from the user's geographic location to the geographic location of the operation event source node is calculated using latency awareness.

[0093] It should also be noted that the latency awareness explicitly measures, estimates, and uses end-to-end network latency to evaluate the real-time cost of candidate transmission paths in routing and distribution decisions, thereby prioritizing low-latency and stable transmission paths.

[0094] The optimal transmission path from the user's geographical location to the geographical location of the operation event source node is selected based on the topology results of the operation event source node;

[0095] The process of selecting the optimal transmission path from the user's geographical location to the geographical location of the operation event source node specifically includes,

[0096] For each operation event source node, list all feasible transmission paths from the user's geographical location to the geographical location of the operation event source node based on the topology information;

[0097] The cost of initializing the transmission path is the sum of the link delays;

[0098] The comprehensive cost of each transmission path is calculated by accumulating the link delay, processing delay, queuing delay, path congestion, path length penalty, and the product of their respective weight coefficients.

[0099] The weighting coefficients are dynamically adjusted based on latency priority or packet loss sensitivity.

[0100] Define topology constraints to suppress transmission paths through inefficient nodes and require an exit point to be specified via an operation event;

[0101] The overall cost is sorted, and the transmission path with the lowest cost is selected as the first priority transmission path. The transmission path with the second lowest cost is selected as the alternative priority transmission path under the same operation event source node.

[0102] The routing table is dynamically generated based on network latency, and the geographical location of the operation event source node closest to the user's geographical location is used as the priority routing target;

[0103] For each user request, the node with the lowest latency and the closest geographical location is selected as the optimal event source node through the routing table;

[0104] Update the routing table based on changes in network latency;

[0105] It should also be noted that the candidate paths are re-evaluated and a new routing table is generated every 30 seconds, with the minimum improvement threshold being that the new transmission path must be more than 5% better than the current transmission path;

[0106] Whenever there are new user behaviors or changes in network conditions, the routing selection and transmission path of operation events are updated in real time through an event push mechanism.

[0107] After each request, the routing table is dynamically adjusted based on new latency data and user behavior data.

[0108] Distribution and scheduling module: dynamically calculates popularity based on the access frequency and user behavior patterns of operation events in real time and marks them as popular events, sets distribution priorities, and optimizes caching strategies;

[0109] Based on the access frequency of operation events and user behavior patterns, the popularity of operation events is dynamically calculated and marked as popular events;

[0110] For content marked as trending events, copies are retained at multiple user locations and the transport node with the lowest latency.

[0111] The distribution priority is dynamically set based on the urgency of user behavior and the frequency of access to operation events;

[0112] The distribution priority specifically includes,

[0113] P1: Highest priority corresponds to high urgency and high access frequency;

[0114] P2: The second highest priority corresponds to medium urgency, medium to high access frequency, or high user preference;

[0115] P3: Medium priority corresponds to general urgency or low to medium access frequency and the trend is unclear;

[0116] P4: Low priority corresponds to low urgency, low access frequency, and a decreasing trend;

[0117] If the current user behavior corresponds to a trending event, the operation event should be obtained based on the optimal event source node first.

[0118] When a user action is triggered, it is determined whether the corresponding operation event already exists in the most recent cache node, specifically including:

[0119] If the operation event exists in the cache and is valid, then return the operation event directly.

[0120] If the operation event is not in the cache or the operation event in the cache has expired, retrieve the latest operation event from the operation event source node;

[0121] For high-profile events, select cache nodes based on the optimal storage strategy and set a longer cache validity period;

[0122] For cold events, the cache validity period is short, or no cache is used;

[0123] When an operation event changes, the operation events on each cache node are updated in a timely manner according to the set cache update strategy.

[0124] When an operation event changes, the cache invalidation mechanism is triggered, and the changed operation event is synchronized to each cache node;

[0125] Set a cache eviction policy to periodically evict unused operation events based on cache usage frequency and storage space.

[0126] For high-frequency access events, cached copies should be retained first; for low-frequency access events, the cache should be cleared promptly.

[0127] By coordinating the event-driven module, the geographic latency module, and the distribution and scheduling module, millisecond- to second-level event processing, routing decisions, and distribution execution are achieved. In contrast, traditional systems are mostly static scheduling or batch processing modes, lacking latency awareness and geographic optimization mechanisms.

[0128] For different types of events such as user behavior, medium-level events, and external operations, a dynamic push strategy is implemented that clearly assigns priorities and enables instant push, queuing and parallel processing, and timed delivery. Traditional systems often use linear queues or simple distribution strategies, lacking fine-grained scheduling based on event type and importance.

[0129] Event priority-driven instant push combined with latency-aware routing significantly reduces downstream consumption latency for critical events, improving the response speed and accuracy of key scenarios such as risk control, recommendation, and payment.

[0130] Dynamic path selection, adaptive routing table adjustment, and hot-driven replica distribution reduce redundancy and bandwidth consumption in cross-regional transmissions, thereby reducing overall data transmission costs and resource waste.

[0131] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A real-time e-commerce data platform dynamic scheduling and distribution system, characterized in that, Specifically, it includes the following modules: event-driven module, geographic latency module, and distribution and scheduling module; Event-driven module: Collects operation events from the e-commerce platform in real time, assigns event priorities based on the type and importance of the operation events, and dynamically adjusts the push strategy; Geographic latency module: Extracts the user's geographic location, the geographic location of the event source node, and node latency data; dynamically calculates and sorts the comprehensive cost of multiple feasible paths to each operation event source node; filters the optimal and alternative paths; and generates and dynamically updates the routing table. Distribution and scheduling module: dynamically calculates popularity based on the access frequency of operation events and user behavior patterns in real time, marks them as popular events, sets distribution priorities, and optimizes caching strategies.

2. The real-time e-commerce data platform dynamic scheduling and distribution system according to claim 1, characterized in that: The event-driven module Real-time collection of operational events from e-commerce platforms and conversion into standardized structured data; The operation events include user behavior events, background operation events, and external operation events; Transmitted via an event queue; Assign event priorities based on the type and importance of the operation event; The push strategy is dynamically adjusted based on event priority, specifically including... High-priority operation events trigger instant push notifications; Medium-priority operational events trigger queuing and parallel push; Low-priority operation events are pushed out during off-peak hours based on the current load.

3. The real-time e-commerce data platform dynamic scheduling and distribution system according to claim 2, characterized in that: The allocation of event priorities specifically includes, User behavior events: low priority; Background operation events: Medium priority; External operation events: high priority.

4. The real-time e-commerce data platform dynamic scheduling and distribution system according to claim 1, characterized in that: The geographic latency module, Extracting user geolocation based on user behavior; Extract the geographical location and node latency data of the source node of the operation event based on the operation event IP; Network latency from the user's geographic location to the geographic location of the operation event source node is calculated using latency awareness. The optimal transmission path from the user's geographical location to the geographical location of the operation event source node is selected based on the topology results of the operation event source node; Define topology constraints to suppress transmission paths through inefficient nodes and require an exit point to be specified via an operation event; The overall cost is sorted, and the transmission path with the lowest cost is selected as the first priority transmission path. The transmission path with the second lowest cost is selected as the alternative priority transmission path under the same operation event source node. The routing table is dynamically generated based on network latency, and the geographical location of the operation event source node closest to the user's geographical location is used as the priority routing target; For each user request, the node with the lowest latency and the closest geographical location is selected as the optimal event source node through the routing table; Update the routing table based on changes in network latency.

5. The real-time e-commerce data platform dynamic scheduling and distribution system according to claim 4, characterized in that: The process of selecting the optimal transmission path from the user's geographical location to the geographical location of the operation event source node specifically includes, For each operation event source node, list all feasible transmission paths from the user's geographical location to the geographical location of the operation event source node based on the topology information; The cost of initializing the transmission path is the sum of the link delays; The comprehensive cost of each transmission path is calculated by accumulating the link delay, processing delay, queuing delay, path congestion, path length penalty, and the product of their respective weight coefficients. The weighting coefficients are dynamically adjusted based on latency priority or packet loss sensitivity.

6. The real-time e-commerce data platform dynamic scheduling and distribution system according to claim 1, characterized in that: Based on the access frequency of operation events and user behavior patterns, the popularity of operation events is dynamically calculated and marked as popular events; For content marked as trending events, copies are retained at multiple user locations and the transport node with the lowest latency. The distribution priority is dynamically set based on the urgency of user behavior and the frequency of access to operation events; If the current user behavior corresponds to a trending event, the operation event should be obtained based on the optimal event source node first. When a user action is triggered, it is determined whether the corresponding operation event already exists in the most recent cache node; When an operation event changes, the cache invalidation mechanism is triggered, and the changed operation event is synchronized to each cache node; Set a cache eviction policy to periodically evict unused operation events based on cache usage frequency and storage space. For high-frequency access events, cached copies should be retained first; for low-frequency access events, the cache should be cleared promptly.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements a module of the real-time e-commerce data platform dynamic scheduling and distribution system as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the module of the real-time e-commerce data platform dynamic scheduling and distribution system as described in any one of claims 1 to 6.

Images