Method for constructing ten-million-level high-load middle platform system oriented to intra-city distribution business

Abstract

The invention relates to the technical field of middle station systems of city-wide delivery services, and discloses a ten-million-level high-load middle station system architecture method oriented to city-wide delivery services, which comprises the following steps of: aggregating order streams into an order-time fluctuation sequence; a resource availability signal is generated based on the core calculation module, and a mapping relation between the idle calculation capability and the order load under different time period lengths is established; outputting accurate aggregation delivery computing resource requirements of a plurality of prediction time periods in the future after the current time period to determine the configuration and the number of core computing modules meeting the prediction order processing requirements; and a resource preheating instruction used for scheduling the dynamic redundancy pool is generated according to the prediction time period, so that the calculation state of the service calculation cluster is prepared in advance before the prediction time period comes, and the calculation cold start overhead and the central cache access delay in the peak period are reduced. The method is beneficial for avoiding the mismatching condition of idle computing resources in the non-peak period and huge data processing pressure in the peak period of the computing resources of the intra-city distribution business.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of city distribution business middle platform system, and particularly relates to a method for a city distribution business-oriented ten-million-level high-load middle platform system architecture. BACKGROUND

[0002] The data of city distribution business (such as Meituan distribution, Beebirdie instant distribution, etc.) has extremely distinctive characteristics compared with traditional e-commerce or logistics data, for example: the data generation rate is not uniform, but presents a sharp pulse peak with meal peak, weather change, holiday promotion and other events, and tens of thousands of users may simultaneously place orders at a moment, and tens of thousands of concurrent requests per second need to be processed, the data shows the characteristics of mass and simultaneity. Further, the city distribution business has high requirements for data processing speed and timeliness, and many operations must be completed within milliseconds or seconds; and the state and position of the core data object of the city distribution business change dramatically every minute and every second, and each order information will also change after being generated until the order task is completed. At the same time, the data is not unidirectional, and has a large amount of unstructured data.

[0003] That is, the data characteristics of city distribution business are: high concurrency and transient peak, high real-time and strong time sequence, high mobility and geographical space attribute, and downstream complexity.

[0004] The data background of city distribution business faces significant challenges when processing massive data, and data congestion during the order peak period is one of the core challenges. In order to cope with the data processing pressure during the peak period, the background system must reserve a large amount of redundant computing resources (computing, memory, bandwidth), which are idle during off-peak periods, causing waste; and during the peak period, these data resources face very large data processing pressure, resulting in a mismatch between the idle and waste of computing resources during off-peak periods and the huge data processing pressure during the peak period due to the design of a large number of redundant computing resources.

[0005] The middle platform is a service-oriented organizational structure and technical architecture, which aims to integrate and coordinate various business systems within the platform, build a shared and scalable core platform, and provide standardized infrastructure, shared business capabilities and data resources to support business innovation and digital transformation. The core idea of the middle platform is to abstract each business system as an independent business capability and manage and schedule it through a unified intermediate layer. The architecture of the middle platform usually includes data middle platform, application middle platform and technology middle platform. The data middle platform is responsible for integrating and managing data resources inside and outside the enterprise, the application middle platform provides standardized business functions and services, and the technology middle platform provides unified technical infrastructure and support for business systems.

[0006] In order to solve the mismatching situation that the non-peak period of the computing resources is idle and wasted, and the data processing pressure of the peak period is huge due to the design of a large number of redundant computing resources of the same city distribution business, the application provides a method for constructing a million-level high-load middle platform system for the same city distribution business. SUMMARY

[0007] The main purpose of the application is to provide a method for constructing a million-level high-load middle platform system for the same city distribution business, and to solve the technical problem of the mismatching of the non-peak period of the computing resources being idle and wasted, and the data processing pressure of the peak period being huge due to the design of a large number of redundant computing resources of the data background of the existing same city distribution business.

[0008] In order to achieve the above purpose, the application provides a method for constructing a million-level high-load middle platform system for the same city distribution business, wherein the middle platform system comprises a global resource coordinator, and an order collector, a predictive resource modeling module, a business computing cluster and a dynamic redundancy pool which are in communication connection with the global resource coordinator; the business computing cluster comprises a plurality of parallel processing core computing modules, and the dynamic redundancy pool comprises a plurality of idle computing groups; the method comprises the following steps: The order collector continuously captures the new order flow to input the business computing cluster for the same city order aggregation distribution calculation, and aggregates the new order flow into an order-time fluctuation sequence according to a preset time period length; The global resource coordinator detects the processing state of each core computing module in the business computing cluster in real time, and generates a resource availability signal based on the idle state of the core computing module; The global resource coordinator fuses the resource availability signal and the order-time fluctuation sequence in real time, generates load-resource correlation time series data, and establishes the mapping relationship between the idle computing capacity and the order load under different time period lengths; The predictive resource modeling module receives the load-resource correlation time series data, and calls historical resource load data for comparative analysis, outputs the accurate aggregation distribution calculation resource demand of a plurality of predicted time periods after the current time period, and determines the configuration and quantity of the core computing module meeting the predicted order processing demand; The global resource coordinator receives the accurate aggregation distribution calculation resource demand, generates a resource preheating instruction for scheduling the dynamic redundancy pool, so that the computing state of the business computing cluster is ready in advance before each predicted time period, and the computing cold start overhead and central cache access delay of the peak period are reduced.

[0009] Optionally, the predictive resource modeling module receives load-resource association time series data, and calls historical resource load data for comparative analysis, and outputs accurate aggregated distribution computing resource requirements of future prediction periods after the current period, to determine the configuration and quantity of the core computing modules that meet the predicted order processing requirements, including: The predictive resource modeling module receives load-resource association time series data. The load-resource association time series data is input into the neural network model, and historical resource load data is called for comparative analysis. The neural network model encodes the load-resource association time series data as spatiotemporal features, and outputs accurate aggregated distribution computing resource requirements of future prediction periods after the current period, wherein the accurate aggregated distribution computing resource requirements include the configuration and quantity of the core computing modules that meet the order processing requirements of each prediction period.

[0010] Optionally, the global resource coordinator receives the accurate aggregated distribution computing resource requirements, generates resource warm-up instructions for scheduling dynamic redundancy pools, so that the computing state of the business computing cluster is ready in advance before each prediction period, reducing the computing cold start overhead and central cache access delay during peak periods, including: The global resource coordinator receives the accurate aggregated distribution computing resource requirements of each prediction period, and generates resource warm-up instructions according to the accurate aggregated distribution computing resource requirements. Through the resource warm-up instructions of each prediction period, the currently idle core computing modules in the dynamic redundancy pool that meet the computing resource requirements are dynamically organized into a warm-up computing group. Before the prediction period arrives, the warm-up computing group performs pre-computation tasks to pre-load and process the geographic grid, distribution path and merchant hotspot data required for the prediction period, and generates warm-up results. The warm-up results are warmed up to the distributed cache module corresponding to the business computing cluster, so that the computing state of the business computing cluster is ready in advance before the prediction period arrives, to reduce the computing cold start overhead and central cache access delay during peak periods.

[0011] Optionally, the order collector continuously captures new order streams to input the business computing cluster for same-city order aggregated distribution computing, and aggregates the new order streams into order-time fluctuation sequences according to a preset period length, including: The order collector listens to the order message queue to continuously subscribe and capture new order messages from the order generation source in an asynchronous streaming processing manner, forming a new order stream. The new order stream is written into the order processing pipeline of the business computing cluster in real time, triggering the business computing cluster to perform same-city order aggregated distribution computing. The newly added order stream is accumulated and counted based on a sliding time window through a time length aggregator arranged in the order collector; At the end of each preset time length, the time length aggregator outputs the total number of orders within the time length and arranges them in time sequence to form the order-time fluctuation sequence.

[0012] Optionally, the global resource coordinator detects the processing state of each core computing module in the business computing cluster in real time, and the step of generating a resource availability signal based on the core computing module in an idle state comprises: The global resource coordinator collects real-time performance indicators of each core computing module at a preset frequency through an agent program deployed on each core computing module in the business computing cluster; The global resource coordinator compares the received real-time performance indicators with a set of preset multi-dimensional thresholds through an internal state decision maker; According to the comparison result, the state decision maker outputs a state identifier for each core computing module; The state decision maker encapsulates the state identifiers of all core computing modules currently marked as idle state and the corresponding idle computing resource specifications as a resource availability signal and sends it to the resource mapping unit of the global resource coordinator.

[0013] Optionally, the step of the global resource coordinator fusing the resource availability signal with the order-time fluctuation sequence in real time to generate load-resource association time series data to establish the mapping relationship between idle computing capacity and order load under different time lengths comprises: The global resource coordinator maintains a two-dimensional association model with time as the horizontal axis and system resources as the vertical axis through an internal space-time fusion unit; The space-time fusion unit receives the resource availability signal and the order-time fluctuation sequence aligned by timestamp; for each same sampling time point, the space-time fusion unit takes the order aggregation amount at the current time point as the load characteristic value and takes the set of idle core computing modules and idle computing power at the current time point as the resource supply characteristic value; The load characteristic value and the resource supply characteristic value are associated to generate a resource-load mapping point; The resource-load mapping points of continuous multiple time points are combined and serialized in time sequence to form the load-resource association time series data, so as to dynamically represent the coupling relationship and evolution trend between system resource supply and business load in the historical period and the current time period through the load-resource association time series data.

[0014] Optionally, the date type to which the current time point belongs, the holiday flag and the promotion activity flag are dynamically adjusted by the dynamic weight value in the resource-load mapping point, so as to identify the importance of the resource-load mapping point in subsequent prediction model training.

[0015] Optionally, after the step of associating the load feature value with the resource supply feature value to generate a resource-load mapping point, the method further comprises: The spatio-temporal fusion unit detects whether the current time point reaches an order load mutation point through the resource-load mapping point; If yes, the sampling frequency before and after the current time point is increased according to the order load mutation variable, so as to generate a mapping point sequence with higher time resolution for accurately describing the mutation process.

[0016] Optionally, the step of writing the new order flow into the order processing pipeline of the business computing cluster in real time, and triggering the business computing cluster to perform the same-city order aggregation and distribution calculation comprises: Constructing a real-time calculation module with multiple processing units; Writing the new order flow into an order partition unit of the real-time calculation module, and the order partition unit distributes the orders to corresponding geographic calculation units in real time based on the pickup address and delivery address geocodes of the orders, wherein the geographic calculation units correspond one-to-one to the geographic grid areas of the city; The geographic calculation unit performs spatio-temporal density-based clustering analysis on the orders flowing into the geographic grid area under its jurisdiction; Pushing the aggregated batches after clustering analysis to a parallel path optimization unit to generate a distribution trajectory; Submitting the distribution trajectory and locking the distribution resources, thereby completing the aggregation and distribution calculation process from the new order flow to the distribution scheme output.

[0017] Optionally, the step of performing spatio-temporal density-based clustering analysis on the orders flowing into the geographic grid area under the jurisdiction of the geographic calculation unit comprises: The geographic calculation unit adopts a dynamic parameter clustering algorithm on the orders flowing into the geographic grid area under its jurisdiction; According to the order inflow rate and distribution density, the neighborhood threshold is automatically scaled to adaptively generate the optimal aggregated batch.

[0018] The technical solution of the present application is beneficial to solve the technical problem that the data back-end of the existing same-city distribution business has a large number of redundant computing resources, and the non-peak period computing resources are idle and wasted, while the data processing pressure during the peak period is huge.

[0019] Specifically, the order collector sends the received new order stream to the business computing cluster for intra-city order aggregation and distribution calculation, and aggregates the new order stream into order-time fluctuation sequences according to a preset time period length; meanwhile, the global resource coordinator generates a resource availability signal based on the idle state of the core computing module in the order processing process; the global resource coordinator fuses the resource availability signal with the order-time fluctuation sequence in real time to generate load-resource association time series data, so as to establish a mapping relationship between idle computing capacity and order load under different time period lengths; the predictive resource modeling module outputs accurate aggregation and distribution calculation resource demand of a plurality of predicted time periods after the current time period according to the load-resource association time series data; the global resource coordinator receives the accurate aggregation and distribution calculation resource demand and generates a resource warm-up instruction for scheduling the dynamic redundancy pool, so that the computing state of the business computing cluster is ready in advance before each predicted time period, thereby reducing the computing cold start overhead and central cache access delay in the peak period. Therefore, the global resource coordinator uses the idle core computing resources in the dynamic redundancy pool to provide warm-up calculation for intra-city order aggregation and distribution before each predicted time period arrives, so as to load and process the background data, service data, business data and the like in the intra-city aggregation and distribution calculation process in advance by using the idle core computing module before the arrival of the high-load period, thereby reducing the computing pressure of the core processing module in the business order peak period. Thus, in the technical solution of the present application, the computing resources that are idle during the off-peak period are not wasted, but are used to provide warm-up calculation services for the data peak, and the data processing amount in the peak period is transferred to the off-peak period for processing, which is beneficial to solve the technical problem of mismatch between idle waste of computing resources during the off-peak period and huge data processing pressure during the peak period. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of the method for the million-level high-load mid-platform architecture for intra-city distribution business in the first embodiment of the present application is shown. Figure 2 A functional module diagram in the present application is shown. Figure 3 A resource warm-up flowchart in the present application is shown. Figure 4 A flowchart of generating clustering batches and distribution trajectories for the new order stream in the present application is shown.

[0021] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0023] In the following description, the suffixes such as "unit", "part", or "unit" used to represent elements are used only for the convenience of the description of the present application, and have no specific meaning by themselves. Therefore, "unit", "part", or "unit" can be used mixedly.

[0024] Referring to Figures 1 to 4 In the first embodiment of the present application, a method for providing a million-level high-load middle station system architecture for a same-city distribution service is provided, wherein the middle station system comprises a global resource coordinator, an order collector, a predictive resource modeling module, a service computing cluster, and a dynamic redundancy pool, which are respectively connected in communication with the global resource coordinator; the service computing cluster comprises a plurality of parallel processing core computing modules, and the dynamic redundancy pool comprises a plurality of idle computing groups; the method comprises the following steps: Step S10, the order collector continuously captures new order streams to input the service computing cluster for same-city order aggregation distribution calculation, and aggregates the new order streams into order-time fluctuation sequences according to a preset time period length; Step S20, the global resource coordinator detects the processing state of each core computing module in the service computing cluster in real time, and generates a resource availability signal based on the idle state of the core computing module; Step S30, the global resource coordinator fuses the resource availability signal with the order-time fluctuation sequence in real time, generates load-resource correlation time series data, and establishes a mapping relationship between idle computing capacity and order load under different time period lengths; Step S40, the predictive resource modeling module receives the load-resource correlation time series data, and calls historical resource load data for comparative analysis, outputs accurate aggregation distribution calculation resource requirements for a plurality of predicted time periods after the current time period, and determines the configuration and quantity of the core computing modules to meet the predicted order processing requirements; Step S50, the global resource coordinator receives the accurate aggregation distribution calculation resource requirements, generates a resource preheating instruction for scheduling the dynamic redundancy pool, so that the computing state of the service computing cluster is ready in advance before each predicted time period, and the computing cold start overhead and central cache access delay during the peak period are reduced.

[0025] The technical solution of the present application is beneficial to solve the technical problem that the data background of the existing same-city distribution service has a large amount of idle and wasted computing resources during the off-peak period, and a huge data processing pressure during the peak period.

[0026] Specifically, the order collector sends the received new order stream to the business computing cluster for intra-city order aggregation and distribution calculation, and aggregates the new order stream into an order-time fluctuation sequence according to a preset time period length; meanwhile, the global resource coordinator generates a resource availability signal based on the idle state of the core computing module in the order processing process; the global resource coordinator fuses the resource availability signal and the order-time fluctuation sequence in real time to generate load-resource association time series data, so as to establish the mapping relationship between the idle computing capacity and the order load under different time period lengths; the predictive resource modeling module outputs the accurate aggregation and distribution calculation resource demand of a plurality of predicted time periods after the current time period according to the load-resource association time series data; the global resource coordinator receives the accurate aggregation and distribution calculation resource demand and generates a resource preheating instruction for scheduling the dynamic redundancy pool, so that the computing state of the business computing cluster is ready in advance before each predicted time period, reducing the computing cold start overhead and central cache access delay during the peak period. Therefore, the global resource coordinator uses the idle core computing resources in the dynamic redundancy pool to provide preheating calculation for intra-city order aggregation and distribution before each predicted time period arrives, so as to load and process the background data, service data, business data and the like in the intra-city aggregation and distribution calculation process in advance using the idle core computing module before the arrival of the high-load period, thereby reducing the computing pressure of the core processing module during the peak period of business orders. Therefore, in the technical scheme of the present application, the computing resources that are idle during the off-peak period are not wasted, but are used to provide preheating calculation services for the data peak, and the data processing amount during the peak period is transferred to the off-peak period for processing, which is beneficial to solve the technical problem of mismatch between the idle waste of computing resources during the off-peak period and the huge data processing pressure during the peak period.

[0027] Specifically, the order-time fluctuation sequence reflects the fluctuation of the order quantity with the time period, and reflects the order growth characteristics in a time sequence.

[0028] The resource availability signal is used to feed back the currently available core computing module and the total idle computing resources provided by the available core computing module.

[0029] The load-resource association time series data reflects the mapping relationship between the idle computing capacity and the order load under different time period lengths.

[0030] According to the load-resource association time series data in the application, the approximate historical resource load data meeting the approximate time period, approximate idle computing capacity and approximate order load is called, and the aggregated distribution computing resource demand in the historical period equivalent to the prediction period is called to generate the configuration and quantity of the core computing module required for processing new orders in the future prediction period after the current period, and the remaining computing resources of each prediction period are used to establish the redundancy pool corresponding to the prediction period. Since the redundancy pool changes with the demand computing module of each prediction period, the redundancy pool changes dynamically, which is a dynamic redundancy pool.

[0031] Based on the first embodiment of the method for constructing a million-level high-load middle platform system architecture for a same-city distribution business of the application, the method for constructing a million-level high-load middle platform system architecture for a same-city distribution business of the application in the second embodiment comprises the following steps: Step S41, the predictive resource modeling module receives load-resource association time series data; Step S42, inputting the load-resource association time series data into a neural network model and calling historical resource load data for comparative analysis; Step S43, the neural network model encodes the load-resource association time series data as a space-time feature and outputs the accurate aggregated distribution computing resource demand in the future prediction period after the current period, wherein the accurate aggregated distribution computing resource demand includes the configuration and quantity of the core computing module meeting the order processing demand of each prediction period.

[0032] Specifically, the load-resource association time series data has three-dimensional data features: time dimension, load characteristic quantity and resource characteristic quantity. The time dimension refers to specific time data, including set time factors: Gregorian calendar date, day of the week, lunar calendar date and specific time, which are all influencing factors of the distribution business volume; for example: May 1, 2025, Thursday, the fourth day of the fourth lunar month, Beijing time 12:00.

[0033] The load characteristic quantity and the resource characteristic quantity are both collected in the form of set sampling interval as the length of the time period to record the load characteristic quantity and the resource characteristic quantity in discrete time series.

[0034] The load characteristic quantity includes: total order quantity, i.e. the total number of orders in the time period; order peak rate, i.e. the highest value of order requests per second in the time period; average order value, i.e. the average amount of orders in the time period; order geographic distribution discrete data, which is used to quantify the dispersion degree of orders in geography, and the higher the value is, the more dispersed it is.

[0035] Resource characteristics include: a list of idle core computing module IDs, which is a set of identifiers of core computing modules that are currently in an idle state (including those with no computing tasks and light load states with load rates below a preset value); total idle computing power, which is the total idle computing power of core computing modules in an idle state; total idle memory, which is the total available memory of all the aforementioned idle core computing modules; and network latency data, which refers to the network latency of the idle core computing modules.

[0036] The load characteristics and resource characteristics at each time point constitute the resource-load feature vector; Each time point The resource-load feature vector is transformed into a high-dimensional dense vector through an embedding layer. Thus, the resource-load feature vectors at all time points form the input sequence; In neural network models, temporal convolutional networks extract local spatiotemporal features from the input sequence, efficiently capturing short-term local dependencies and periodic patterns between order load and resource status. For example, it can identify local causal patterns such as a decrease in idle resources two time points after a small increase in order volume. It then calculates the correlation between current local spatiotemporal features and historical spatiotemporal features, and performs global comparative analysis and importance weighting. For instance, the model might automatically learn that the system state pattern at 8 PM last Wednesday has high reference value for predicting resource demand at 8 PM this Wednesday (periodic similarity). The attention mechanism assigns very high weights to the features at 8 PM last Wednesday. Simultaneously, it can also capture the impact of non-periodic anomalous events, such as: although it's a normal day, the current order surge pattern is very similar to a historical sudden promotional event, thus referencing the resource consumption of that event for this prediction.

[0037] Therefore, in this embodiment, when the model is input with a certain (load + resource) mode, it outputs the actual resource demand situation within several historical periods following the occurrence of the same similar mode in history.

[0038] The model's output includes resource quantity information and resource configuration information. The resource quantity information indicates the required number of core computing modules, while the resource configuration information is a predefined core computing resource specification configuration, including the configuration details of the core computing resources.

[0039] Please refer to Figure 3 In the first embodiment of the high-load middleware system architecture method for same-city delivery business based on the present invention, and in the third embodiment of the same-city delivery business high-load middleware system architecture method based on the present invention, step S50 includes: Step S51, the global resource coordinator receives the accurate aggregate distribution computing resource demand of each prediction period, and generates a resource warm-up instruction according to the accurate aggregate distribution computing resource demand; Step S52, through the resource warm-up instruction of each prediction period, the currently idle core computing module in the dynamic redundancy pool that meets the computing resource demand is dynamically organized into a warm-up computing group; Step S53, before the prediction period arrives, the pre-computation task is executed by the warm-up computing group to pre-load and process the geographic grid, distribution path and merchant hotspot data required in the prediction period, and generate a warm-up result; Step S54, the warm-up result is warmed up to the distributed cache module corresponding to the business computing cluster, so that the computing state of the business computing cluster is ready in advance before the prediction period arrives, to reduce the cold start overhead and central cache access delay in the peak period.

[0040] Figure 3 In this embodiment, three prediction periods are used for illustration, wherein, denotes the first prediction period, denotes the second prediction period, denotes the second prediction period.

[0041] In this embodiment, the whole high-load middle platform system provides an active warm-up process by the resource warm-up instruction using the idle core computing module, and is no longer passive waiting for resource shortage, but based on accurate prediction, actively organizes idle resources into a warm-up group, executes a pre-computation task, and pre-loads the result into the cache, so that the system is in a hot standby state before the business peak arrives, thereby realizing a fundamental change from cold start to hot switching.

[0042] Specifically, the global resource coordinator scans the dynamic redundancy pool according to the accurate aggregate distribution computing resource demand of the future prediction periods after the current period, and finds all idle core computing modules with matching computing specifications. The selected core computing modules are logically divided into a warm-up computing group associated with the prediction period in the scanning process. The group is marked with the corresponding prediction period and warm-up group label to isolate it from the business computing cluster processing real-time production traffic within the corresponding prediction period, to ensure that the marked warm-up computing group executes the warm-up instruction before the corresponding prediction period arrives, and is not assigned a same-city order aggregate distribution computing task, so that the warm-up task does not compete for CPU, memory and I / O of the computing resource, thereby ensuring the smoothness of real-time business.

[0043] Further, before the prediction period arrives, the computing state of the business computing cluster is prepared in advance, and the data generated in the prediction period is stored in the distributed cache module. After the prediction period ends, when the new order flow is below the preset threshold, the cache data in the distributed cache module is migrated to the central cache. At the same time, after the prediction period ends, when the new order flow is below the preset threshold, the newly generated actual load-resource association time series data can also be input into the predictive resource modeling module as new historical resource load data to train the neural network model, so that the model can be trained and optimized during the non-order peak period, and more accurate prediction can be achieved.

[0044] Before the prediction period arrives, the geographic grid, distribution path and merchant hotspot data required for the prediction period are preheated and loaded in advance, and the hotspot data corresponding to the prediction period is preferentially loaded, including hotspot geographic grid, hotspot distribution path and hotspot merchant, so that the hotspot data can be preferentially loaded, thereby achieving high preheating efficiency in the early stage of executing the preheating task. The hotspot data refers to data with high access frequency in the prediction period.

[0045] Based on the first embodiment of the method for constructing a million-level high-load middle platform system architecture for a same-city distribution business of the application, in the fourth embodiment of the method for constructing a million-level high-load middle platform system architecture for a same-city distribution business of the application, step S10 comprises: Step S11, the order collector listens to the order message queue to continuously subscribe and capture new order messages from the order generation source in an asynchronous streaming processing manner, to form a new order flow; Step S12, the new order flow is written into the order processing pipeline of the business computing cluster in real time, triggering the business computing cluster to perform same-city order aggregation and distribution calculation; Step S13, the order collector is set to a period length aggregator, and the new order flow is accumulated and counted based on a sliding time window; Step S14, at the end of each preset period length, the period length aggregator outputs the total number of orders in the period length, and arranges them in time sequence to form the order-time fluctuation sequence.

[0046] The length of the sliding time window matches the prediction minimum granularity period of the predictive resource modeling module. The derived indicators of each data point in the order-time fluctuation sequence include: order peak rate in the period length, order geographic distribution discrete data and average order value weight.

[0047] The order collector subscribes to the message queue of the order, and the order collector pulls the message from the message queue in an asynchronous and non-blocking manner, so that the order collector does not wait for an order to be completely processed before taking the next one, but continuously processes the messages in the queue to form a stable data stream.

[0048] The order collector writes the message to the input interface of the business computing cluster almost synchronously while processing the queue message. The interface can be connected to a stream processing module. Once the order data flows into the processing process of the business computing cluster, the predefined processing logic (i.e., the intra-city order aggregation and distribution calculation) is immediately triggered for execution. It is equivalent to driving the process of the processing logic through the event of the order flow. Seamless real-time connection from data collection to business processing is realized, ensuring that the end-to-end delay from order generation to start of processing is minimized.

[0049] In the first embodiment of the method for constructing a million-level high-load middle platform system architecture for intra-city distribution business of the application, the step S20 in the fifth embodiment of the method for constructing a million-level high-load middle platform system architecture for intra-city distribution business of the application comprises: Step S21, the global resource coordinator collects real-time performance indicators of each core computing module through the agent program deployed on each core computing module in the business computing cluster at a preset frequency; Step S22, the global resource coordinator compares the received real-time performance indicators with a set of preset multi-dimensional thresholds through the built-in state decision maker; Step S23, according to the comparison result, the state decision maker outputs a state identifier for each core computing module; Step S24, the state decision maker encapsulates the state identifiers of all core computing modules currently marked as idle state and the corresponding idle computing resource specifications into a resource availability signal, and sends it to the resource mapping unit of the global resource coordinator.

[0050] The real-time performance indicators include CPU utilization, memory occupancy, the number of orders in the processing queue, and network I / O load; the state identifier is one of busy, light load, idle, and unavailable.

[0051] The preset multi-dimensional thresholds include CPU utilization threshold, memory occupancy threshold, order quantity threshold in the processing queue, and network I / O load threshold.

[0052] This embodiment provides an intelligent state decision mechanism based on multi-dimensional indicator fusion. Rich performance data is collected in real time through a lightweight agent, and a comprehensive judgment is made through a decision maker, and finally a standardized resource availability signal is generated. The signal not only contains the quantity of resources, but also contains the specifications and state of the resources, providing reliable and high-quality input for subsequent accurate resource mapping and scheduling.

[0053] In the first embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service, in the sixth embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service, the step S30 comprises: In step S31, the global resource coordinator maintains a two-dimensional correlation model with time as the horizontal axis and system resources as the vertical axis through the built-in space-time fusion unit. In step S32, the space-time fusion unit receives the resource availability signal and the order-time fluctuation sequence aligned by time stamp. In step S33, for each same sampling time point, the space-time fusion unit takes the order aggregation amount at the current time point as a load characteristic value, and takes the idle core computing module set and the idle computing power at the current time point as resource supply characteristic values. In step S34, the load characteristic value is associated with the resource supply characteristic value to generate a resource-load mapping point. In step S35, the resource-load mapping points of continuous multiple time points are combined and serialized in time sequence to form the load-resource correlation time series data, so as to dynamically represent the coupling relationship and evolution trend between system resource supply and business load in the historical period and the current time period through the load-resource correlation time series data.

[0054] In the resource availability signal, the idle core computing module set (the idle core computing module is a light-load core computing module with a load amount lower than a preset value) and the corresponding idle computing power at each sampling time are included. The order aggregation amount at the end of each period length is included in the order-time fluctuation sequence.

[0055] Specifically, the space-time fusion unit does not simply store historical data, but actively and dynamically constructs and maintains the load-resource correlation time series data. The resource state and the business state of the system at each time point are accurately corresponded and linked to reveal the evolution trend, thereby providing the neural network model with high-quality input features rich in space-time semantics describing the overall behavior of the system.

[0056] In the sixth embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service, in the seventh embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service: The date type, holiday flag and promotion activity flag to which the current time point belongs are dynamically adjusted by a dynamic weight value in the resource-load mapping point, so as to identify the importance of the resource-load mapping point in subsequent prediction model training.

[0057] In the eighth embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service according to the application, the step S34 comprises: In the eighth embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service according to the application, the step S34 comprises: If yes, the step S342 is performed, and the sampling frequency before and after the current time point is increased according to the order load mutation variable, so as to generate a mapping point sequence with higher time resolution for accurately describing the mutation process.

[0058] Specifically, fixed-frequency sampling (such as once per minute) is a common practice for monitoring systems. However, when the system encounters a sudden traffic (such as a flash sale, a hot event), fixed sampling will miss a lot of key details. For example, a 2-minute traffic peak may only be captured as 1-2 abnormally high data points under 1 minute / sampling, and the information such as the steep rising edge and falling edge shape, the exact position and duration of the peak cannot be recorded. This causes the subsequent prediction model to be unable to learn the complete pattern of the sudden traffic, and it is difficult to make accurate responses.

[0059] Therefore, in this embodiment, the sampling frequency before and after the current time point is increased according to the order load mutation variable, so as to generate a mapping point sequence with higher time resolution for accurately describing the mutation process, capture the transient behavior of order changes, improve the learning ability of the prediction model to the sudden pattern, optimize the preheating time of the computing resource, more accurately control the starting time of the preheating operation, neither waste resources too early nor come too late to preheat, and realize more accurate scheduling.

[0060] Please refer to Figure 4 In the ninth embodiment of the method for constructing a million-level high-load middle platform system for a same-city distribution service according to the application, the step S12 comprises: Step S121, constructing a real-time computing module with multiple processing units; Step S122, writing the new order stream into the order partition unit of the real-time computing module, and the order partition unit distributes the orders in real time to the corresponding geographic computing unit based on the geographic coding of the pickup address and the geographic coding of the delivery address of the orders, wherein the geographic computing unit corresponds one-to-one to the geographic grid area of the city; Step S123, the geographic computing unit performs time-space density-based clustering analysis on the orders flowing into the geographic grid area under its jurisdiction; Step S124, pushing the aggregated batches after the clustering analysis to the parallel path optimization unit to generate a distribution trajectory; Step S125, submit the delivery trajectory and lock the delivery resource, so as to complete the aggregated delivery calculation process from the new order inflow to the delivery scheme output.

[0061] The embodiment provides a hierarchical, parallel and intelligent real-time calculation pipeline, which divides the huge order processing task into four clear and parallel processing stages: partition, clustering, parallel optimization and submission, and realizes low delay and high throughput automatic processing from order inflow to delivery scheme output through the cooperation of dynamic partition algorithm, clustering algorithm and hybrid optimization algorithm.

[0062] The real-time calculation module includes an order partition unit, a geographic calculation unit and a parallel path optimization unit. This structure realizes function decoupling and horizontal expansion. Each unit can be independently scaled, and the bottleneck unit (such as the parallel path optimization unit) can be solved by increasing the parallelism.

[0063] In the geographic coding, the order partition unit extracts the longitude and latitude of the pickup address and the delivery address of each order, and maps it to the geographic grid where it is located. In the algorithm, the system maintains a virtual calculation ring, and the nodes on the ring correspond to each geographic calculation unit. When an order needs to be allocated, its set key identifier (such as the delivery address grid) is mapped to the ring. The global resource coordinator monitors the load (such as the number of orders to be processed) of each geographic calculation unit. If the load of a unit is too high, the global resource coordinator will dynamically add virtual nodes to the virtual calculation ring, so that new orders are more likely to be balanced to other units with lighter load.

[0064] In the tenth embodiment of the method for constructing a million-level high-load middle platform system for a same-city delivery business of the application, the step S123 includes: Step S123a, the geographic calculation unit uses a dynamic parameter clustering algorithm on the orders flowing into the geographic grid under its jurisdiction. Step S123b, automatically scale the neighborhood threshold according to the order inflow rate and distribution density to adaptively generate the optimal aggregation batch.

[0065] Specifically, the order collector calculates the traffic of the new order flow, determines the order cluster granularity of clustering and the coverage area of single path planning. When the traffic is larger, the order cluster granularity of clustering is smaller, and the coverage area of single path planning is also smaller, so as to improve the processing speed and quickly form the order cluster batch for clustering; on the contrary, when the traffic is smaller, the order cluster granularity of clustering is larger, and the coverage area of single path planning is also larger, so as to appropriately slow down the processing speed and realize more optimal path planning.

[0066] Those skilled in the art can clearly understand the method of the above-mentioned embodiments can be realized by means of software and necessary general hardware platform, or by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions for making a terminal device enter the method described in various embodiments of the present application.

[0067] In the description of the present specification, the description of the terms "one embodiment", "another embodiment", "other embodiments", or "first embodiment to Xth embodiment" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, method steps or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0068] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0069] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0070] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields based on the content of the present application specification and drawings, are also included in the patent protection scope of the present application.

Claims

1. A high-load middleware system architecture method for same-city delivery services with a capacity of tens of millions of users, characterized in that: The middleware system includes a global resource coordinator, and an order collector, a predictive resource modeling module, a business computing cluster, and a dynamic redundancy pool, all of which are communicatively connected to the global resource coordinator. The business computing cluster includes multiple core computing modules that process in parallel, and the dynamic redundancy pool includes several idle computing groups. The method includes the following steps: The order collector continuously captures new order streams to input into the business computing cluster for same-city order aggregation and delivery calculation, and aggregates the new order streams into an order-time fluctuation sequence according to a preset time period. The global resource coordinator monitors the processing status of each core computing module in the business computing cluster in real time and generates resource availability signals based on the idle status of the core computing modules. The global resource coordinator fuses resource availability signals with order-time fluctuation sequences in real time to generate load-resource correlation time-series data, thereby establishing a mapping relationship between idle computing capacity and order load under different time periods. The predictive resource modeling module receives load-resource correlation time series data and calls historical resource load data for comparative analysis. It outputs the accurate aggregated delivery computing resource requirements for several future predicted time periods after the current time period, so as to determine the configuration and quantity of core computing modules that meet the predicted order processing requirements. The global resource coordinator receives precise aggregated and distributed computing resource demands, generates resource preheating instructions for scheduling dynamic redundancy pools, thereby ensuring that the computing state of the business computing clusters is ready in advance before each forecast period, reducing the overhead of computing cold start and central cache access latency during peak periods.

2. The high-load middleware system architecture method for same-city delivery services with tens of millions of users as described in claim 1, characterized in that: The predictive resource modeling module receives load-resource correlation time-series data, calls historical resource load data for comparative analysis, and outputs accurate aggregated delivery computing resource requirements for several predicted time periods after the current time period. This process, which determines the configuration and quantity of core computing modules to meet the predicted order processing requirements, includes: The predictive resource modeling module receives load-resource correlation time-series data; Input load-resource correlation time series data into the neural network model, and call historical resource load data for comparative analysis; The neural network model encodes load-resource correlation time series data as spatiotemporal features and outputs the precise aggregated delivery computing resource requirements for several future predicted time periods after the current time period. The precise aggregated delivery computing resource requirements include the configuration and quantity of core computing modules to meet the order processing needs of each predicted time period.

3. The high-load middleware system architecture method for same-city delivery services with tens of millions of users as described in claim 1, characterized in that, The global resource coordinator receives precise aggregated and distributed computing resource demands, generates resource preheating instructions for scheduling the dynamic redundancy pool, and thus prepares the computing state of the business computing cluster in advance before each forecast period, reducing the overhead of cold start computing and the central cache access latency during peak periods. These steps include: The global resource coordinator receives the precise aggregated delivery computing resource requirements for each forecast period and generates resource preheating instructions based on these requirements. By issuing resource preheating instructions for each prediction period, the currently idle core computing modules in the dynamic redundancy pool that meet the computing resource requirements are dynamically organized into a preheating computing group. Before the forecast period arrives, pre-computation tasks are performed through a pre-computation group to preload and process the geographic grid, delivery routes, and merchant hotspot data required for the forecast period, and generate pre-computation results. The preheating results are preheated to the distributed cache module corresponding to the business computing cluster, so that the computing state of the business computing cluster can be prepared in advance before the predicted period arrives, thereby reducing the overhead of cold start computing and the central cache access latency during peak periods.

4. The high-load middleware system architecture method for same-city delivery business with tens of millions of users as described in claim 1, characterized in that, The order collector continuously captures new order streams and inputs them into the business computing cluster for same-city order aggregation and delivery calculation. The step of aggregating the new order streams into an order-time fluctuation sequence according to a preset time period includes: The order collector listens to the order message queue and continuously subscribes to and captures new order messages from the order generation source in an asynchronous streaming manner, forming a new order stream; The new order stream is written into the order processing pipeline of the business computing cluster in real time, triggering the business computing cluster to perform same-city order aggregation and delivery calculation. The new order stream is cumulatively counted based on a sliding time window by using a time period aggregator set within the order collector. At the end of each preset time period, the time period aggregator outputs the total number of orders within the time period and arranges them in chronological order to form the order-time fluctuation sequence.

5. The high-load middleware system architecture method for same-city delivery services with tens of millions of users as described in claim 1, characterized in that, The step of the global resource coordinator monitoring the processing status of each core computing module in the service computing cluster in real time and generating a resource availability signal based on the idle core computing modules includes: The global resource coordinator collects real-time performance metrics of each core computing module at a preset frequency through agent programs deployed on each core computing module in the business computing cluster. The global resource coordinator compares the received real-time performance metrics with a set of preset multi-dimensional thresholds through its built-in state decision-maker. Based on the comparison results, the state decision-maker outputs a state identifier for each core computing module; The state decision-maker encapsulates the state identifiers of all core computing modules currently marked as idle and their corresponding idle computing resource specifications into a resource availability signal, and sends it to the resource mapping unit of the global resource coordinator.

6. The high-load middleware system architecture method for same-city delivery services with tens of millions of users as described in claim 1, characterized in that, The global resource coordinator fuses resource availability signals with order-time fluctuation sequences in real time to generate load-resource correlated time-series data, establishing a mapping relationship between idle computing capacity and order load for different time periods. This includes the following steps: The global resource coordinator maintains a two-dimensional relational model with time as the horizontal axis and system resources as the vertical axis through its built-in spatiotemporal fusion unit. The spatiotemporal fusion unit receives resource availability signals aligned to timestamps and order-time fluctuation sequences. For each identical sampling time point, the spatiotemporal fusion unit uses the order aggregation volume at the current time point as the load characteristic value and the set of idle core computing modules and idle computing power at the current time point as the resource supply characteristic value. By associating load characteristics with resource supply characteristics, resource-load mapping points are generated; The resource-load mapping points of multiple consecutive time points are combined and serialized in chronological order to form the load-resource associated time series data. This data is used to dynamically represent the coupling relationship and evolution trend between system resource supply and business load in historical and current time periods.

7. The high-load middleware system architecture method for same-city delivery business with tens of millions of users as described in claim 6, characterized in that, The resource-load mapping points are dynamically adjusted using dynamic weight values ​​to indicate the importance of the resource-load mapping points in subsequent prediction model training.

8. The high-load middleware system architecture method for same-city delivery business with tens of millions of users as described in claim 6, characterized in that, After the step of associating load characteristic values ​​with resource supply characteristic values ​​to generate resource-load mapping points, the method further includes: The spatiotemporal fusion unit detects whether the current time point has reached the order load mutation point by using resource-load mapping points; If so, increase the sampling frequency before and after the current time point based on the order load mutation amount to generate a higher time resolution mapping point sequence for accurately describing the mutation process.

9. The high-load middleware system architecture method for same-city delivery business with tens of millions of users as described in claim 4, characterized in that, The step of writing new order streams into the order processing pipeline of the business computing cluster in real time, triggering the business computing cluster to perform same-city order aggregation and delivery calculations, includes: Construct a real-time computing module with multi-level processing units; The new order stream is written into the order partitioning unit of the real-time computing module. The order partitioning unit allocates the order to the corresponding geographic computing unit in real time based on the order's pickup address geocode and delivery address geocode. The geographic computing unit corresponds one-to-one with the geographic grid area pre-divided in the city. The geographic computing unit performs spatiotemporal density-based clustering analysis on the influx of orders within its geographic grid. The clustered batches after cluster analysis are pushed to the parallel path optimization unit to generate delivery trajectories; Submit delivery routes and lock delivery resources to complete the aggregated delivery calculation process from new order inflow to delivery plan output.

10. The high-load middleware system architecture method for same-city delivery business with tens of millions of users as described in claim 9, characterized in that, The geographic computing unit performs a spatiotemporal density-based clustering analysis on the orders flowing into its geographic grid, including: The geographic computing unit uses a dynamic parameter clustering algorithm to handle the influx of orders within its geographic grid. The neighborhood threshold is automatically scaled based on the order inflow rate and distribution density to adaptively generate the optimal aggregate batch.

Images