A timing class information data dynamic regularizing cooperative scheduling processing method

Abstract

This invention relates to the field of computer data processing technology and discloses a dynamic regularization and collaborative scheduling method for time-series information data. The method accesses asynchronous multi-source time-series data streams, initializes a regularization pool based on a global tracking identifier, and sets a basic survival time window. It scans the regularization pool to calculate the associated state feedback factor and collects execution layer load data to calculate the resource water level. When the feedback factor reaches the high readiness state locking threshold and the resource water level is below the congestion threshold, the regularization pool is locked, a dedicated thread is allocated, and a data block access descriptor is generated for reading and computation. If the regularization pool approaches the time window and the data is not fully ready, the execution state dimensionality reduction or time window compensation operation is dynamically selected based on the resource water level status. This invention avoids memory accumulation and effective data truncation problems caused by fixed time windows by establishing a joint determination mechanism of data readiness and underlying resources, thereby improving the utilization and throughput of computing resources in concurrent scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, specifically to a method for dynamic regularization and collaborative scheduling of time-series information data. Background Technology

[0002] In application scenarios such as e-commerce for office supplies, centralized procurement transactions by government and enterprises, and the management of office equipment and consumables circulation, the system typically needs to access asynchronous time-series data streams from multiple business subsystems, including order systems, payment systems, inventory systems, warehousing systems, logistics systems, and invoicing notification systems. Due to differences in data generation timing, interface response speed, network transmission paths, and upstream processing queues among different business subsystems, the status data of various dimensions under the same business object often cannot arrive at the computing node synchronously. Therefore, before performing subsequent calculations such as order fulfillment, inventory locking, warehouse sorting, logistics updates, or status archiving, it is necessary to first organize, temporarily store, merge, and align the discretely arriving multi-source time-series status data.

[0003] Existing technologies typically employ a fixed time window approach for data waiting and merging when processing the aforementioned asynchronous time-series data. While simple to implement, this method struggles to adapt to the differences in state links and network latency fluctuations among different business objects within an office supplies trading platform. For example, for multi-SKU purchases, cross-warehouse inventory locks, or bulk purchase orders from government and enterprises, the arrival times of order status, payment status, inventory status, warehousing status, and logistics status may vary significantly. If the time window is set too short, some late-arriving state data may be truncated or discarded before arrival, affecting the integrity of subsequent business calculations; if the time window is set too long, a large amount of incompletely aligned local state data will remain in memory for an extended period, increasing the risk of consolidation pool accumulation and memory overflow.

[0004] Meanwhile, traditional data shaping logic is often independent of the execution layer's runtime environment, failing to incorporate underlying resource load states such as thread pool usage and memory consumption into shaping and delivery decisions. When execution layer thread resources are scarce or memory is under pressure, the shaping end still delivers data downstream according to the predetermined time window, which may exacerbate task queuing and concurrent scheduling conflicts. Conversely, when system resources are relatively abundant and the core business state is ready for execution, the system mechanically waits for the entire time window to end, resulting in unnecessary idle waiting time and reducing computing resource utilization.

[0005] Therefore, existing processing methods lack a dynamic coordination mechanism between data readiness and underlying system resource load, making it difficult to simultaneously ensure state data integrity, memory usage control, task execution latency, and system throughput efficiency during multi-source asynchronous time-series data processing. Summary of the Invention

[0006] The technical problem addressed by this invention is that in existing e-commerce of office supplies, government and enterprise centralized procurement transactions, and related online data processing scenarios, the time-series status data generated by multi-source business subsystems exhibits asynchronous arrival, delayed callbacks, and out-of-order writing. Traditional methods use fixed time windows for status merging, which easily leads to the truncation of effective late-arriving data or incomplete data occupying memory for extended periods. Furthermore, they fail to coordinate scheduling with the load status of underlying resources such as the execution layer thread pool and memory, resulting in high processing latency, severe task queuing, increased concurrency conflicts, and reduced system resource utilization.

[0007] To address the above problems, the present invention provides the following technical solution: The first aspect of this invention provides a method for dynamic regularization and collaborative scheduling of time-series information data, comprising: An asynchronous multi-source time-series data stream is accessed, and a global tracing identifier and business attribute tag are extracted from it. A corresponding regularization pool is initialized based on the global tracing identifier, and a basic survival time window is set for the regularization pool. The regularization pool is a logically contiguous data storage area applied for with the global tracing identifier as the primary key index, used to temporarily store asynchronous time-series data belonging to the same global tracing identifier. The regularization pool is scanned according to a preset scanning cycle to determine the completeness of the arrival of the associated state data, and to calculate the associated state feedback factor used to quantify the readiness of the current regularization pool data. Collect hardware or runtime environment load data of the execution layer and calculate the resource level line to characterize the current computing resource pressure; When the associated state feedback factor is detected to have reached the high ready state locking threshold but not the full execution threshold, and the resource level is lower than the congestion threshold, the regularization pool is switched to the locked state, a dedicated thread is allocated, and a corresponding data block access descriptor is generated; the congestion threshold is a critical value of the resource level used to compare with the resource level to determine whether the execution layer has entered a congested back pressure state. When the associated state feedback factor reaches the full execution threshold, the dedicated thread reads the data in the regularization pool and performs calculations based on the data block access descriptor. When the elapsed time of the consolidation pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, the execution state dimensionality reduction or time window compensation operation is selected based on the relationship between the current resource level and the congestion threshold. The elapsed time of the consolidation pool is determined by the time difference between the creation timestamp of the consolidation pool and the current system clock. When the remaining time between the basic survival time window and the elapsed time is less than or equal to the preset tolerance time threshold, it is determined that the elapsed time of the consolidation pool is approaching the basic survival time window.

[0008] In a preferred embodiment, the process of accessing asynchronous multi-source time-series data streams, extracting global tracing identifiers and service attribute tags from them, initializing a corresponding regularization pool based on the global tracing identifiers, and setting a basic liveness time window for the regularization pool includes: The asynchronous multi-source time-series data stream is parsed to extract the global tracking identifier and the service attribute tag; The business attribute tags are passed as input parameters to a preset business classification mapping function to obtain the corresponding business weight coefficients; wherein, the business classification mapping function is a preset mapping rule that takes the business scale field, processing level field, and state link complexity field in the business attribute tags as input and outputs the discrete score value of the business classification; the business weight coefficients are obtained by normalizing the discrete score value of the business classification. The global tracking identifier is used as the primary key index to apply for a logically contiguous data storage area to construct the regularization pool; The basic survival time window and initial memory capacity quota of the consolidation pool are calculated and set based on the business weight coefficient. The basic survival time window is determined based on the minimum survival time threshold, the maximum time scaling range, and the business weight coefficient. The minimum survival time threshold is the minimum waiting time that the consolidation pool must retain in memory after its creation. The maximum time scaling range is determined based on the difference between the maximum survival time threshold and the minimum survival time threshold. The maximum survival time threshold is the longest time that the consolidation pool is allowed to remain in a waiting state in memory.

[0009] In a preferred embodiment, the step of scanning the regularization pool according to a preset scanning cycle to determine the completeness of the arrival of associated state data and calculating an associated state feedback factor used to quantify the readiness of the current regularization pool data includes: Parse the type identifier of the data record in the regularization pool and match it with the state dimension in the baseline state set; Based on the matching results and internal verification results, maintain a state arrival indicator variable for each state dimension; Extract the inherent attribute weights of each state dimension in the baseline state set, and determine the associated state feedback factor based on the inherent attribute weights and the corresponding state arrival indicator variables.

[0010] In a preferred embodiment, the step of collecting hardware or runtime environment load data from the execution layer and calculating a resource level to characterize the current computing resource pressure includes: The number of threads in the thread pool that are in an unavailable state in the execution layer is counted, and the current thread occupancy rate is determined by combining the system's preset maximum allowable thread pool capacity parameter. Collect and obtain the runtime memory usage and the consolidation pool memory pool usage to determine the current memory consumption rate; Based on preset thread-weighted smoothing coefficients and memory-weighted smoothing coefficients, the thread occupancy rate and the memory consumption rate are weighted and merged to generate the resource water level; wherein, the sum of the thread-weighted smoothing coefficient and the memory-weighted smoothing coefficient is 1, and both are preset according to the hardware configuration of the computing node, the task type, and the historical load characteristics.

[0011] In a preferred embodiment, generating the corresponding data block access descriptor includes: The memory management module's access descriptor generation interface is called to generate the data block access descriptor, and the data block access descriptor is written into the reserved execution context of the dedicated thread; When the regularization layer and the execution layer are located in the same process address space, the generated data block access descriptor is a reference handle pointing to the regularization pool logical memory block; When the regularization layer and the execution layer are located in different processes or on different physical nodes, the generated data block access descriptor includes at least the regularization pool identifier, data block offset, data length, version number, and access token.

[0012] In a preferred embodiment, when the associated state feedback factor reaches the full execution threshold and all strongly dependent states in the baseline state set have been reached, the dedicated thread reads data from the regularization pool and performs calculations based on the data block access descriptor, including: The dedicated thread is awakened, and a consistency check is performed between the version number in the data block access descriptor and the current version number in the metadata area of ​​the regularization pool header. If the verification is successful, increment the reference count field of the regularization pool by one, locate the target data block according to the data block access descriptor, and read and execute the business operation logic. After the business operation logic is completed, the reference count field is decremented by one, the data block access descriptor in the execution context is cleared, and a release instruction is sent to reclaim the memory resources occupied by the consolidation pool.

[0013] In a preferred embodiment, when the elapsed time of the regularization pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, and the resource water level is greater than or equal to the congestion threshold, the selected execution state dimensionality reduction operation includes: Based on the state template identifier recorded in the header metadata area of ​​the regularization pool, the state template corresponding to the state template identifier is read, and the state dependency level corresponding to the missing state dimension is queried in the state template. The state template is a pre-configured state dependency level for each state dimension in the baseline state set. The state dependency level includes strong dependency state, compensable state, and weak dependency state. The strong dependency state is a state that affects the correctness of the core calculation result after being missing. The compensable state is a state that can be corrected by compensation calculation after the arrival of late data after being missing. The weak dependency state is a state that affects the auxiliary notification, trajectory display, or non-core statistical results after being missing. If the missing state dimension belongs to a compensable state or a weakly dependent state, a state missing placeholder is written to the missing position in the regularization pool, a dimensionality reduction dataset with missing state dimension identifier is generated and delivered to the computing node to trigger the dependency-free dimensionality reduction logic; the state missing placeholder is used to indicate that the corresponding state dimension has not yet been reached and is different from the real business data; the dependency-free dimensionality reduction logic skips the computing branches related to the missing state dimension and performs local business operations based on the state data that has been reached and meets the execution conditions; If the missing state dimension is a strongly dependent state, a persistent wait flag or an anomaly compensation flag is written to the header metadata area of ​​the regularization pool, and the corresponding regularization pool is transferred to the disk persistent wait queue or an anomaly compensation queue.

[0014] In a preferred embodiment, when the elapsed time of the regularization pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, and the resource level is lower than the congestion threshold, the selected execution time window compensation operation includes: Send a liveness probe command to the upstream lagging data source to confirm the network link and interface status; If a valid response is returned within the preset number of liveness detection attempts, extract the duration compensation coefficient used to characterize the current network long-tail delay distribution; Based on the currently effective survival time window threshold, the duration compensation coefficient, and the corresponding business weight coefficient, a new time window threshold is generated to dynamically extend the survival time window of the regularization pool.

[0015] In a preferred embodiment, after generating the dimensionality-reduced dataset with missing state dimension identifiers and submitting it to the computing node, the method further includes: Generate and save a compensation index, which includes at least the global tracking identifier, the dimensionality reduction output version number, the missing state dimension identifier, and the compensation validity period; When late arrival status data arrives within the compensation validity period, the compensation index is queried based on the global tracking identifier, and the late arrival status data is written into the compensation queue. The delayed state data, the dimensionality reduction output version number, and the missing state dimension identifier are read from the compensation queue. The corresponding dimensionality reduction output record is located based on the dimensionality reduction output version number. The missing state dimension identifier is used to determine the missing state placeholder or data item to be corrected in the dimensionality reduction output record corresponding to the delayed state data. The event sequence number or state data version number of the delayed state data is compared with the event sequence number or state data version number of the data processed in the same state dimension to determine whether the delayed state data has been compensated. If it has been compensated, the correction calculation is not repeated. If it has not been compensated, the delayed state data is used to replace the corresponding missing state placeholder, or the output field related to the missing state dimension in the dimensionality reduction output record is corrected to generate a compensation result.

[0016] In a preferred embodiment, after initializing the corresponding regularization pool according to the global tracking identifier, the method further includes performing capacity monitoring and expansion control on the regularization pool: The relationship between the used capacity of the regularized pool and the initial memory capacity quota is continuously monitored. When the used capacity reaches a preset expansion threshold, it is determined whether the regularized pool is in the locked state; If the regularization pool is not in the locked state, request an expansion slot and update the data block location metadata and capacity metadata of the regularization pool; If the regularization pool is already in the locked state, migration of existing data blocks is prohibited, and new status data is written to the reserved expansion slot or transferred to the overflow queue.

[0017] A second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dynamic regularization and collaborative scheduling processing method for time-series information data as described in any of the preceding claims.

[0018] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the dynamic regularization and collaborative scheduling processing method for time-series information data as described in any of the preceding claims.

[0019] This invention provides a method for dynamic regularization and collaborative scheduling of time-series information data. It has the following beneficial effects: 1. This invention changes the data processing mode that relies solely on fixed time windows for triggering. By calculating associated state feedback factors and resource water level, it constructs a joint judgment logic between data readiness and system load. The regularization layer calculates associated state feedback factors based on the arrival status of each state data under the same global tracking identifier, while the scheduling layer calculates resource water level based on thread occupancy and memory consumption. When the data arrival progress reaches the high readiness state locking condition, and the execution layer has not yet entered a congested state, the scheduler switches the corresponding regularization pool to the locked state in advance and reserves a dedicated computing thread for it. This reduces the secondary queuing time after state data reaches the execution condition, improving the overall flow efficiency of multi-source time-series data in the office supplies transaction processing workflow.

[0020] 2. This invention introduces an anomaly handling strategy that dynamically adjusts based on the congestion status of underlying resources. When the consolidation pool approaches the basic survival time window and the associated state feedback factor has not yet reached the complete execution threshold, the system determines the current pressure status of the execution layer based on the resource level. When system resources are under pressure, the consolidation layer performs differentiated processing based on the dependency level of the missing state: for compensable states or weakly dependent states, a state missing placeholder is written and a reduced-dimensional dataset is generated so that the execution layer can complete the core business calculations first and release the consolidation pool resources; for strongly dependent states, forced delivery is not performed, but the data is transferred to a disk persistence waiting queue or an anomaly compensation queue. When system resources are not congested, the system initiates a probe to the delayed data source and extends the consolidation pool survival time window by combining the duration compensation coefficient and the business weight coefficient to continue waiting for delayed state data. Thus, it can avoid the extreme cases of simple truncation or infinite waiting in traditional methods, and balance the integrity of business calculations and system throughput in scenarios with network latency or upstream interface lag.

[0021] 3. This invention achieves data access control between the regularization layer and the execution layer by generating data block access descriptors. The data block access descriptor can be represented as a logical reference handle, shared memory offset, memory mapping offset, remote access descriptor, or data transfer descriptor, depending on the deployment configuration, and includes at least a regularization pool identifier, data block offset, data length, version number, and access token. A dedicated computation thread in the execution layer locates the data to be processed in the regularization pool based on the data block access descriptor and performs secure reading using a reference counting mechanism and a version number verification mechanism. This approach reduces the overhead of repeated serialization, repeated deserialization, and cross-queue data copying between the regularization layer and the execution layer. Simultaneously, by combining regularization pool capacity monitoring, prohibiting the migration of existing data blocks under locked conditions, and spill queue control, the risk of access failure caused by data block migration or version changes in high-concurrency read scenarios can be reduced, ensuring data consistency and access security during concurrent processing. Attached Figure Description

[0022] Figure 1This is a schematic diagram of the dynamic regularization scheduling system of the present invention; Figure 2 This is a flowchart illustrating the dynamic regularization and collaborative scheduling method of the present invention. Figure 3 This is a schematic diagram illustrating the processing principle of multi-source data access and dynamic mapping in this invention; Figure 4 This is a schematic diagram illustrating the processing principle of state inspection and associated state feedback factor calculation in this invention. Figure 5 This is a schematic diagram illustrating the processing principle of resource water level acquisition and weighted calculation in this invention; Figure 6 This is a schematic diagram of the phase-change locking and data reading timing based on data block access descriptors of the present invention; Figure 7 This is a schematic diagram illustrating the processing principles of anomaly handling, state dimensionality reduction, and late state compensation in this invention.

[0023] The components are as follows: 110, Gateway Layer; 111, Parsing Engine; 120, Regularization Layer; 121, State Backend; 122, Regularization Pool; 123, Inspection Thread; 130, Scheduling Layer; 131, Scheduler; 140, Execution Layer; 141, Thread Pool; 142, Compute Node. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0025] Reference Figure 1 The dynamic regularization scheduling system provided by the present invention includes a gateway layer 110, a regularization layer 120, a scheduling layer 130, and an execution layer 140.

[0026] Gateway layer 110 is equipped with parsing engine 111. Gateway layer 110 is used to receive asynchronous data streams from external business systems and hand over the received data streams to parsing engine 111 for message parsing. Parsing engine 111 reads the asynchronous message header information and extracts a global tracking identifier used to identify the same business object, as well as business attribute tags used to characterize the business type, business scale, or processing level.

[0027] The regularization layer 120 includes a state backend 121. The state backend 121 is divided into multiple independent regularization pools 122 according to a global tracking identifier. The regularization layer 120 uses regularization pools 122 to temporarily store asynchronous time-series data belonging to the same global tracking identifier. Each regularization pool 122 corresponds to a business object to be regularized, used to hold the associated state data reported by that business object at different times from different data sources. The regularization layer 120 also includes an inspection thread 123. The inspection thread 123 scans the data arrival status within the regularization pools 122 according to a preset scanning cycle to determine the arrival completeness of the associated state data in each regularization pool 122.

[0028] The scheduling layer 130 is equipped with a scheduler 131. The scheduler 131 collects hardware or runtime environment load data such as threads and memory, and issues control commands, regularization pool state switching commands, and data block access descriptors to the corresponding nodes based on the collected results. The data block access descriptor represents the accessible location of the data to be processed in the regularization pool 122. When the regularization layer 120 and the execution layer 140 are located in the same process address space, the data block access descriptor can be a logical reference handle pointing to the regularization pool 122; when the regularization layer 120 and the execution layer 140 are located in different processes or on different physical nodes, the data block access descriptor includes at least the regularization pool identifier, data block offset, data length, version number, and access token.

[0029] The execution layer 140 is configured with a thread pool 141. The thread pool 141 is deployed in the compute node 142. The compute node 142 reads data from the regularization pool 122 and executes the corresponding computational logic according to the instructions and data block access descriptors issued by the scheduler 131. When the compute node 142 and the regularization pool 122 are in the same process address space, the compute node 142 accesses the data in the regularization pool 122 through logical reference handles; when they are in different processes or different physical nodes, the compute node 142 obtains the data blocks corresponding to the regularization pool 122 through shared memory, memory-mapped files, remote access interfaces, or data transmission channels.

[0030] It should be noted that the deployment of the regularization layer 120, scheduling layer 130, and execution layer 140 is not limited to a single form. In some embodiments, the regularization layer 120, scheduling layer 130, and execution layer 140 can be deployed on the same computing node; in other embodiments, they can be deployed on different computing nodes. When the three are deployed in the same process address space, the execution layer 140 can access the data in the regularization pool 122 through logical reference handles; when the three are deployed in different processes on the same computing node, the execution layer 140 can access the data in the regularization pool 122 through shared memory, memory-mapped files, or off-heap memory managers; when the three are deployed on different physical nodes, the data block access descriptor issued by the scheduler 131 to the execution layer 140 is used for remote access or data transfer, and the execution layer 140 obtains the corresponding data block according to the data block access descriptor and then performs computation.

[0031] Therefore, this invention establishes the data access relationship between the regularization pool 122 and the execution layer 140 through data block access descriptors under different deployment modes, without relying on passing ordinary memory pointers across physical nodes, thereby improving the feasibility and operational stability of the system in a distributed deployment environment.

[0032] Reference Figure 2 The dynamic regularization and collaborative scheduling method provided by the present invention may include the following steps: S100, accesses multi-source time-series data streams and establishes mappings.

[0033] Gateway layer 110 receives multi-source time-series data streams from different service subsystems, and parsing engine 111 performs packet parsing on the incoming data streams to extract global tracking identifiers and service attribute tags. The system calculates service weight coefficients based on the service attribute tags, and regularization layer 120 initializes regularization pool 122 in state backend 121 according to the service weight coefficients, and sets the basic liveness time window of regularization pool 122.

[0034] S200 performs status inspection and calculates associated status feedback factors.

[0035] During the process of writing multi-source time-series data into the state backend 121, the inspection thread 123 scans the regularization pool 122 according to a preset scanning cycle. The inspection thread 123 calculates the ratio of the sum of the weights of the currently arrived state data to the sum of the total weights of the baseline state based on the arrival status and corresponding state weights of each state data, and uses this ratio as the associated state feedback factor.

[0036] S300, collect resource water level line.

[0037] Scheduler 131 collects the occupancy rate of thread pool 141 in execution layer 140 and the overall system memory consumption rate, and performs weighted calculation on thread occupancy rate and memory consumption rate to obtain a resource water level line to characterize the current computing resource pressure.

[0038] S400, execute phase transition locking.

[0039] When scheduler 131 detects that the associated state feedback factor of consolidation pool 122 has reached the high ready state locking threshold but has not yet reached the full execution threshold, and the resource level is lower than the congestion threshold, scheduler 131 determines that the current execution layer 140 has not yet entered the congestion back pressure state. The congestion threshold is a critical value of the resource level used to compare with the resource level to determine whether the execution layer 140 or compute node 142 has entered the congestion back pressure state. The congestion threshold is between 0 and 1 and is preset based on the system load capacity pressure test results. In one embodiment, the congestion threshold is a floating-point number between 0.7 and 0.85. When the resource level is greater than or equal to the congestion threshold, the execution layer 140 is determined to be in the congestion back pressure state. When the resource level is lower than the congestion threshold, the execution layer 140 has not yet entered the congestion back pressure state. Scheduler 131 switches the corresponding consolidation pool 122 to the locked state and generates a data block access descriptor for the consolidation pool 122. Subsequently, scheduler 131 writes the data block access descriptor into a dedicated thread context reserved within thread pool 141.

[0040] When the correlation state feedback factor of the regularization pool 122 reaches the system's preset full execution threshold, the dedicated thread reads the data in the regularization pool 122 and performs calculations based on the data block access descriptor.

[0041] S500 performs state reduction and compensation.

[0042] When the elapsed time of the regularization pool 122 approaches the basic survival time window, and the associated state feedback factor has not reached the full execution threshold, the scheduler 131 determines the current resource level.

[0043] When the resource level exceeds the congestion threshold, the regularization layer 120 selects to perform logical dimensionality reduction, disk persistence waiting, or anomaly compensation processing based on the dependency level of the missing state. Specifically, if the missing state is a compensable state or a weakly dependent state, the regularization layer 120 writes a state missing placeholder at the missing point, generates a dimensionality reduction dataset with missing state identifiers, and outputs it to the execution layer 140; if the missing state is a strongly dependent state, the regularization layer 120 does not perform forced output, but instead transfers the corresponding regularization pool 122 to the disk persistence waiting queue or the anomaly compensation queue.

[0044] When the resource level is below the congestion threshold, the system extends the survival time window of the regularization pool 122 according to the business weight coefficient, so as to continue waiting for the arrival of the delayed state data if resources allow.

[0045] The following section provides a further explanation of the dynamic regularization and collaborative scheduling method in conjunction with the various execution stages of the system.

[0046] Reference Figure 3 The step S100 provided by this invention, namely, accessing multi-source time-series data streams and establishing mapping, may include the following steps in specific implementation: S101, Gateway layer 110 accesses asynchronous data streams sent by different business subsystems.

[0047] In some embodiments, business subsystems such as order processing, payment, inventory, warehousing, and logistics generate data packets when underlying state change events occur, and send them asynchronously to gateway layer 110 via network channels. Gateway layer 110 maintains multiple concurrent connections to receive time-series state data from different business subsystems.

[0048] For the underlying implementation methods such as network connection maintenance, data packet reception, and basic transmission control, those skilled in the art can implement them based on existing network protocol stacks, and will not be elaborated here.

[0049] S102, parsing engine 111 extracts global tracking identifiers and business attribute tags.

[0050] After the data packet enters the gateway layer 110, it is handed over to the parsing engine 111. The parsing engine 111 reads the protocol header area of ​​the data packet and reads the data sequence of a specified length according to the pre-configured field offset rules to extract the global tracking identifier.

[0051] The aforementioned header area, field length, and field offset rules can be pre-defined according to the internal data transmission protocol of the business system. The global tracking identifier is used to uniquely identify the lifecycle of a business transaction throughout the entire system, enabling data from different sources, such as order creation, payment confirmation, inventory locking, warehouse sorting, and logistics updates, to be identified as related status data under the same business object.

[0052] The parsing engine 111 also extracts business attribute tags from the data packet header. Business attribute tags are used to identify the business characteristics to which the current data belongs. They may include type fields to characterize the business scale, attribute fields to characterize the processing level, and extended fields to characterize the complexity of the state chain.

[0053] S103, Gateway Layer 110 calculates the service weight coefficient.

[0054] The system has a pre-defined and loaded business level mapping function. This function takes the business scale field, processing level field, and status link complexity field from the business attribute tags as input and outputs a discrete score value for the business level. The parsing engine 111 passes the extracted business attribute tags as input parameters to the business level mapping function.

[0055] In one implementation, the business hierarchy mapping function internally maintains a parameter lookup table, which records discrete scores corresponding to different types of fields and attribute field combinations. The discrete score is a quantified score representing the business complexity, processing priority, and state chain complexity of the current business object. A larger discrete score indicates a more complex state dependency chain, a higher processing priority, or a higher requirement for state integrity. After obtaining the corresponding discrete score from the table based on the business attribute label, the business hierarchy mapping function performs normalization on the discrete score to obtain a business weight coefficient. The value range of this business weight coefficient is distributed within the interval (0,1].

[0056] Specifically, the parameter lookup table pre-configured the correspondence between the values ​​of business attribute label fields and discrete scores; when a business attribute label includes multiple fields, the business hierarchical mapping function determines the current discrete score based on the score corresponding to each field value or the score corresponding to a combination of fields. And based on the current discrete score value The maximum allowed discrete value in the parameter comparison table The ratios between them are normalized to obtain the business weight coefficients. ,in, ,and .

[0057] Through the above processing, business data involving bulk purchases, multi-SKU transactions, cross-warehouse inventory locking, long state dependency chains, or high state integrity requirements can obtain higher business weight coefficients; while routine business data involving single small transactions or simple state dependencies obtain lower business weight coefficients. Thus, business attributes are converted into technical parameters required for subsequent regularization pool initialization, survival time window setting, and memory resource allocation.

[0058] S104, Establish an independent regularization pool 122 in regularization layer 120.

[0059] The state backend 121 receives the global tracing identifier and business weight coefficient transmitted by the gateway layer 110, and uses the global tracing identifier as the primary key index. Based on the primary key index, the state backend 121 requests a logically contiguous data storage area from the pre-allocated regularization pool memory pool, off-heap memory, shared memory, or memory-mapped area. This data storage area is constructed as a regularization pool 122 corresponding to the global tracing identifier.

[0060] In this embodiment, the regularization pool 122 includes a header metadata area, a status slot area, a status bitmap area, and a verification area.

[0061] The header metadata area is used to record the global tracking identifier, business weight coefficient, basic survival time window, current pool status, version number, locked thread identifier, and reference count field; The status slot area is used to store the corresponding status data according to the status dimension in the baseline status set; The state bitmap area is used to record whether each state dimension has been reached; The verification area is used to record the data length, verification value, and write completion marker for each status slot.

[0062] Through the above structure, the state backend 121 can perform slotted temporary storage and state marking of multi-source asynchronous state data under the same business object within the same regularization pool 122.

[0063] If the current available capacity of the regularized pool memory pool, off-heap memory, or shared memory region is insufficient to allocate the aforementioned logically contiguous data storage region, the status backend 121 does not directly discard the data corresponding to the global tracking identifier, but instead temporarily stores the global tracking identifier and its associated data in the disk spill queue and waits for subsequent retries.

[0064] When memory resources become available again, the state backend 121 re-establishes the regularization pool 122 based on the global tracking identifier in the spill queue and writes the temporary data back to the corresponding state slot. This avoids the direct loss of associated state data due to momentary memory shortage.

[0065] The regularization pool 122 serves as a temporary memory storage area, used to receive associated status data with the same global tracking identifier that arrive at different time points later.

[0066] S105, the regularization layer 120 dynamically sets the basic survival time window of the regularization pool 122.

[0067] To reduce premature truncation or excessive waiting caused by static time windows, the regularization layer 120 calculates the initial lifespan period of the regularization pool 122 using business weight coefficients. For data with long state dependency links, high state integrity requirements, or high correlation complexity, the system provides a longer waiting time; for regular data with short state links, the system adopts a shorter timeout cleanup rhythm to reduce the long-term residence pressure on the regularization pool 122.

[0068] The system calculates the base survival time window using the following formula: ; in, This indicates the basic survival time window allocated by the system to the current regularization pool 122; This indicates the minimum survival time threshold for jitter prevention set by the system. The minimum survival time threshold for jitter prevention is the minimum waiting time that the regularization pool 122 must retain in memory after its creation. This is used to prevent the regularization pool 122 from entering timeout processing prematurely due to short-term network jitter or instantaneous interface delay. This represents the maximum time scaling range set by the system, and its value is equal to the difference between the maximum survival time threshold and the minimum survival time threshold allowed by the system. The maximum survival time threshold is the longest time that the consolidation pool 122 is allowed to remain in a waiting state in memory. It is used to limit the maximum residence time of the consolidation pool 122 and avoid incomplete data occupying memory resources for a long time. This represents the business weight coefficient calculated in step S103.

[0069] The maximum survival time threshold can be determined based on the average arrival time of regular business status in historical transaction logs. It can be configured by combining the system's maximum allowed network wait time, upstream interface response latency, or the maximum capacity limit of the status backend. These parameters can be adjusted in the configuration file according to actual business scenarios. Specifically, the system configures the minimum and maximum survival time thresholds in the configuration file according to business attribute tags or status template identifiers. For each type of business attribute tag or status template, the system statistically analyzes the time distribution from creation of the business object in the regularization pool 122 to the arrival of associated status data based on historical transaction logs. The average arrival time of regular business or core status data is configured as the corresponding minimum survival time threshold, or used as a reference value for determining the minimum survival time threshold. The system determines the corresponding maximum survival time threshold based on the upstream interface response latency, the maximum allowed network wait time, and the maximum capacity limit of the status backend for that type of business object. Business attribute tags with longer status dependency links, higher processing levels, or higher status integrity requirements are configured with larger maximum survival time thresholds, while business attribute tags with simpler status dependencies or lower processing levels are configured with smaller maximum survival time thresholds. After configuration, the system reads the corresponding minimum survival time threshold and maximum survival time threshold based on the business attribute tag or status template identifier of the current business object, and calculates the maximum time scaling range.

[0070] In addition, the regularization layer 120 synchronously calculates the initial memory capacity quota of the regularization pool 122. The status backend 121 reads the system's preset baseline memory block size in bytes and determines the initial capacity of the regularization pool 122 in combination with the business weight coefficient.

[0071] For example, the system has a pre-defined amplification variable. The state backend 121 multiplies the amplification variable by the business weight coefficient and then adds it to the base memory block size to obtain the initial memory capacity quota of the corresponding regularization pool 122.

[0072] In this way, the survival time and memory capacity of the regularization pool 122 are both related to the business weight coefficient, so that business objects with complex state dependencies can obtain more waiting time and cache space in the initial stage, while regular business objects will not occupy too much underlying storage resources.

[0073] S106, Performs monitoring and expansion control of the structured pool capacity.

[0074] When writing status data to the consolidation pool 122, the status backend 121 continuously monitors the relationship between the used capacity of the consolidation pool 122 and the initial memory capacity quota. When the used capacity reaches a preset expansion threshold, the status backend 121 determines whether the consolidation pool 122 is in a locked state.

[0075] If the consolidation pool 122 is not in a locked state, the state backend 121 requests an expansion slot from the consolidation pool memory pool and updates the data block location metadata, capacity metadata, and version number fields of the consolidation pool 122 so that a data block access descriptor can be generated based on the updated metadata.

[0076] If the regularization pool 122 is already locked, the state backend 121 prohibits the migration of existing data blocks and only allows new state data to be written to the reserved extension slot or transferred to the overflow queue, so as to avoid the data block access descriptor that has been injected with a dedicated thread becoming invalid due to data block migration.

[0077] When the state backend 121 is unable to complete the expansion, and continued waiting may lead to an abnormal rise in the memory level, the state backend 121 transfers the consolidation pool 122 to the exception handling path. Under this path, the system can choose to perform logical dimensionality reduction, disk persistence waiting, or exception compensation processing based on the dependency level of the missing state.

[0078] Reference Figure 4 The step S200 provided by the present invention, namely performing state inspection and calculating the associated state feedback factor, may include the following steps in a specific implementation: S201, the set of loading baseline states for the 120-level regularization layer.

[0079] Before loading the baseline state set in the regularization layer 120, the regularization layer 120 first selects a state template that matches the current business transaction type from the preset state template library based on the business attribute tags extracted in step S102, and writes the template identifier of the selected state template into the header metadata area of ​​the regularization pool 122.

[0080] For different business transaction types, the system predefines the state dimensions required to complete a business flow. For example, in order-related businesses, this state dimension may include order creation status, payment confirmation status, inventory lock status, warehouse picking status, and logistics update status. The system summarizes these state dimensions into a baseline state set. ; in, Represents the set of baseline states The total number of independent state flows included.

[0081] Furthermore, the system provides an independent state flow for each of the reference state sets S. Configure inherent attribute weights Configure the corresponding state dependency level. Attribute weights. The state dependency level is used to characterize the degree of influence of the state dimension in the subsequent execution layer 140 operation logic; the state dependency level is used to characterize the type of impact on the business calculation results after the state dimension is missing.

[0082] In some embodiments, state dependency levels include strongly dependent states, compensable states, and weakly dependent states. A strongly dependent state indicates that the absence of this state dimension will affect the correctness of the core calculation results; a compensable state indicates that the absence of this state dimension can be corrected through compensation calculations after late data arrives; and a weakly dependent state indicates that the absence of this state dimension mainly affects auxiliary notifications, trajectory displays, or non-core statistical results.

[0083] For example, states involving fund settlement, inventory locking, or fulfillment confirmation can be configured as strongly dependent states and assigned higher attribute weights; states involving logistics trajectory display, message notification, or ancillary statistics can be configured as weakly dependent states and assigned lower attribute weights.

[0084] To facilitate subsequent convergence calculations, the system can assign attribute weights to each state flow. Perform normalization to ensure it meets the following constraints: ; The above state templates and attribute weights The state dependency level can be preloaded through the system configuration file or adjusted according to the business system's operating strategy.

[0085] S202, Inspection thread 123 performs periodic scanning.

[0086] Internally, the regularization layer 120 has an inspection thread 123. The inspection thread 123 triggers a scan task according to a preset scan cycle to check the arrival status of state data in each regularization pool 122. The underlying implementation of thread scheduling, timer triggering, and scan task wake-up can be implemented by those skilled in the art using existing concurrent programming techniques, and will not be described in detail here.

[0087] When each scan task is triggered, the inspection thread 123 reads the regularization pool 122 in the status backend 121. When the number of regularization pools 122 is small, the inspection thread 123 can traverse all regularization pools 122; when the number of regularization pools 122 is large, the inspection thread 123 can use an incremental scanning method to scan only the active regularization pools that have experienced data write events in the current period, so as to reduce the computational overhead caused by full traversal.

[0088] Inspection thread 123 reads the data records cached in the regularization pool 122 and parses the type identifier in each data record. Then, inspection thread 123 matches the type identifier with the state dimension in the baseline state set S to determine the state dimension corresponding to the data record.

[0089] Based on the matching results, inspection thread 123 maintains a state arrival indicator variable for each state dimension in the baseline state set S. For the first In each state dimension, when inspection thread 123 determines that the corresponding state data has been successfully written to the regularization pool 122, and the state data passes format verification, length verification, or verification value verification, the state reaches the indicator variable. Assign a value of 1; when the corresponding status data has not yet arrived, or has arrived but failed the internal check, the status arrival indicator variable will be set to 1. The value is assigned to 0.

[0090] Thus, the physical arrival status of each state data in the regularization pool 122 is converted into a state arrival marker that can participate in subsequent calculations.

[0091] S203, Inspection thread 123 calculates the associated state feedback factor.

[0092] To quantify the overall readiness of the associated state data in the current consolidation pool 122, the inspection thread 123 determines the state arrival indicator variable obtained in step S202. and the attribute weights configured in step S201 Calculate the associated state feedback factor .

[0093] Specifically, inspection threads 123 extract the baseline state set. Attribute weights of each state dimension and weight each attribute The corresponding state arrival indicator variable The products are multiplied and then summed. The system then divides the sum by the base state set. The sum of the weights of all state-dimensional attributes yields the associated state feedback factor. .

[0094] The formula for calculating the correlation state feedback factor is as follows: ; in, This represents the feedback factor of the current associated state of the regularization pool 122; Represents the set of baseline states The total number of independent state flows in the middle; Indicates the first The inherent attribute weights of each state dimension; Indicates the first Each state dimension corresponds to a state arrival indicator variable; Represents the set of baseline states The Middle The inherent attribute weights of each state dimension.

[0095] Because the state reaches the indicator variable The value is 0 when the corresponding state has not been reached or the verification has failed. Therefore, missing states will not participate in the effective weight accumulation on the molecular side; states that have been reached and verified will participate in the readiness calculation according to their attribute weights.

[0096] Through the above calculations, the system converts the arrival status of multiple discrete state data in the regularization pool 122 into a continuous quantification index. (Associated state feedback factor) The value ranges from 0 to 1; the closer the value is to 1, the more complete the key state data in the regularization pool 122 is, and the closer the regularization pool 122 is to a state that can be read and processed by the subsequent execution layer 140.

[0097] Inspection thread 123 will calculate the associated state feedback factor Write to the shared memory region and synchronously update the version number field or write completion flag field of the corresponding regularization pool 122. Scheduler 131 subsequently reads the associated state feedback factor. Simultaneously, the scheduler 131 reads the version number field of the regularization pool 122. If the version number before and after the read is the same, the read result is considered valid; if the version number changes, the scheduler 131 rereads the associated state feedback factor. .

[0098] By using the above version verification method, the risk of concurrent conflict between the inspection thread 123 writing the associated state feedback factor and the scheduler 131 reading the factor can be reduced, and the scheduler 131 can be prevented from making scheduling decisions based on expired or incomplete state data.

[0099] Reference Figure 5 The step S300 provided by this invention, namely collecting the resource water level, may include the following steps in specific implementation: S301, Scheduler 131 collects the thread occupancy status of thread pool 141.

[0100] Scheduler 131 can obtain the thread occupancy status of thread pool 141 in execution layer 140 by calling operating system performance indicator interface, runtime environment management component or process resource probe.

[0101] In some embodiments, the scheduler 131 counts the number of threads in the thread pool 141 that are in an unavailable state. Unavailable states include those that are executing computational tasks, those that are locked by the scheduler 131, and those that have been reserved but have not yet started executing tasks.

[0102] Meanwhile, scheduler 131 reads the maximum allowed capacity parameter of the thread pool preset by the system. This maximum allowed capacity parameter of the thread pool is a positive integer greater than zero, used to limit the maximum number of threads that thread pool 141 can simultaneously support in the current computing node 142.

[0103] Scheduler 131 divides the number of threads in an unavailable state by the maximum allowed capacity parameter of the thread pool to obtain the current thread occupancy rate. The value of this thread occupancy rate ranges from 0 to 1.

[0104] By including running threads, locked threads, and reserved threads in the statistics, the scheduler 131 can reflect the actual availability of the thread pool 141, avoiding underestimating thread resource consumption by only counting threads that are currently executing tasks.

[0105] For the acquisition of underlying metrics, deployment of process resource probes, and methods for thread status statistics, those skilled in the art can use existing system load monitoring technologies, which will not be elaborated here.

[0106] S302, Scheduler 131 collects the overall memory consumption index of the system.

[0107] During the continuous writing of multi-source time-series data into the consolidation pool 122, the remaining memory directly affects the data retention time and the timing of exception handling. To this end, the scheduler 131 collects memory metrics related to the current deployment configuration, including the total runtime memory of the compute node 142 where the execution layer 140 is located, the total capacity of the consolidation pool memory pool, and the available capacity of off-heap memory or shared memory regions participating in data access under the current deployment configuration.

[0108] Scheduler 131 further reads the size of runtime memory bytes currently allocated or occupied, the size of allocated bytes in the consolidation pool memory pool, and the usage of off-heap memory or shared memory participating in the current deployment configuration. Based on this data, the system calculates the runtime memory usage rate and the consolidation pool memory pool usage rate, and then weights and combines them to obtain the current memory consumption rate.

[0109] The consolidation pool memory pool utilization rate is calculated based on the ratio between the allocated consolidation pool capacity in the state backend 121 and the total consolidation pool memory pool capacity. The memory consumption rate is limited to a range of 0 to 1.

[0110] If the total runtime memory, total capacity of the regularized pool memory pool, off-heap memory capacity, or shared memory region capacity of the current deployment are detected as zero, or if the corresponding indicators cannot be obtained normally, the system returns the highest memory consumption rate according to the preset security protection policy, so that the subsequent scheduling logic can be processed as a resource shortage state and trigger the corresponding resource shortage alarm process.

[0111] S303, scheduler 131 sets the weighted smoothing coefficient.

[0112] Since different computing nodes 142 carry different types of tasks, their main resource bottlenecks may also be different. The scheduler 131 sets a weighted smoothing coefficient based on the hardware configuration, task type, or historical load characteristics of the computing nodes 142.

[0113] For compute node 142, which focuses on data-intensive computing, the load pressure is usually more reflected in the CPU scheduling resources and thread execution resources. The system can add a thread-weighted smoothing coefficient. Set a higher value.

[0114] For compute nodes 142 that focus on multi-source data temporary storage, large window state maintenance, or long-term resident status in the consolidation pool 122, their load pressure is usually more reflected in the occupancy of the consolidation pool memory pool, off-heap memory, or shared memory regions. The system can use a memory-weighted smoothing coefficient. Set a higher value.

[0115] To ensure that thread occupancy and memory consumption are calculated on the same scale, the system sets a thread-weighted smoothing coefficient in the configuration file. With memory-weighted smoothing coefficient The sum is 1. Wherein, the thread-weighted smoothing coefficient... The memory-weighted smoothing coefficient is used to characterize the weight of thread occupancy in thread resource pressure calculation. This is used to characterize the weight of memory consumption rate in resource pressure calculation. The system presets this based on the hardware configuration, task type, and historical load characteristics of compute node 142. and The value of ; when historical load characteristics indicate that compute node 142 is mainly limited by thread execution resources, Set to greater than The value; when historical load characteristics indicate that compute node 142 is primarily limited by the consolidation pool memory pool, off-heap memory, or shared memory usage, Set to greater than The value; when the impact of thread execution resources and memory resources on the load of compute node 142 is comparable, will and Set to the same or similar value.

[0116] In one implementation, the system determines the degree of thread bottleneck based on the number of times or the average number of times the thread occupancy rate exceeds the thread pressure threshold within a preset statistical period, and determines the degree of memory bottleneck based on the number of times or the average number of times the memory consumption rate exceeds the memory pressure threshold within a preset statistical period; if the degree of thread bottleneck is higher than the degree of memory bottleneck, then a setting is made. The value range is 0.6 to 0.8, and is set as follows: =1- If the memory bottleneck is greater than the thread bottleneck, then set... The value range is 0.6 to 0.8, and is set as follows: =1- If the difference between the thread bottleneck level and the memory bottleneck level is less than a preset difference threshold, then set... = =0.5.

[0117] S304, Scheduler 131 generates resource water level lines.

[0118] Scheduler 131 calculates the current resource level based on the thread occupancy rate obtained in step S301 and the memory consumption rate obtained in step S302. This resource level is used to characterize the overall back pressure currently borne by computing node 142 and serves as one of the input parameters for subsequent phase transition locking, anomaly dimensionality reduction, and time window compensation judgments.

[0119] Scheduler 131 calculates the resource water level according to the following formula: ; in, This represents the dynamic resource water level calculated by scheduler 131; Indicates thread occupancy; Indicates memory consumption rate; Indicates the thread-weighted smoothing coefficient; Represents the memory-weighted smoothing coefficient, and satisfies .

[0120] because , , and All of them are within the range of 0 to 1, therefore, The value range is also between 0 and 1. The closer it is to 1, the higher the resource pressure on the current computing node 142; The closer it is to 0, the more schedulable the current computing node 142 still has.

[0121] Through the above calculations, scheduler 131 maps thread execution resources and memory storage resources into a unified resource level indicator. Compared to judging based on a single CPU, thread, or memory indicator, this resource level indicator can more stably reflect the overall stress state of computing node 142, thereby reducing the impact of instantaneous fluctuations in a single indicator on scheduling decisions.

[0122] Scheduler 131 can cyclically execute steps S301 to S304 according to a preset sampling period to maintain the resource level. Update.

[0123] Reference Figure 6 Before performing phase transition locking, the header metadata area of ​​regularization pool 122 sets a pool status field to identify the current processing stage of the regularization pool 122. The pool status field includes ACTIVE, LOCKED, READY, PROCESSING, DEGRADED, and RELEASED.

[0124] Among them, ACTIVE indicates that the consolidation pool 122 is in a normal write and inspection state; LOCKED indicates that the consolidation pool 122 has been locked by the scheduler 131 and is waiting for the missing state to arrive; READY indicates that the state data in the consolidation pool 122 has met the execution conditions; PROCESSING indicates that the execution layer 140 is reading and processing the data of the consolidation pool 122; DEGRADED indicates that the consolidation pool 122 has entered the state dimensionality reduction, persistence waiting or anomaly compensation processing path; RELEASED indicates that the memory resources occupied by the consolidation pool 122 have been released.

[0125] Scheduler 131 updates the pool state of regularization pool 122 through atomic state switching operations to avoid concurrent conflict operations on the same regularization pool 122 by inspection thread 123, scheduler 131, state backend 121 and dedicated computing thread.

[0126] Step S400 provided by this invention, namely the specific implementation process of performing phase transition locking, may include the following steps: S401, Scheduler 131 performs trigger determination based on bidirectional threshold.

[0127] In typical streaming architectures, data only enters the task queue and requests computing resources after it meets the execution conditions. In high-concurrency scenarios, this approach may result in data needing to queue again even after it is ready. This invention uses a phase-change locking mechanism to reserve computing resources for the regularization pool 122 in advance when it is about to meet the execution conditions and the system resources have not yet become congested.

[0128] Specifically, the system loads a high-readiness state lock threshold before running or through a configuration file. Full execution threshold and system congestion threshold .

[0129] Among them, the high-readiness state locking threshold The associated state data in the regularization pool 122 is close to the execution condition, and its value range can be set to a floating-point number between 0.8 and 0.95; the complete execution threshold. This is used to characterize that the regularization pool 122 has met the minimum state integrity requirements for reading and processing by the execution layer 140, and satisfies... .

[0130] when When the value is 1, it indicates that all state data in the baseline state set has been reached; when A value less than 1 indicates that although consolidation pool 122 has not obtained all state data, it has met the system's allowed execution conditions. System congestion threshold. The value can be set based on the system load capacity stress test results, for example, it can be set to a floating-point number between 0.7 and 0.85 to characterize that computing node 142 has not yet entered the saturation queuing state.

[0131] Scheduler 131 periodically reads the associated state feedback factors of regularization pool 122 And the resource water level of the current computing node 142. When satisfied and At that time, scheduler 131 determines that the current regularization pool 122 is in a highly ready and resource-capable state, and triggers the phase change locking mechanism.

[0132] when When all the strongly dependent states in the baseline state set have been reached, the scheduler 131 no longer performs phase transition locking, but directly switches the corresponding regularization pool 122 to the READY state, or notifies the execution layer 140 to enter the execution delivery process.

[0133] S402, scheduler 131 issues a pre-allocation control instruction to reserve a dedicated thread.

[0134] After the triggering condition of step S401 is met, the scheduler 131 first performs an atomic state switch on the regularization pool 122, switching the regularization pool 122 from the ACTIVE state to the LOCKED state. If the state switch fails, it means that the regularization pool 122 has been occupied by other processing paths, and the scheduler 131 cancels this phase transition locking operation.

[0135] After the state transition is successful, the scheduler 131 sends a pre-allocation control instruction to the thread pool 141 of the execution layer 140. After receiving the pre-allocation control instruction, the thread pool 141 first checks whether the number of dedicated threads currently in a locked state exceeds the preset maximum locked thread ratio.

[0136] If the preset maximum locked thread ratio has been exceeded, thread pool 141 rejects the current pre-allocation control instruction; scheduler 131 rolls back the corresponding regularization pool 122 from the LOCKED state to the ACTIVE state, and adds the global tracking identifier of the regularization pool 122 to the lock candidate queue. Regularization pools 122 in the lock candidate queue are still allowed to receive subsequent state data writes and will re-participate in phase transition locking determination in the next scheduling cycle.

[0137] If the preset maximum locked thread ratio is not exceeded, thread pool 141 allocates an available thread from the idle worker thread queue as the dedicated computing thread for the regularization pool 122.

[0138] If there are no idle worker threads in the current thread pool 141, the system cancels the current phase transition locking operation; the scheduler 131 rolls back the regularization pool 122 from the LOCKED state to the ACTIVE state, and allows the regularization pool 122 to complete execution once the full execution threshold is reached. Then it enters the regular asynchronous queuing process.

[0139] When an available thread is successfully allocated, the system marks it as a locked thread, making it the dedicated computing thread for the corresponding regularization pool 122. During the locking period, this dedicated computing thread will no longer receive other concurrent task assignments.

[0140] At the same time, scheduler 131 sets a maximum lock duration for this dedicated computing thread. If in The associated state feedback factor of the inner regularization pool 122 The full execution threshold has not been reached. Scheduler 131 removes the binding relationship between the dedicated computing thread and the regularization pool 122, restores the thread to the idle state, and rolls the regularization pool 122 back from the LOCKED state to the ACTIVE state or switches to the exception handling path.

[0141] S403, scheduler 131 generates and injects data block access descriptors.

[0142] In typical cross-level data transfer processes, data often needs to undergo processing such as task queue transfer, serialization, cross-process or cross-network copying, and deserialization, which can easily increase CPU and memory overhead. To reduce the overhead of repeated serialization, repeated deserialization, and cross-queue data copying, the scheduler 131 calls the access descriptor generation interface of the memory management module for the regularized pool 122 in the high-ready state to generate the corresponding data block access descriptor.

[0143] When the regularization layer 120 and the execution layer 140 are located in the same process address space, the data block access descriptor can be a reference handle pointing to the logical memory block of the regularization pool 122.

[0144] When the regularization layer 120 and the execution layer 140 are located in different processes or on different physical nodes, the data block access descriptor includes the regularization pool identifier, data block offset, data length, version number, and access token.

[0145] Subsequently, scheduler 131 passes the data block access descriptor to execution layer 140 and writes it into the execution context of a reserved dedicated computation thread.

[0146] Through the aforementioned data block access descriptor injection operation, the scheduler 131 establishes an access mapping relationship between the dedicated computing thread and the data blocks to be processed in the regularization pool 122 before the data formally meets the execution conditions. The execution layer 140 can then locate the target data block based on the same regularization pool identifier, the same data version, and the corresponding access token.

[0147] Within the same process address space, the above access mapping relationship is represented as a logical reference handle; in deployment scenarios involving different processes or different physical nodes, the above access mapping relationship is represented as a shared memory handle, memory mapping offset, remote access descriptor, or data transfer descriptor.

[0148] S404, a dedicated computing thread reads data and performs calculations based on the data block access descriptor.

[0149] As external asynchronous data streams continue to be received, the regularization pool 122 receives one or more missing state data, causing the inspection thread 123 to calculate the associated state feedback factor. Reaching the full execution threshold .

[0150] when When the value is 1, it indicates that all state data in the baseline state set has been reached; when When the value is less than 1, it indicates that the regularization pool 122 has met the system's allowed execution conditions.

[0151] State backend 121 detects associated state feedback factors Reaching the full execution threshold After all the strongly dependent states in the baseline state set have been reached, the regularization pool 122 is switched from the LOCKED state to the READY state, and a wake-up signal is sent to the reserved dedicated computing thread through the event bus.

[0152] After the dedicated computing thread is awakened, it first attempts to switch the regularization pool 122 from the READY state to the PROCESSING state through an atomic state transition operation. If the state transition is successful, the dedicated computing thread extracts the data block access descriptor from its own execution context.

[0153] Before reading the data, the dedicated computing thread performs a consistency check between the version number in the data block access descriptor and the current version number in the header metadata area of ​​consolidation pool 122. If they match, the dedicated computing thread increments the reference count field of consolidation pool 122 and reads the corresponding data according to the data block access descriptor.

[0154] If the version numbers are inconsistent, the dedicated computing thread requests scheduler 131 to regenerate the data block access descriptor, or refreshes the local execution context based on the updated data length and offset; the dedicated computing thread does not increment the reference count field before the new version verification is completed.

[0155] If regenerating the data block access descriptor or refreshing the local execution context fails, the dedicated computation thread clears the data block access descriptor in the execution context, rolls back the regularization pool 122 from the PROCESSING state to the READY state or switches to the exception handling path, and restores itself to the idle state.

[0156] If the smoothing pool 122 fails to switch from the READY state to the PROCESSING state, it means that the smoothing pool 122 has been occupied by another processing path. At this time, the dedicated computing thread clears the data block access descriptor in the execution context, abandons the current read operation, and returns to the idle state.

[0157] After the version verification is passed, the dedicated computing thread locates the corresponding data block based on the data block access descriptor, reads the associated state dataset that currently meets the execution conditions in the regularization pool 122, and uses the dataset to execute the business operation logic.

[0158] When the regularization layer 120 and the execution layer 140 are deployed in the same process address space or the same shared memory domain, dedicated computing threads can directly access data in the regularization pool 122 based on logical reference handles or shared memory handles, thus achieving zero-copy access in this deployment mode. When the regularization layer 120 and the execution layer 140 are deployed on different physical nodes, dedicated computing threads trigger remote reads or data transfers through data block access descriptors; in this case, the system does not limit it to zero-copy access in a physical sense, but it can still avoid repeated serialization and deserialization.

[0159] After the business logic is completed, the dedicated computing thread decrements the reference count field of the consolidation pool 122 by one, clears the data block access descriptor in the execution context, and restores it to an idle state. At the same time, the dedicated computing thread sends a release instruction to the status backend 121 to release and reclaim the memory pool resources, off-heap memory resources, or shared memory resources occupied by the consolidation pool 122.

[0160] Before releasing the consolidation pool 122, the state backend 121 reads its reference count field and pool status field. When the reference count field is zero and the pool status is PROCESSING, the state backend 121 switches the consolidation pool 122 to the RELEASED state and performs memory reclamation.

[0161] When the pool is in the READY state but has not yet entered the PROCESSING state, the state backend 121 does not release the regularization pool 122 directly, but waits for the execution layer 140 to read it; or, after the preset ready timeout period is reached, the scheduler 131 reschedules the regularization pool 122 or transfers it to the exception handling path.

[0162] When the reference count field is not zero, the state backend 121 delays the release of the consolidation pool 122 to avoid prematurely reclaiming memory resources when there are still threads holding valid data block access descriptors.

[0163] Reference Figure 7 The step S500 provided by this invention, namely performing state dimensionality reduction and compensation, may include the following steps in a specific implementation: S501, the regularization layer 120 determines the boundary critical state of the regularization pool 122.

[0164] During system operation, the regularization layer 120 obtains the creation timestamp of each regularization pool 122 and the current system clock timestamp, and determines the current elapsed time of the regularization pool 122 based on the time difference between the two. The regularization layer 120 then compares the current elapsed time with the basic survival time window determined in step S100. A comparison is made to determine whether the regularized pool 122 is close to the timeout boundary.

[0165] In this embodiment, the system sets a tolerance time threshold. This tolerance time threshold Used to achieve the basic survival time window in the regulation tank 122 Priority was given to allow time for exception handling and path switching.

[0166] When the basic survival time window The remaining time between the current elapsed time and the current elapsed time is less than or equal to the tolerance time threshold. And the current elapsed time has not exceeded the basic survival time window. At that time, the system reads the associated state feedback factor currently calculated by inspection thread 123. .

[0167] If the current elapsed time has exceeded the base survival time window The system no longer waits for the tolerance time threshold. Instead of triggering conditions, it directly reads the current associated state feedback factor. Then proceed to timeout determination.

[0168] When the associated state feedback factor The system's preset full execution threshold has not been reached. When the timeout occurs, it indicates that the associated state data in the regularization pool 122 is close to or has already timed out, and the read processing conditions of the execution layer 140 have not yet been met, and the system is preparing to enter the exception handling path.

[0169] When the associated state feedback factor The full execution threshold has been reached. When all strongly dependent states in the baseline state set have been reached, the regularization layer 120 does not enter the exception handling path, but instead switches the corresponding regularization pool 122 to the READY state, or notifies the scheduler 131 to enter the execution delivery path. If the associated state feedback factor... The full execution threshold has been reached. However, if there are still strong dependency states that have not been reached, then the regularization layer 120 will not perform a READY state switch, but will instead enter disk persistence waiting or exception compensation processing according to the missing path of the strong dependency state.

[0170] Before actually entering the exception handling path, the regularization layer 120 reads the pool status field of the regularization pool 122. If the regularization pool 122 is already in the LOCKED or PROCESSING state, it means that the regularization pool 122 is being occupied by a phase change locking process or an execution process. At this time, the regularization layer 120 suspends the exception handling path, and the scheduler 131 determines the subsequent processing based on the locking timeout mechanism or the execution completion status. If the regularization pool 122 is in the ACTIVE state, it is allowed to enter the exception handling path. If the regularization pool 122 is in the READY state, the regularization layer 120 does not repeatedly wake up the exception handling path, but the scheduler 131 continues to execute delivery or rescheduling; if the regularization pool 122 is in the DEGRADED or RELEASED state, the regularization layer 120 does not repeat the current exception handling.

[0171] Tolerance time threshold The time can be set according to the average execution cycle of the system's scheduled tasks, for example, it can be set to 50 milliseconds to 200 milliseconds.

[0172] S502, scheduler 131 performs state dimensionality reduction delivery under congestion back pressure.

[0173] After the exception handling path is triggered, scheduler 131 reads the current resource level. And compare it with the system congestion threshold. Compare them.

[0174] When the resource water level Greater than or equal to the system congestion threshold When this occurs, it indicates that the execution layer 140 and the corresponding compute node 142 are already under high load. In this state, if a large number of unmet execution conditions in the consolidation pool 122 are retained, it can easily lead to increased memory usage in the consolidation pool and exacerbate queuing for subsequent tasks.

[0175] The regularization layer 120 switches the regularization pool 122 from the ACTIVE state to the DEGRADED state through an atomic state switching operation. If the state switching fails, it means that the regularization pool 122 has entered a phase transition locking path, an execution path, or other abnormal path, and the regularization layer 120 stops the current state dimensionality reduction process.

[0176] After a successful state transition, the regularization layer 120 performs logical dimensionality reduction marking on the state dimensions that have not yet been reached in the baseline state set. Specifically, the regularization layer 120 writes a preset state missing placeholder into the state slot corresponding to the missing data in the regularization pool 122. This logical dimensionality reduction marking does not physically delete the corresponding state dimension, but records the missing state dimension identifier in the metadata area of ​​the output data, so that the execution layer 140 can identify that the state dimension has not yet been reached during subsequent processing and skip the calculation branches strongly related to the missing state.

[0177] The missing state placeholder includes a missing flag, a state dimension number, a data length field, a placeholder type field, a validation field, and a reserved field. It is used to explicitly mark missing state dimensions in the data stream. This placeholder is distinguished from actual business data by metadata identification, thereby avoiding confusion between placeholder encoding and actual business data.

[0178] The system then determines the state dependency level of the missing state dimension.

[0179] If the missing state dimension is a strongly dependent state, the system will not force the dataset to be delivered to the compute node 142. Instead, it will write a persistent waiting flag or an anomaly compensation flag in the header metadata area of ​​the regularization pool 122 and transfer the associated data of the regularization pool 122 to the disk persistent waiting queue or anomaly compensation queue.

[0180] If the missing state dimension belongs to a compensable state or a weakly dependent state, the system delivers the dimensionality-reduced dataset with state missing placeholders and missing state dimension identifiers to compute node 142. After parsing the state missing placeholders, compute node 142 triggers dependency-free dimensionality reduction logic, performing local business operations only based on the core state data that has been reached and meets the execution conditions.

[0181] After the local business operation is completed, the computing node 142 sends a release command to the state backend 121 to reclaim or mark the memory resources occupied by the regularization pool 122 for reclamation. For cases where the missing state is a compensable state, the state backend 121 generates a compensation index before releasing the regularization pool 122 so that when subsequent late state data arrives, the original dimensionality reduction output record can be located and idempotent correction calculation can be performed.

[0182] S503, the system performs time window compensation when resources are plentiful.

[0183] When scheduler 131 determines the current resource level Below the system congestion threshold This indicates that the execution layer 140 and the corresponding computing node 142 still have a certain schedulable margin. Under this condition, the system can allow a portion of the regularization pool 122 to continue residing for a period of time to improve the integrity of the associated state data.

[0184] At this point, the system sends a liveness detection command to the upstream delayed data source through gateway layer 110 to confirm the network link and interface status of the corresponding data source. For the underlying implementation of liveness detection command issuance, link status confirmation, and interface keep-alive, those skilled in the art can use existing network heartbeat detection or keep-alive mechanisms, which will not be elaborated upon here.

[0185] If the upstream delayed data source does not return a valid response within a preset number of liveness probes, the system marks the data source corresponding to the missing state as an abnormal data source, and, based on the dependency level of the missing state, transitions the regularization pool 122 into state reduction, disk persistence wait, or abnormal compensation path. Simultaneously, the system switches the pool state of regularization pool 122 to DEGRADED, or writes the corresponding persistence wait flag or abnormal compensation flag to the header metadata area of ​​regularization pool 122.

[0186] If the upstream delayed data source returns a valid response within the preset number of liveness detection attempts, the system determines that the link is normal and accesses the internal log statistics module to extract the duration compensation coefficient of the current network long-tail latency distribution. The lifespan of the regularization layer 120 is dynamically extended based on the duration compensation coefficient.

[0187] During the initial compensation, the formula in This represents the base survival time window; in multiple compensation scenarios, the formula... It can be replaced with the currently effective survival time window threshold, that is, the currently effective survival time window threshold is used as the base time window for this compensation calculation.

[0188] The system generates a new time window threshold according to the following formula: ; in, This represents the threshold value of the new time window generated after system compensation. During the first compensation, it represents the original allocated base survival time window; during multiple compensations, it represents the currently effective survival time window threshold. This represents the duration compensation coefficient extracted by the system based on the long-tail delay distribution of the network; This represents the business weight coefficient corresponding to the current data.

[0189] Duration compensation coefficient The value can be set from 0.1 to 0.5, and its specific value can be determined by linear mapping between the ratio of the standard deviation of recent network packet delay time and the average delay time.

[0190] By introducing business weight coefficients Business transaction entities with longer state dependency links, higher state integrity requirements, or higher correlation complexity can obtain relatively longer waiting tolerance; while low-weight business transaction entities only receive basic time compensation.

[0191] The new time window threshold obtained by calculation is used in the regularization layer 120. Update the current survival time window of the consolidation pool 122, so that the consolidation pool 122 continues to wait for the arrival of lag state data until the new time window threshold is reached, or the associated state feedback factor is triggered. Reaching the system's preset full execution threshold .

[0192] To prevent consolidation pool 122 from being retained indefinitely, the system presets a maximum number of compensation attempts and a maximum lifespan. When the number of compensation attempts for the same consolidation pool 122 reaches the maximum number of compensation attempts, or the cumulative lifespan after compensation reaches the maximum lifespan, consolidation layer 120 will no longer extend the time window. Instead, it will enter a state dimensionality reduction, disk persistence waiting, or abnormal compensation path based on the dependency level of the current missing state.

[0193] S504, perform late status compensation.

[0194] For the dimensionality reduction output record corresponding to the global tracking identifier that has already undergone state dimensionality reduction delivery, the state backend 121 generates and saves a compensation index. The compensation index includes at least the global tracking identifier, the dimensionality reduction output version number, the missing state dimension identifier, the original regularization pool release time, and the compensation validity period.

[0195] When missing state data arrives at the gateway layer 110 within the compensation validity period, the parsing engine 111 queries the compensation index based on the global tracking identifier and writes the late state data into the compensation queue. The execution layer 140 reads the late state data, the dimensionality reduction output version number, and the missing state dimension identifier from the compensation queue, and performs idempotent correction calculations based on the above information to generate the corresponding compensation result.

[0196] If late status data arrives after the compensation validity period has expired, the system will mark the late status data as expired status data and perform archiving, alarm, or discarding according to the preset policy.

[0197] In some embodiments, the header metadata area of ​​the regularization pool 122 also records a status data version number or an event sequence number for handling duplicate and out-of-order arrival status data.

[0198] When data of the same state dimension arrives repeatedly, the state backend 121 compares the event sequence number of the newly arrived data with the event sequence number already written to the state slot. If the event sequence number of the newly arrived data is not higher than that of the already written data, the state backend 121 determines that the newly arrived data is duplicate data and performs discard or archiving processing; if the event sequence number of the newly arrived data is higher than that of the already written data, the state backend 121 updates the corresponding state slot and synchronously updates the version number field of the regularization pool 122.

[0199] Through the above processing, the system can maintain the version consistency of state data within the regularization pool 122 even when there are duplicate reports or out-of-order arrivals in the multi-source asynchronous data streams.

[0200] In embodiments with multiple computing nodes 142, the scheduler 131 maintains the resource level of each computing node 142.

[0201] When selecting an execution node for the consolidation pool 122, the scheduler 131 prioritizes the compute node 142 located on the same physical node or in the same shared memory domain as the consolidation pool 122. When the resource level of the local node exceeds the congestion threshold, the scheduler 131 selects a remote compute node 142 with a lower resource level and access permissions, and sends a data block access descriptor to the remote compute node 142.

[0202] By employing the node selection strategy described above, scheduler 131 can reduce cross-node data transmission overhead while ensuring data accessibility, and avoid long-term waiting of the regularization pool 122 due to local node resource congestion.

[0203] In one specific application embodiment, the present invention is applied to the scenario of government and enterprise centralized procurement order processing on an office supplies e-commerce platform. This platform is used to process online procurement of goods such as stationery, printing consumables, office equipment, and office furniture. For a single government and enterprise centralized procurement order, the platform needs to receive status data reported by the order system, payment system, inventory system, warehousing system, logistics system, and invoicing notification system. Due to differences in processing latency, interface callback timing, and network link status among the various business subsystems, multiple status data points for the same procurement order typically arrive asynchronously.

[0204] A government / enterprise client submitted a bulk procurement order for office supplies. The order includes multiple SKUs such as printer paper, ink cartridges, folders, and office chairs, with a total order amount of 18,500 yuan and involves inventory locking in two warehouses. The system designates the business object corresponding to this order as transaction object A and generates a global tracking identifier (Trace-A) for it. After receiving the order creation data, the gateway layer 110's parsing engine 111 extracts the global tracking identifier Trace-A and business attribute tags from the message header. These business attribute tags include fields such as bulk procurement by government / enterprise, multi-SKU transaction, cross-warehouse inventory locking, and high processing level. The system calculates the business weight coefficient for transaction object A based on the business hierarchy mapping function. =0.82.

[0205] In this embodiment, the system sets a minimum survival time threshold for anti-shake. =2s, the maximum survival time threshold is 12s, then the maximum time scaling range =10s. According to the formula The base survival time window of the regularization pool 122 corresponding to transaction object A can be calculated as follows: =2 + 10 × 0.82 = 10.2 s; Therefore, the state backend 121 establishes an independent regularization pool 122 for the global tracing identifier Trace-A, and allows this regularization pool 122 to wait for 10.2 seconds in the basic state to receive associated state data from different business subsystems. Meanwhile, if the system sets the baseline memory block size to 64KB and the memory amplification variable to 256KB, the initial memory capacity quota of this regularization pool 122 can be calculated as follows: 64KB + 256KB × 0.82 = 273.92KB; In the state template corresponding to transaction object A, the baseline state set includes order creation state s1, payment confirmation state s2, cross-warehouse inventory lock state s3, warehouse sorting state s4, logistics pickup state s5, and invoice notification state s6. Among these, s1, s2, and s3 are configured as strongly dependent states, s4 and s5 are configured as compensable states, and s6 is configured as a weakly dependent state. The attribute weights corresponding to each state are as follows: s1 has a weight of 0.20, s2 has a weight of 0.25, s3 has a weight of 0.25, s4 has a weight of 0.15, s5 has a weight of 0.10, and s6 has a weight of 0.05, with the sum of all weights being 1.

[0206] During the data access process, the order creation status s1 arrives at t=0s, the payment confirmation status s2 arrives at t=0.8s, the cross-warehouse inventory lock status s3 arrives at t=2.2s, the warehouse sorting status s4 arrives at t=3.4s, and the logistics pickup status s5 arrives at t=4.1s. Inspection thread 123 reads the status arrival status within the regularization pool 122 according to a preset scanning cycle and calculates the associated status feedback factor. When s1 arrives, =0.20; when both s1 and s2 are reached, =0.20+0.25=0.45; When s1, s2 and s3 are all reached, =0.20+0.25+0.25=0.70; When s4 reaches a further point, =0.85; when s5 arrives, =0.95.

[0207] In this embodiment, the system sets a high-readiness state locking threshold. =0.70, full execution threshold =0.90, system congestion threshold =0.75. When t=2.2s, the regularized pool 122 corresponds to = Reached 0.70, which satisfies the requirement. Furthermore, the strongly dependent states s1, s2, and s3 have all been reached. Scheduler 131 further collects the resource status of the computing node 142 where the execution layer 140 is located. Assuming the maximum allowed capacity of thread pool 141 is 100 threads, of which 40 threads are currently executing tasks, 5 threads are locked, and 3 threads are reserved, then the number of unavailable threads is 48, and the thread occupancy rate is... =48 / 100=0.48.

[0208] Meanwhile, the total runtime memory of compute node 142 is 32GB, with a current runtime memory usage of 9GB. The total capacity of the consolidation pool is 8GB, and the allocated capacity of the consolidation pool is 2.6GB. The system calculates the memory consumption rate based on the runtime memory usage and the consolidation pool memory usage. =0.31. Let the thread-weighted smoothing coefficient be... =0.60, memory-weighted smoothing coefficient =0.40, then the resource water level line for: =0.60×0.48+0.40×0.31=0.412; because =0.412, which is less than the system congestion threshold. =0.75, scheduler 131 determines that the current computing node 142's resources are sufficient and triggers the phase change locking mechanism. Scheduler 131 switches the regularization pool 122 corresponding to transaction object A from the ACTIVE state to the LOCKED state and sends a pre-allocation control instruction to thread pool 141. Thread pool 141 allocates a thread from the idle worker thread queue as a dedicated computing thread for regularization pool 122.

[0209] Subsequently, scheduler 131 generates a data block access descriptor and writes it into the execution context of the dedicated computing thread. In this embodiment, the data block access descriptor includes the basing pool identifier Pool-Trace-A, the data block offset, the data length, the version number V3, and the access token Token-A17. Thus, before the data formally reaches the execution conditions, the dedicated computing thread has already established an access mapping relationship with the data blocks to be processed in the basing pool 122.

[0210] When t=4.1s, the logistics pickup status s5 is reached, and inspection threads 123 recalculate. =0.95. Because Since the strong dependency states s1, s2, and s3 have all been reached, the state backend 121 switches the regularization pool 122 from the LOCKED state to the READY state and wakes up the dedicated computation thread via the event bus. After being woken up, the dedicated computation thread switches the regularization pool 122 from the READY state to the PROCESSING state and reads the data block access descriptor in the execution context. If the version number V3 in the data block access descriptor matches the current version number in the metadata area of ​​the regularization pool 122 header, the dedicated computation thread increments the reference count field of the regularization pool 122 and reads the associated state dataset in the regularization pool 122 that has met the execution conditions.

[0211] During this business operation, a dedicated computing thread performs processing such as order fulfillment confirmation, inventory lock confirmation, warehouse task archiving, and initial logistics status writing based on the order creation status, payment confirmation status, cross-warehouse inventory lock status, warehouse sorting status, and logistics pickup status. After the business operation is completed, the dedicated computing thread decrements the reference count field of the regularization pool 122 by one and clears the data block access descriptors in the execution context. When the status backend 121 detects that the reference count field is zero and the pool status is PROCESSING, it switches the regularization pool 122 to the RELEASED state and reclaims the corresponding memory resources.

[0212] In another application scenario, if the office supplies bulk purchase order corresponding to transaction object B only receives the order creation status s1, payment confirmation status s2, cross-warehouse inventory lock status s3, and warehouse sorting status s4 near the basic survival time window, while the logistics pickup status s5 and invoice notification status s6 have not yet arrived, then at this time... =0.20+0.25+0.25+0.15=0.85, which has not yet reached the full execution threshold. =0.90. If the scheduler 131 calculates the resource water level at this time... =0.868, and If the system determines that the current computing node 142 is in a state of congestion and back pressure, then the system will determine that the current computing node 142 is in a state of congestion and back pressure.

[0213] Since the missing logistics pickup state s5 is a compensable state and the invoice notification state s6 is a weakly dependent state, the regularization layer 120 switches the regularization pool 122 corresponding to transaction object B from the ACTIVE state to the DEGRADED state, and writes state missing placeholders into the state slots corresponding to s5 and s6. The state missing placeholder includes a missing flag, a state dimension number, a data length field, a placeholder type field, a validation field, and a reserved field. Subsequently, the system delivers the dimensionality reduction dataset with the state missing placeholder and the missing state dimension identifier to the compute node 142. The compute node 142 performs local business operations only based on the already arrived strong dependent states and core available states, and sends a release command to the state backend 121 after the operation is completed.

[0214] For the missing logistics pickup status s5, the status backend 121 generates a compensation index before releasing the regularization pool 122. This compensation index includes the global tracking identifier Trace-B, the dimensionality reduction output version number, the missing status dimension identifier s5, the original regularization pool release time, and the compensation validity period. If the logistics system reissues the logistics pickup status s5 corresponding to Trace-B within the compensation validity period, the parsing engine 111 queries the compensation index based on the global tracking identifier Trace-B and writes the late status data into the compensation queue. The execution layer 140 reads the late status data, the dimensionality reduction output version number, and the missing status dimension identifier from the compensation queue, performs idempotent correction calculations to rewrite the initial logistics node or correct the order display status.

[0215] If, in another transaction object C, the payment confirmation status s2 has not yet arrived, while other statuses have arrived successively, since the payment confirmation status s2 is a strongly dependent status, even if the system detects that the related status feedback factor is close to the complete execution threshold, it will not switch the consolidation pool 122 to the READY state, nor will it forcibly deliver it to the compute node 142. Instead, it will write a persistent waiting flag or an exception compensation flag to the header metadata area of ​​the consolidation pool 122, and transfer the corresponding data to the disk persistent waiting queue or exception compensation queue. Through this processing, core business logic such as inventory deduction, fulfillment confirmation, or shipment confirmation can be incorrectly executed when the payment confirmation status is missing.

[0216] As can be seen from the above application examples, the present invention is applicable to e-commerce and government / enterprise centralized procurement scenarios for office supplies. It aggregates multi-source asynchronous time-series status data such as orders, payments, inventory, warehousing, logistics, and invoicing notifications into the same regularization pool 122, and uses associated status feedback factors. This characterizes the data readiness level of the consolidation pool 122. Simultaneously, the scheduler 131 incorporates the resource water level. The system assesses the stress state of compute node 142. When data is nearing readiness and resources are sufficient, phase-change locking is performed in advance. When data is nearing timeout and the system is congested, dimensionality reduction delivery, persistent waiting, or late compensation is performed according to the dependency level of the missing state. This reduces memory pressure caused by the backlog of multi-source time-series data, reduces queuing waiting after data reaches the execution condition, and avoids erroneous calculation results when strong dependent states are missing.

[0217] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic regularization and collaborative scheduling of time-series information data, characterized in that, Includes the following steps: An asynchronous multi-source time-series data stream is accessed, and a global tracing identifier and business attribute tag are extracted from it. A corresponding regularization pool is initialized based on the global tracing identifier, and a basic survival time window is set for the regularization pool. The regularization pool is a logically contiguous data storage area applied for with the global tracing identifier as the primary key index, used to temporarily store asynchronous time-series data belonging to the same global tracing identifier. The regularization pool is scanned according to a preset scanning cycle to determine the completeness of the arrival of associated state data, and to calculate the associated state feedback factor used to quantify the readiness of the current regularization pool data. The associated state feedback factor is the ratio of the sum of the inherent attribute weights of the currently arrived and verified state dimensions to the sum of the inherent attribute weights of all state dimensions in the baseline state set. Collect hardware or runtime environment load data of the execution layer and calculate a resource level line to characterize the current computing resource pressure; wherein, the resource level line is a value obtained by weighting and combining thread occupancy rate and memory consumption rate according to the corresponding weighted smoothing coefficient; the thread occupancy rate is determined according to the ratio between the number of threads in the thread pool in the execution layer that are in an unavailable state and the maximum allowable capacity parameter of the thread pool; the memory consumption rate is determined according to the runtime memory occupancy rate and the regularization pool memory pool occupancy rate. When the associated state feedback factor is detected to have reached the high ready state locking threshold but not the full execution threshold, and the resource level is lower than the congestion threshold, the regularization pool is switched to the locked state, a dedicated thread is allocated, and a corresponding data block access descriptor is generated; the congestion threshold is a critical value of the resource level used to compare with the resource level to determine whether the execution layer has entered a congested back pressure state. When the associated state feedback factor reaches the full execution threshold, the dedicated thread reads the data in the regularization pool and performs calculations based on the data block access descriptor; When the elapsed time of the consolidation pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, the execution state dimensionality reduction or time window compensation operation is selected based on the relationship between the current resource level and the congestion threshold. The elapsed time of the consolidation pool is determined by the time difference between the creation timestamp of the consolidation pool and the current system clock. When the remaining time between the basic survival time window and the elapsed time is less than or equal to the preset tolerance time threshold, it is determined that the elapsed time of the consolidation pool is approaching the basic survival time window.

2. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, The asynchronous multi-source time-series data stream is accessed, from which a global tracing identifier and a service attribute tag are extracted. A corresponding regularization pool is initialized based on the global tracing identifier, and a basic liveness time window is set for the regularization pool, including: The asynchronous multi-source time-series data stream is parsed to extract the global tracking identifier and the service attribute tag; The business attribute tags are passed as input parameters to a preset business classification mapping function to obtain the corresponding business weight coefficients; wherein, the business classification mapping function is a preset mapping rule that takes the business scale field, processing level field, and state link complexity field in the business attribute tags as input and outputs the discrete score value of the business classification; the business weight coefficients are obtained by normalizing the discrete score value of the business classification. The global tracking identifier is used as the primary key index to apply for a logically contiguous data storage area to construct the regularization pool; The basic survival time window and initial memory capacity quota of the consolidation pool are calculated and set based on the business weight coefficient. The basic survival time window is determined based on the minimum survival time threshold, the maximum time scaling range, and the business weight coefficient. The minimum survival time threshold is the minimum waiting time that the consolidation pool must retain in memory after its creation. The maximum time scaling range is determined based on the difference between the maximum survival time threshold and the minimum survival time threshold. The maximum survival time threshold is the longest time that the consolidation pool is allowed to remain in a waiting state in memory.

3. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, The step of scanning the regularized pool according to a preset scanning cycle to determine the completeness of the arrival of associated state data and calculating the associated state feedback factor used to quantify the readiness of the current regularized pool data includes: Parse the type identifier of the data record in the regularization pool and match it with the state dimension in the baseline state set; Based on the matching results and internal verification results, maintain a state arrival indicator variable for each state dimension; Extract the inherent attribute weights of each state dimension in the baseline state set, and determine the associated state feedback factor based on the inherent attribute weights and the corresponding state arrival indicator variables.

4. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, The process of collecting hardware or runtime environment load data from the execution layer and calculating a resource level to characterize the current computing resource pressure includes: The number of threads in the thread pool that are in an unavailable state in the execution layer is counted, and the current thread occupancy rate is determined by combining the system's preset maximum allowable thread pool capacity parameter. Collect and obtain the runtime memory usage and the consolidation pool memory pool usage to determine the current memory consumption rate; Based on preset thread-weighted smoothing coefficients and memory-weighted smoothing coefficients, the thread occupancy rate and the memory consumption rate are weighted and merged to generate the resource water level. The sum of the thread-weighted smoothing coefficient and the memory-weighted smoothing coefficient is 1, and both are preset according to the hardware configuration, task type, and historical load characteristics of the computing node. When the computing node is mainly limited by thread execution resources, the thread-weighted smoothing coefficient is greater than the memory-weighted smoothing coefficient; when the computing node is mainly limited by memory resources, the memory-weighted smoothing coefficient is greater than the thread-weighted smoothing coefficient.

5. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, The generation of the corresponding data block access descriptor includes: The memory management module's access descriptor generation interface is called to generate the data block access descriptor, and the data block access descriptor is written into the reserved execution context of the dedicated thread; When the regularization layer and the execution layer are located in the same process address space, the generated data block access descriptor is a reference handle pointing to the regularization pool logical memory block; When the regularization layer and the execution layer are located in different processes or on different physical nodes, the generated data block access descriptor includes at least the regularization pool identifier, data block offset, data length, version number, and access token.

6. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 5, characterized in that, When the associated state feedback factor reaches the complete execution threshold, the dedicated thread reads the data in the regularization pool according to the data block access descriptor and performs calculations, including: The dedicated thread is awakened, and a consistency check is performed between the version number in the data block access descriptor and the current version number in the metadata area of ​​the regularization pool header. If the verification is successful, increment the reference count field of the regularization pool by one, locate the target data block according to the data block access descriptor, and read and execute the business operation logic. After the business operation logic is completed, the reference count field is decremented by one, the data block access descriptor in the execution context is cleared, and a release instruction is sent to reclaim the memory resources occupied by the consolidation pool.

7. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, When the elapsed time of the regularization pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, and the resource water level is greater than or equal to the congestion threshold, the selected execution state dimensionality reduction operation includes: Based on the state template identifier recorded in the header metadata area of ​​the regularization pool, the state template corresponding to the state template identifier is read, and the state dependency level corresponding to the missing state dimension is queried in the state template. The state template is a pre-configured state dependency level for each state dimension in the baseline state set. The state dependency level includes strong dependency state, compensable state, and weak dependency state. The strong dependency state is a state that affects the correctness of the core calculation result after being missing. The compensable state is a state that can be corrected by compensation calculation after the arrival of late data after being missing. The weak dependency state is a state that affects the auxiliary notification, trajectory display, or non-core statistical results after being missing. If the missing state dimension belongs to a compensable state or a weakly dependent state, a state missing placeholder is written to the missing position in the regularization pool, a dimensionality reduction dataset with missing state dimension identifier is generated and delivered to the computing node to trigger the dependency-free dimensionality reduction logic; the state missing placeholder is used to indicate that the corresponding state dimension has not yet been reached and is different from the real business data; the dependency-free dimensionality reduction logic skips the computing branches related to the missing state dimension and performs local business operations based on the state data that has been reached and meets the execution conditions; If the missing state dimension is a strongly dependent state, a persistent wait flag or an anomaly compensation flag is written to the header metadata area of ​​the regularization pool, and the corresponding regularization pool is transferred to the disk persistent wait queue or an anomaly compensation queue.

8. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, When the elapsed time of the regularization pool approaches the basic survival time window, and the associated state feedback factor has not reached the complete execution threshold, and the resource water level is lower than the congestion threshold, the selected execution time window compensation operation includes: Send a liveness probe command to the upstream lagging data source to confirm the network link and interface status; If a valid response is returned within the preset number of liveness detection attempts, extract the duration compensation coefficient used to characterize the current network long-tail delay distribution; Based on the currently effective survival time window threshold, the duration compensation coefficient, and the corresponding business weight coefficient, a new time window threshold is generated to dynamically extend the survival time window of the regularization pool.

9. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 7, characterized in that, After generating the dimensionality-reduced dataset with missing state dimension identifiers and submitting it to the computing node, the following steps are also included: Generate and save a compensation index, which includes at least the global tracking identifier, the dimensionality reduction output version number, the missing state dimension identifier, and the compensation validity period; When late arrival status data arrives within the compensation validity period, the compensation index is queried based on the global tracking identifier, and the late arrival status data is written into the compensation queue. The delayed state data, the dimensionality reduction output version number, and the missing state dimension identifier are read from the compensation queue. The corresponding dimensionality reduction output record is located based on the dimensionality reduction output version number. The missing state dimension identifier is used to determine the missing state placeholder or data item to be corrected in the dimensionality reduction output record corresponding to the delayed state data. The event sequence number or state data version number of the delayed state data is compared with the event sequence number or state data version number of the data processed in the same state dimension to determine whether the delayed state data has been compensated. If it has been compensated, the correction calculation is not repeated. If it has not been compensated, the delayed state data is used to replace the corresponding missing state placeholder, or the output field related to the missing state dimension in the dimensionality reduction output record is corrected to generate a compensation result.

10. The method for dynamic regularization and collaborative scheduling of time-series information data according to claim 1, characterized in that, After initializing the corresponding regularization pool based on the global tracking identifier, the process also includes performing capacity monitoring and expansion control on the regularization pool: The relationship between the used capacity of the regularized pool and the initial memory capacity quota is continuously monitored. When the used capacity reaches a preset expansion threshold, it is determined whether the regularized pool is in the locked state; If the regularization pool is not in the locked state, request an expansion slot and update the data block location metadata and capacity metadata of the regularization pool; If the regularization pool is already in the locked state, migration of existing data blocks is prohibited, and new status data is written to the reserved expansion slot or transferred to the overflow queue.

Images