A distributed data processing method, device, storage medium and electronic equipment

Abstract

This specification discloses a distributed data processing method, apparatus, storage medium, and electronic device. The method includes: acquiring streaming change events and node baseline dimension data for target business data nodes in an online database through a preset non-intrusive log data capture method; transmitting the streaming change events to a stream-batch integrated distributed storage system; using a real-time computing engine to perform dynamic time sharding processing on the streaming change events according to a preset time window strategy; calculating the multi-dimensional incremental aggregation value of each target business node in the current time window based on the baseline dimension data and superimposing it with the underlying baseline state snapshot to obtain a real-time aggregated state value; and updating the data of the target business nodes based on this.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a distributed data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the rapid development of the Internet and distributed technologies, high-concurrency data processing architectures are widely used in various scenarios, such as product services on e-commerce platforms, hotspot status synchronization in large-scale online games, real-time aggregation of massive sensor data from the Internet of Things, and user service data processing on financial user service platforms. In these application scenarios, certain core business nodes in the system (i.e., hotspot data objects) often face up to hundreds of thousands or even millions of concurrent status update requests (TPS) in a very short period of time.

[0003] Currently, industry service platforms typically rely on underlying relational databases to handle state updates for these core nodes. To alleviate the database write pressure caused by extremely high concurrency, related technologies often employ message queues for asynchronous buffering and data persistence, or rely on scheduled tasks for periodic (e.g., half-hourly, hourly) aggregation updates. However, these solutions are inherently limited by the physical performance bottlenecks of the underlying relational database (such as row lock conflicts, concurrent connection limits, and disk I / O bottlenecks). This results in the latest aggregated state values ​​of the target nodes not being reflected in a timely manner, leading to significant computational and display delays. In scenarios with extremely high data consistency requirements, this lag in state reflection can easily trigger risks to the service platform. Summary of the Invention

[0004] This specification provides a distributed data processing method, apparatus, storage medium, and electronic device. By employing a non-intrusive capture computation and dynamic sharding mechanism, this specification avoids the high-concurrency performance bottleneck of databases, significantly improving the timeliness of node status data while reducing computational costs. The technical solution is as follows: Firstly, embodiments of this specification provide a distributed data processing method, the method comprising: By using a pre-defined non-intrusive log data capture method, the system obtains streaming change events and node baseline dimension data for the target business data nodes in the online database of the computer platform service. The streaming change event is transmitted to the integrated batch and stream distributed storage system, and the streaming change event is dynamically sliced ​​according to the preset time window strategy in the real-time computing engine. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the benchmark dimension data. The multidimensional incremental aggregated value is superimposed with the underlying baseline state snapshot to obtain the real-time aggregated state value, and the target business node is updated based on the real-time aggregated state value.

[0005] Secondly, embodiments of this specification provide a distributed data processing apparatus, the apparatus comprising: The data acquisition module is used to acquire streaming change events and node baseline dimension data for the target business data nodes in the online database of the computer platform service through a preset non-intrusive log data capture method. The incremental aggregation module is used to transmit the streaming change event to the integrated batch storage system and use a real-time computing engine to perform dynamic time slicing processing on the streaming change event according to a preset time window strategy. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the baseline dimension data. The data update module is used to overlay the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain a real-time aggregated state value, and to perform node data update processing on the target business node based on the real-time aggregated state value.

[0006] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0007] Fourthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0008] Fifthly, this specification provides a computer program product storing at least one instruction adapted to be loaded by a processor and to execute the method steps of one or more embodiments of this specification.

[0009] Fifthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0010] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: In one or more embodiments of this specification, streaming change events and baseline dimension data from an online database are obtained indirectly through a non-intrusive underlying log capture method. These are then transmitted to a stream-batch integrated distributed storage system. A real-time computing engine, based on a dynamic time sharding strategy, calculates a multi-dimensional incremental aggregate value from a massive number of scattered heterogeneous change events within a specific time window. Finally, this incremental aggregate value is overlaid with the underlying baseline state snapshot to derive and update the real-time aggregated state value of the target business node. This solves the system processing performance issues caused by underlying physical row lock conflicts, limited concurrent connections, and disk I / O bottlenecks when relational databases face extremely high concurrency state update requests. The system addresses fundamental technical issues such as severe limitations, delayed response times to target node status, and the resulting risks of data status exceeding limits (overdraft) and system avalanche. By employing a non-intrusive underlying log capture method and a unified stream-batch computing architecture, it achieves physical and logical decoupling between the main service link and the node status calculation link of the computer service, completely avoiding the concurrent read / write impact on the online master database caused by high-frequency status calculations. Simultaneously, it introduces dynamic time sharding and multi-dimensional incremental aggregation mechanisms, cleverly reducing the cost of queuing hundreds of thousands of expensive single database row locks and I / O disk write operations into a single memory-level differential merging and a single full snapshot overlay within a time window, thus lowering the overall data computation and processing costs. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a distributed data processing system provided in the embodiments of this specification; Figure 2 This is a schematic diagram illustrating an example of distributed data processing provided in the embodiments of this specification; Figure 3 This is a schematic diagram of a multidimensional incremental aggregation value calculation provided in the embodiments of this specification; Figure 4 This is a schematic diagram of the structure of a distributed data processing device provided in the embodiments of this specification; Figure 5 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0013] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0014] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0015] The present specification will now be described in detail with reference to specific embodiments.

[0016] Please see Figure 1 This is a schematic diagram of a distributed data processing system provided in this specification. Figure 1 As shown, the distributed data processing system may include at least a client cluster and a service platform 100.

[0017] The client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0018] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, electronic device, wireless communication device, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0019] The service platform 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0020] In one or more embodiments of this specification, the service platform 100 can establish a communication connection with at least one client in the client cluster, and complete the data interaction in the distributed data processing process based on the communication connection, such as online transaction data interaction. For example, the service platform 100 can provide computer platform services to the client based on the distributed data processing method of this specification, such as e-commerce services, daily life services, express delivery services, financial user services, and cloud computing services. It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0021] The distributed data processing system embodiments provided in this specification and the distributed data processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the distributed data processing method involved in one or more embodiments in this specification can be the aforementioned service platform 100. The implementation process of the distributed data processing system embodiments can be found in the following method embodiments, which will not be repeated here.

[0022] In one embodiment, such as Figure 1 As shown, a distributed data processing method is proposed. This method can be implemented using a computer program and can run on a distributed data processing device based on the von Neumann architecture. The computer program can be integrated into applications or run as a standalone utility application. The distributed data processing device can serve as a service platform.

[0023] Specifically, this distributed data processing method includes: S102: Obtain streaming data change events and node baseline dimension data for the target business data node from the online database of the computer platform service through a preset non-intrusive log data capture method; The default non-intrusive log data capture method refers to a data extraction technique that directly parses transaction logs (such as binary logs in relational databases) from the underlying file system or memory buffer of the database without modifying the core business logic of the original computer platform service, adding database triggers, or imposing additional table-level or row-level exclusive locks on the database tables. This default non-intrusive log data capture method can be implemented by deploying a separate, lightweight heterogeneous data change monitoring component, using a bypass log parsing proxy, etc. Online databases refer to production environment transactional databases that carry the core transaction links of computer platform services and are oriented towards high-frequency concurrent read and write requests.

[0024] Data streaming change events refer to a set of incremental data fragments with strict time-series attributes generated in the underlying logs during the operation of a target business data node due to state transitions. These fragments include the operation type and specific value that caused the state change.

[0025] Node baseline dimension data refers to reference data that is independent of high-frequency streaming change events and is used to describe the basic attributes, verification rules, and historical full status of the target business data node in the previous settlement cycle.

[0026] In one feasible implementation, the service platform can deploy a standalone, lightweight heterogeneous data change monitoring component. This component, posing as a slave node of the online database, establishes a long-lived connection with the master node of the online database on the computer platform service using a pre-defined, non-intrusive log data capture method, based on the underlying replication protocol. When the computer platform service performs concurrent state update operations on the target business data node, the underlying log engine of the online database pushes the generated streaming data change events to this monitoring component in real time. Simultaneously, this monitoring component subscribes to and retrieves the corresponding node baseline dimension data (such as the node configuration strategy resident in memory) in real time through a parallel caching interface. This process achieves lossless capture of incremental logs and baseline data within seconds, completely avoiding contention for input / output (I / O) resources on the main service link.

[0027] In one feasible implementation, the service platform can set up a bypass log parsing proxy module. This proxy module proactively initiates block read requests for the underlying log files to the server disk system where the online database of the computer platform service resides at fixed time intervals (e.g., 100 milliseconds) to parse out data streaming change events. Because data sources are diverse in high-concurrency systems, these streaming change events mix real-time baseline details, asynchronous delayed details, and periodic summary details. Simultaneously with acquiring the change events, a separate asynchronous thread initiates a slice query request to the offline data processing service to retrieve the node baseline dimension data (e.g., daily historical baseline state snapshots) of the target business data node. Subsequently, the proxy module binds and packages the change events belonging to the same target business data node with the baseline dimension data in memory, completing the acquisition of heterogeneous data.

[0028] S104: The streaming change event is transmitted to the integrated batch and stream distributed storage system, and the streaming change event is dynamically time-sliced ​​in the real-time computing engine according to the preset time window strategy. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the benchmark dimension data. Integrated stream-batch distributed storage system: refers to an underlying data foundation that adopts a log-structured merged tree (LSM-Tree) or other hybrid storage structures, supports extremely low-latency append writing of incremental event streams of computer platform services (such as computing resource services, e-commerce services, and financial resource services), and supports providing full-state sharded queries through primary key capabilities, such as a data lake storage system.

[0029] Real-time computing engines refer to distributed stream processing clusters built on distributed in-memory computing frameworks, capable of high-throughput and low-latency processing of unbounded data streams, such as the Flink engine.

[0030] The time window strategy refers to the triggering and partitioning rules in streaming computing that truncate an infinite number of streaming change events into data blocks of finite size (i.e., time slices) according to a specific time span or number of events.

[0031] Dynamic time slicing: refers to a mechanism that, during operation, automatically changes the time window span (e.g., dynamically adjusting from 5 seconds to 1 minute) based on external time scheduling instructions or internal load indicators without relying on manual intervention.

[0032] Multidimensional incremental aggregated value: refers to the pure difference data output from incremental records with different data structures within a specific time shard for the same target business node, after preprocessing such as state transition filtering, idempotent deduplication, and difference accumulation.

[0033] In one feasible implementation, streaming change events of the computer platform service are transmitted and persisted to the integrated stream-batch distributed storage system, such as a data lake storage system, in the form of message streams. The service platform calls the real-time computing engine for processing by subscribing to the system's data change log (Binlog). The preset time window strategy in the real-time computing engine is manifested as a rule scheduler based on an external clock configuration. For example, during regular business hours, the system sets the time sharding period to 10 seconds according to the timeliness priority principle; when a high-concurrency period instruction is received, the sharding period is automatically dynamically adjusted to 1 minute, sacrificing a slight timeliness for the ultimate improvement in throughput. Within each defined current time window, the engine internally constructs a rolling window and associates the aggregated streaming change events with synchronously acquired baseline dimension data (such as node configuration strategies) through distributed dual-stream association operations. The engine identifies the attributes of the events according to the configuration strategy, eliminates duplicate abnormal events, accumulates the increase and decrease values ​​of valid events, and finally calculates the multi-dimensional incremental aggregation value of the target business node under this specific time window.

[0034] S106: The multidimensional incremental aggregated value is superimposed with the underlying baseline state snapshot to obtain the real-time aggregated state value, and the target business node is updated based on the real-time aggregated state value.

[0035] Multidimensional incremental aggregated value: refers to the pure dynamic difference output by the real-time computing engine after deduplication, rule mapping and accumulation within the time slice of the previous step (e.g., the total reduction in value within a certain time window).

[0036] For example, in an e-commerce scenario, the target business node could be a popular product inventory node. Streaming change events could include user A submitting an order (locking inventory -1), user B canceling an order (releasing inventory +1), and user C completing payment (locked inventory becomes actual deduction from inventory). Multi-dimensional incremental aggregated values ​​could be like this: within a millisecond-level time slice, summarizing the aggregated values ​​output after 100,000 clicks: Pre-deduction (locked) inventory increment: -450 units, Actual released available inventory increment: +20 units, Sold (paid for) inventory increment: +300 units.

[0037] For example, in a high-frequency sensing scenario for the Internet of Things (IoT) and smart grids, the target business node is a smart transformer load monitoring node in a large industrial park. Streaming change events include: Factory A starting a large motor (instantly increasing active power by 500kW, while simultaneously generating a large amount of reactive power); Factory B turning off its lighting (reducing power by 50kW). The sensing devices send hundreds of status change logs per second. Multidimensional incremental aggregated values, such as: within a 1-minute time slice, the stream processing engine filters out noise and redundant signals, calculating the aggregated load change as follows: Active power increment: +2.5MW, Reactive power increment: +0.8MVar, Peak surge current: 1200A.

[0038] Baseline snapshot: refers to the static full baseline data of the target business node at a specific historical moment (such as midnight of each day or the previous reconciliation cycle) stored in the underlying structured data warehouse or integrated stream and batch distributed storage system.

[0039] Real-time aggregated status value: refers to a quantitative indicator that reflects the most accurate and absolutely true status of the target business node at present. Mathematically, it is equal to the algebraic sum of the baseline state snapshot and all multidimensional incremental aggregated values.

[0040] Node data update processing: refers to a series of state solidification and system feedback operations performed based on the latest calculated real-time aggregated state value, including but not limited to memory buffer updates, persistent disk writes, out-of-bounds (such as overdraft) interception and blocking, and visualization dashboard push.

[0041] Indicatively, after obtaining the multi-dimensional incremental aggregated value for a specific target business node through the real-time computing engine, the corresponding baseline state snapshot is loaded from the underlying layer. Algebraic superposition calculation is performed in the operator memory to obtain the latest real-time aggregated state value. During node data update processing, this real-time aggregated state value is not immediately written to the persistent disk. Instead, a memory-level multi-level incremental merging buffer pool is constructed before flushing data to the persistent layer of the stream-batch integrated distributed storage system. For data objects identified as "high-concurrency target data objects," their direct disk write operation is actively intercepted, and their latest real-time aggregated state value resides in this buffer pool. Only when the cumulative difference value of the node in the buffer pool exceeds a preset absolute value physical threshold (e.g., a change of 10,000), or its residence time in the buffer pool reaches the maximum latency tolerance (e.g., 5 seconds), is the merged, latest real-time aggregated state value with a single structured characteristic flushed as a whole data block to the persistent layer of the stream-batch integrated distributed storage system. This operation reduces the concurrent write pressure on the underlying storage by several orders of magnitude.

[0042] In the embodiments described in this specification, streaming change events and baseline dimension data from the online database are obtained indirectly through a non-intrusive underlying log capture method. These are then transmitted to a stream-batch integrated distributed storage system. A real-time computing engine, based on a dynamic time sharding strategy, calculates a multi-dimensional incremental aggregate value from the massive, scattered, heterogeneous change events within a specific time window. Finally, this incremental aggregate value is overlaid with the underlying baseline state snapshot to derive and update the real-time aggregated state value of the target business node. This approach solves the problem of severely impacted system processing performance when relational databases face extremely high concurrency state update requests due to underlying physical row lock conflicts, limited concurrent connections, and disk I / O bottlenecks. This addresses underlying technical issues such as limitations, delayed response times to target node status, and the resulting risks of data status out-of-bounds (overdraft) and system avalanche. By employing a non-intrusive underlying log capture method and a stream-batch integrated computing architecture, the physical and logical decoupling of the main service link and node status calculation link of the computer service is achieved, completely avoiding the concurrent read / write impact on the online master database caused by high-frequency status calculations. Simultaneously, a dynamic time sharding and multi-dimensional incremental aggregation mechanism is introduced, cleverly reducing the originally costly database single-row lock queuing and I / O disk write operations (numbering in the hundreds of thousands) into a single memory-level differential merging and a single full snapshot overlay within a time window, thus lowering the overall data computation and processing costs.

[0043] In one feasible implementation, after specifically executing S106 to calculate the real-time aggregated state value by superimposing the multidimensional incremental aggregated value with the underlying baseline state snapshot, the following method can also be referenced: Step B2: Perform negative threshold overflow detection for the target service node to detect whether the real-time aggregated status value is out of bounds. Negative threshold overflow detection refers to a detection mechanism in which, after the computing engine obtains the latest state value, it compares the state value with the system's preset lower limit extreme value (i.e., negative threshold, such as absolute zero or the minimum allowed credit limit) in real time through a rule matching algorithm to determine whether excessive consumption has occurred.

[0044] Out-of-bounds type refers to a preset system status classification label. When the status value of a target business node breaches the security defense, such as when the threshold does not match, the system assigns a data attribute identifier to that node to distinguish it from a security type or a normal type.

[0045] Step B4: If the real-time aggregated status value is detected as an out-of-bounds type, a blocking signal is triggered for the target service node, and the out-of-bounds status is pushed to the status monitoring terminal.

[0046] A blocking signal is an emergency interruption and isolation control command broadcast by the real-time computing engine to the upstream traffic scheduling layer (such as an API gateway, transaction routing layer, or message queue) of the system, used to reject subsequent incremental write requests from the out-of-bounds node at the physical or logical level.

[0047] Status monitoring terminals refer to nodes that receive and visualize the status of high-time alarms in the system. They can be specifically represented as automated risk portals or visual report dashboards, used to trigger automated interception actions or for maintenance personnel to review trend data.

[0048] To illustrate, after the real-time computing engine obtains the real-time aggregated state value, it calls the rule operator to synchronously extract the rigid negative threshold configured in the baseline dimension data for the target business node (e.g., the safety lower limit is strictly set to 0). The operator executes a fast algebra comparison instruction in memory, and once it determines that the real-time aggregated state value is less than 0, it immediately marks the target business node with an "out-of-bounds type" status label.

[0049] Once the engine detects that the real-time aggregated status value has been marked as out of bounds, it immediately broadcasts a global blocking signal to the upstream system's traffic access layer via a bypass channel. Upon receiving this signal, the upstream gateway immediately adds the target service node to the circuit breaker blacklist, directly discarding all subsequent transaction requests directed to that node. Simultaneously, the engine asynchronously pushes a structured exception log containing the out-of-bounds timestamp, node ID, and out-of-bounds type to a status monitoring terminal (such as a risk portal) with a very high preset alarm timeout for risk isolation and record-keeping.

[0050] In the embodiments described in this specification, by introducing a negative threshold overflow detection and out-of-bounds blocking mechanism for target business nodes after state superposition calculation, not only is efficient bypass calculation of node states achieved, but an automated security closed loop from underlying data calculation to main link access control is also constructed. This mechanism enables the system to identify abnormal states with out-of-bounds characteristics in real time and accurately under extremely high concurrency scenarios, and automatically trigger traffic blocking signals for specific nodes with millisecond-level latency. This effectively avoids risks such as node resource overdraft and system resource exhaustion caused by the lag in database state response from the source. At the same time, combined with highly timely monitoring terminal status push, the automated risk interception capability and operation and maintenance response efficiency of the distributed architecture are improved.

[0051] Optionally, the online database includes an underlying distributed relational database. Specifically, the process of obtaining the data streaming change events and node baseline dimension data for the target business data node from the online database of the computer platform service can refer to the following methods: S202: Capture the first type of incremental record, the second type of incremental record, and the third type of incremental record with different structural characteristics or timeliness characteristics from the underlying distributed relational database corresponding to the computer platform service as data streaming change events; The first type of incremental record is the baseline detailed data that is triggered and written in real time by data interaction between business nodes; it is usually the data change record generated by the main thread directly and synchronously executing the write operation when the front-end business interaction occurs, and has the highest time sensitivity and data real-time performance.

[0052] The second type of incremental record is delayed detailed data written after asynchronous buffering processing by the message queue. Typically, during peak periods of front-end business, in order to protect the database from overload, the interaction commands are first pushed into the middleware for buffering and queuing, and then the change records are asynchronously written to the underlying database by the background consumption thread, with tolerable delays of seconds or minutes.

[0053] The third type of incremental record refers to the summary detail data written after periodic batch aggregation by a scheduled task. Typically, massive amounts of small interactive data are not written to the database one by one. Instead, the scheduling engine performs macro-aggregation at fixed long intervals (such as half an hour or one hour) and writes the macro-differential records to the underlying database in a single batch.

[0054] S204: Synchronize the historical dimension reference values ​​and node configuration strategies to the integrated stream and batch distributed storage system and use them as node baseline dimension data.

[0055] Historical reference values ​​refer to the total absolute state values ​​(such as end-of-day snapshots) that the system has fixed for the target business data node at the end of the previous complete business settlement cycle.

[0056] Node configuration strategy refers to a set of rules and parameters that restrict or guide the static or infrequent modification of the state transition of the target business data node.

[0057] As an illustration, the service platform establishes a one-way subscription link between the non-intrusive capture component and the main log node of the underlying distributed relational database. The component reads the underlying physical change log line by line in chronological order. After capturing the change log, the capture component performs real-time classification and tagging based on the table-level metadata or operation source identifier carried in the underlying log. If the log originates from the core synchronous transaction table, it is marked as a first-type incremental record; if it originates from the asynchronous consumption queuing table, it is marked as a second-type incremental record; and if it originates from the batch processing summary table, it is marked as a third-type incremental record. These three types of heterogeneous records are uniformly encapsulated into a standard format data streaming change event for output.

[0058] Furthermore, during the parallel phase of continuous streaming data capture, the service platform's system activates an independent offline data synchronization channel. During specific daily business downtime, this channel batch-extracts historical dimension reference values ​​from the previous period and the latest node configuration strategies from the offline data warehouse. Using either full coverage or incremental updates, it synchronizes these values ​​to the designated dimension tables of the integrated stream-batch distributed storage system, providing stable and reliable baseline operator data for subsequent real-time stream computing.

[0059] In one feasible implementation, before specifically executing S104 to transmit the data streaming change event to the integrated stream-batch distributed storage system, the following method may also be referenced: S302: Parse the binary log of the underlying distributed relational database to extract streaming change events; A binary log is a low-level system file in a distributed relational database that records all data change operations (such as insertion, update, and deletion) sequentially in the form of a structured binary stream during the transaction commit phase. It records the absolute physical truth of data state transitions.

[0060] Streaming change events are extracted from the aforementioned binary logs, decoded, and converted into incremental data records with a standardized structure.

[0061] S304: Based on the primary key mechanism of the integrated stream and batch distributed storage system, high-frequency repetitive events targeting the same business node within the same time window are deduplicated and intercepted.

[0062] The primary key mechanism refers to the underlying indexing technology within a batch-stream integrated distributed storage system (such as a storage base built on a log structure merging tree) that uses a preset unique identifier (such as a transaction serial number combined with node attribute identifier) ​​to perform precise addressing, merging and overwriting of data records and conflict detection.

[0063] High-frequency recurring events refer to redundant streaming change events with the same transaction characteristics generated for the same target business node within a very short time window due to abnormal reasons such as network timeout retries, failure of idempotency control of upstream distributed gateways, or overlapping concurrent transactions.

[0064] Deduplication interception eliminates the control actions of the aforementioned redundant change events by physically overwriting or logically discarding the data before it is actually written to disk and participates in downstream incremental aggregation calculations.

[0065] As an illustration, the service platform pre-deploys a separate distributed log broker cluster. This distributed log broker cluster does not directly connect to the master database, but instead reads binary log file blocks directly in a micro-batch polling manner by mounting a shared storage volume on the server hosting the underlying distributed relational database. The broker cluster performs distributed parallel decompression and parsing of the file blocks, extracting massive amounts of streaming change events at high speed.

[0066] Before streaming change events officially reach the integrated stream-batch distributed storage system, the agent cluster constructs a Bloom filter and a sliding time window state dictionary in memory based on the primary key mechanism mapping rules. For each extracted streaming change event, its corresponding primary key is first generated in memory and compared against a lookup table. If the primary key already exists in the memory state dictionary within the same time window, it is directly identified as a high-frequency repetitive event. The agent component directly performs deduplication interception at the memory level, discarding the dirty data corresponding to this deduplication interception. Understandably, only unique change events that successfully pass the memory primary key conflict detection are allowed to be transmitted to the integrated stream-batch distributed storage system. This approach protects the network bandwidth and disk resources of the underlying storage.

[0067] Optionally, the specific implementation of dynamic time-slicing processing of the streaming change event according to the preset time window strategy can refer to the following methods: S402: Extract the current operating metrics of the real-time computing engine; Current operating metrics refer to quantitative data reflecting cluster health and load pressure collected in real time by underlying monitoring probes when the real-time computing engine processes unbounded data streams. These data include, but are not limited to, operator back pressure, memory level, CPU utilization, and event ingestion rate.

[0068] S404: When the current operating metric does not match the preset baseline, dynamically adjust the time window span parameter of the sharding.

[0069] The preset baseline refers to the range of indicators or physical threshold boundaries that the service platform pre-calibrates to indicate that the cluster is in a safe and efficient operating state (such as a back pressure indicator of 0 and a CPU utilization rate between 40% and 70%).

[0070] Mismatch refers to the abnormal state where the above-mentioned current operating indicators exceed the highest warning threshold (overload) or fall below the lowest safety threshold (extremely idle).

[0071] The time window span parameter refers to the time scale unit for truncating continuous streaming change events into micro-batch multidimensional incremental aggregation calculations (e.g., the absolute time span parameter changes from 5 seconds to 60 seconds).

[0072] To illustrate, when the real-time computing engine executes stream processing tasks, its internal monitoring and scheduling master node extracts the current operating metrics of each worker node at an extremely high frequency (e.g., every 100 milliseconds). The extracted metrics focus on system resource bottlenecks, such as the backpressure buildup depth between operators and the memory water level percentage in the system status backend. The extracted metrics are then compared to a preset baseline (e.g., a memory water level warning line of 85%). When a sudden surge of massive streaming change events causes the memory water level to spike above the preset baseline (i.e., an "over-limit mismatch"), a protection mechanism is triggered, dynamically and smoothly extending the time window span parameter of the slices (e.g., from the original 10-second span, gradually stretched to 30 seconds, then 60 seconds, according to preset steps). Extending the time window means the engine can fold more change events in a single batch, thereby significantly reducing the frequency of outputting incremental results downstream, reducing I / O overhead, and alleviating backpressure. When the traffic recedes and the memory level falls back to within the safe baseline, the span parameter is adjusted again to shrink it back to the initial 10 seconds to restore the ultimate timeliness of the data.

[0073] In some embodiments, the current operating metrics include operating load metrics and memory level; specifically, the dynamic adjustment of the time window span parameter for sharding includes: 1) When a node event of a target service node is detected to meet the condition of a surge in traffic rate, and the memory level matches the preset tolerance threshold, the time window span parameter is extended. 2) When the traffic rate of the node event is detected to drop back to the preset safe range, the time window span parameter is subjected to time contraction processing.

[0074] The runtime load metric refers to the current physical or logical processing pressure status of each computing node within the real-time computing engine, such as the back pressure between operators, CPU utilization, and thread pool queuing depth.

[0075] Memory water level refers to the percentage of used capacity in the memory buffer pool or state backend of the real-time computing engine used to temporarily store, correlate, and aggregate streaming change events.

[0076] A surge in traffic rate indicates an abnormal input state where the number of concurrent requests to a specific target business node increases non-linearly and steeply within a unit of time (e.g., TPS jumps from 10,000 to 100,000 in 1 second), exceeding the system's normal throughput capacity.

[0077] Preset tolerance threshold: refers to the highest safety level set to prevent the computing engine from experiencing memory overflow (OOM, Out of Memory) (e.g., memory utilization reaching 85% or 90%).

[0078] Time stretching / time shrinking: refers to dynamically increasing or decreasing the time span parameter of a single aggregation calculation (for example, increasing the sharding period from 1 second to 10 seconds is stretching, and reducing it back to 1 second is shrinking). Essentially, it sacrifices a small amount of real-time performance to achieve a multiple increase in throughput or restore the ultimate real-time response.

[0079] To illustrate, the real-time computing engine's monitoring daemon periodically collects runtime load metrics and memory levels. Only when two conditions are simultaneously met (forming a logical "AND" relationship): first, the streaming event arrival rate of a specific target business node doubles within two consecutive probing cycles (meeting a traffic rate surge); second, the memory level used to aggregate the node's status reaches a preset tolerance threshold (i.e., matches), will the engine immediately issue a blocking command, suspending the output of calculation results downstream and forcibly extending the processing time. For example, directly extending the current time window from a fixed 2 seconds to 30 seconds. This gives the engine ample buffer time to merge the massive changes within those 30 seconds in memory, avoiding memory exhaustion due to frequent disk writes.

[0080] Optional, please see Figure 3 , Figure 3 This is a flowchart illustrating the calculation of multidimensional incremental aggregated values. Specifically, the calculation of the multidimensional incremental aggregated values ​​of each target business node under the current time window corresponding to the time slice includes: S502: Within the real-time computing engine, establish a dual-stream associated task based on a scrolling window or a cumulative window; Dual-stream association tasks refer to distributed operator tasks in distributed real-time computing frameworks (such as Flink) that match and assemble continuously input high-frequency dynamic data streams (streaming change events) and low-frequency updated static reference data streams (node ​​configuration strategy tables) in the same memory context based on a common primary key (such as node ID).

[0081] Rolling windows or cumulative windows are time boundary models for truncating unbounded data streams in streaming computing. A rolling window refers to a shard with a fixed time span and no overlap; a cumulative window refers to a shard that continuously collects data based on a specific trigger condition (such as a session period) until settlement is triggered.

[0082] S504: Match and associate the streaming change events with the node configuration strategy table in the baseline dimension data, and convert change events with different structural characteristics into standardized state transition vectors. A standardized state transition vector (SVT) is a mathematical multidimensional array extracted from originally heterogeneous log records after rule-based cleaning and translation. It has a unified dimensional format (e.g., [dimensional A increase / decrease, dimensional B increase / decrease, dimensional C state identifier]). This abstracts away the structural differences in the underlying logs, enabling direct algebraic operations.

[0083] S506: Perform deduplication and differential accumulation operations on multiple standardized state transition vectors belonging to the same target service node within the same current time window to obtain the multidimensional incremental aggregated value of the target service node under the current time window corresponding to the time slice.

[0084] The differential accumulation operation refers to the process of adding or merging the corresponding dimensions of all the standardized state transition vectors of the same physical object within the same time boundary to obtain the final net change.

[0085] To illustrate, using a scrolling window as an example, the real-time computing engine divides strictly non-overlapping scrolling windows in memory. Simultaneously, the real-time computing engine pushes the entire node configuration policy table as a broadcast data stream to the local memory of all distributed computing nodes, forming a broadcast state. Thus, a high-frequency streaming change event stream and the local broadcast configuration stream establish a high-speed dual-stream association task. When heterogeneous streaming change events (including real-time, asynchronous, or batch logs) flow into the scrolling window, the computing operator directly queries the broadcast state in local memory to obtain the configuration policy of the target business node to which the event belongs. At this point, heterogeneous events can be mapped and refined according to the policy rules. For example, a complex "delayed data transfer log initiated by channel A for node B," combined with the "data transfer cost and data transfer destination node account rules" in the policy table, can have redundant fields removed and uniformly mapped into a standard, standardized state transition vector containing only numerical changes (e.g., [data volume dimension: +10, integral dimension: -10]). At the moment the current scrolling window is about to close, the aggregation operator delineates the data set belonging to the same target business node in memory. The hash set can be used to deduplicate vectors with the same transaction serial number (removing dirty vectors generated by retries), and parallel vector addition (differential accumulation operation) can be performed on all remaining valid vectors. When the window closes, the engine uniquely outputs a multidimensional incremental aggregate value representing the sum of all actions of that node within that window.

[0086] To illustrate, taking the cumulative window as an example, the real-time computing engine establishes a cumulative window based on business sessions. Since node configuration policies themselves may change over time (e.g., promotional rules switch at 12:00), the engine constructs the node configuration policy table as a temporal table with a version stamp, thus establishing a dual-stream association task based on precise event time. When a streaming change event arrives, the matching association process relies not only on the node ID but also strictly on the timestamp inherent in the underlying log of the change event. The engine searches the temporal table for the historical version of the node configuration policy corresponding to that timestamp for matching, thereby calculating a fully accurate standardized state transition vector. This temporal shuttle association completely solves the fatal problem of out-of-order events matching incorrect rules due to network latency. During the collection period of the cumulative window, the engine uses a Bloom filter for extremely lightweight real-time deduplication. As streaming change events accumulate, the underlying state backend continuously performs rolling differential accumulation on the generated standardized state transition vector. When the trigger threshold of the cumulative window is reached (e.g., the window duration expires or the accumulated difference is too large), the final multi-dimensional incremental aggregation value is sent out.

[0087] Furthermore, in some embodiments, the specific execution of superimposing the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain and update the real-time aggregated state value of the target service node can be performed in the following manner: Step A2: Overlay the multidimensional incremental aggregated value with the underlying baseline state snapshot to calculate the current real-time aggregated state value of the target service node; To illustrate, just before the current time window closes and the computation engine prepares to output the multi-dimensional incremental aggregated value, the computation operator, through a high-speed cache interface, pre-fetches a snapshot of the target business node's baseline state from the underlying stream-batch integrated distributed storage system to its local memory (such as Flink's StateBackend). The computation operator then directly performs the superposition calculation in its memory. For example, using multi-dimensional vector addition, each dimension of the baseline vector is added to each dimension of the incremental vector one by one. The operator directly generates a unique real-time aggregated state value in memory and outputs it. This approach minimizes I / O interaction, achieves extreme computational speed, and ensures the continuity of state updates under high concurrency.

[0088] Step A4: When updating the real-time aggregated state value to the persistence layer of the integrated stream and batch distributed storage system, construct a memory-level multi-level incremental merging buffer pool; The persistent layer refers to the final physical storage medium built on physical disks (such as solid-state drives or distributed object storage) in a batch-stream integrated distributed storage system to ensure that data is not lost when power is off and can be stored for a long time.

[0089] Memory-level refers to the main memory space of a computer system (such as the heap memory or off-heap memory of computing engine nodes), and its read and write speed is several orders of magnitude higher than that of physical disks, and it can complete the overwriting of state values ​​at the nanosecond or microsecond level.

[0090] A multi-level incremental merge buffer pool refers to one or more hierarchical data structure spaces specifically allocated in memory (such as JVM heap memory) to temporarily store and merge frequently modified instructions for the same object before the data is actually written to the persistent layer on the physical disk.

[0091] To illustrate, the first-level buffer (thread-local buffer) is constructed as follows: Within each worker thread of the real-time computing engine, a local memory region based on hash mapping is independently initialized for each thread. When the engine obtains the real-time aggregated state value of a target business node, it first pushes it into the first-level buffer of the current thread handling the event for state overwriting. Since this region is thread-private, state merging at this level requires no locking, achieving extremely fast lock-free pre-aggregation.

[0092] A second-level buffer (node-level global buffer) is constructed: Outside of individual worker threads, a global concurrent mapping table is initialized in the shared memory area of ​​the worker nodes as a second-level buffer pool. The system is configured with an asynchronous micro-batch scheduling thread to extract local state values ​​from the first-level buffer of each thread and aggregate them into the second-level buffer at extremely short intervals (e.g., every 10 milliseconds) without locking. Within the second-level buffer, the state values ​​of the same target business node are double-overlapped and merged to form the unique latest state of that node across the entire computing node dimension. Then, it waits for certain conditions to be met before being uniformly flushed into the persistence layer.

[0093] Step A6: Calculate the status update frequency of each target business node within a unit of time. When the status update frequency exceeds the preset high concurrency threshold, identify the corresponding target business node as a high concurrency target data object. Unit time refers to the length of the observation window used by the system for high-frequency calculations and statistical sampling. It is usually configured as an extremely short time scale in the millisecond or second range (such as 1 second or 500 milliseconds).

[0094] The state update frequency refers to the absolute number of concurrent write or state modification requests received by the real-time computing engine within the aforementioned unit of time for a specific physical or logical data unit (i.e., the target business node).

[0095] The preset high concurrency threshold refers to the water level boundary (e.g., set to 5000 times / second) that is pre-configured by the system to distinguish between normal concurrent traffic and extreme hotspot traffic. This threshold can be a statically configured physical limit value or a relative water level line dynamically generated based on the current overall load of the system.

[0096] High-concurrency target data objects refer to business nodes whose traffic characteristics exceed the high-concurrency threshold within the current observation period, are dynamically labeled as hotspots by the system, and thus need to be routed to a multi-level incremental merging buffer pool for special write folding protection.

[0097] To illustrate, the real-time computing engine maintains a concurrent-safe hash dictionary in memory based on unique node identifiers (IDs) and introduces a sliding time window mechanism. When a multi-dimensional incremental aggregation value containing a specific target business node ID is generated, the counter for the corresponding ID in the hash dictionary performs a lock-free atomic increment operation. As the time window slides forward, the system accurately calculates the absolute cumulative number of modifications for each active node in the past unit of time (e.g., the most recent second), i.e., the state update frequency. Then, the system controls its built-in guardian probe thread to poll the hash dictionary at an extremely high frequency (e.g., every 10 milliseconds). When the state update frequency of node ID X is detected to have reached a certain frequency, and this frequency value is greater than a preset high-concurrency threshold, the probe thread immediately triggers a callback mechanism, dynamically marking node X as a high-concurrency target data object in the system's global routing table. Subsequent write operations to node X will be immediately redirected to the incremental merge buffer pool for interception and disk write.

[0098] Step A8: For the identified high-concurrency target data objects, perform pre-aggregation and differential accumulation operations in the multi-level incremental merge buffer pool to intercept the update write action. When the differential accumulation value exceeds the absolute value threshold or the buffer residence time reaches the maximum latency tolerance, refresh the latest real-time aggregated state value after merging to the persistence layer.

[0099] Intercepting update write actions refers to an isolation mechanism built by the system between the application layer and the operating system kernel, which actively suspends or bypasses the disk synchronous write calls that should be triggered immediately, forcibly trapping data changes in high-speed memory.

[0100] Pre-aggregation and differential accumulation operations refer to the process of overwriting or vector-adding multiple temporary state values ​​continuously generated by the same target business node in memory within a multi-level incremental merging buffer pool, so that no matter how many changes occur, only one latest state record of the node is always retained in memory (i.e., write folding).

[0101] The absolute value threshold refers to the maximum numerical difference boundary that the system presets to allow the node's state to accumulate in memory. It is used to prevent excessive data jumps during a single disk write from causing a disruption in the perception of upstream services.

[0102] Maximum latency tolerance refers to the maximum physical time (e.g., 3 seconds) a predefined state update record is allowed to reside in volatile memory after being merged. It is used to ensure that data can still meet eventual consistency when traffic suddenly drops, preventing excessive data loss due to unexpected power outages.

[0103] As an illustration, when the computing engine outputs incremental status for data objects marked as high-concurrency targets, the system's storage routing module directly cuts off the write-through channel to the underlying database. The engine redirects these status values ​​to a multi-level incremental merge buffer pool. Within the hash slots of this buffer pool, a dedicated aggregation operator performs high-frequency differential accumulation operations, continuously updating the real-time aggregated status value of the node, thus achieving memory-level interception of massive write operations.

[0104] Furthermore, the service platform's system simultaneously attaches two lightweight listeners to the node's record in the buffer pool: one is an incremental accumulator counter, and the other is a dwell time timer. These two listeners are in a contention-triggered state (i.e., logical OR relation). Whenever the node's cumulative change reaches the absolute value threshold during the accumulation process, or its dwell time in the buffer pool exhausts the maximum latency tolerance, the listener immediately sends an interrupt signal to the I / O scheduler. The scheduler quickly locks the latest merged real-time aggregated state value, treats it as a single transaction, actively refreshes it to the underlying persistence layer (stream-batch integrated distributed storage system), then clears the node's counter and timer in the buffer pool, and begins the next folding cycle.

[0105] This specification introduces a dynamic frequency-based hotspot identification and multi-level buffered controlled disk write mechanism in the state overlay calculation stage. This mechanism endows the system with the ability to dynamically route ordinary traffic and extreme hotspot traffic. It cleverly intercepts and transforms massive discrete synchronous write operations for high-concurrency business nodes into high-speed memory-level lock-free pre-aggregation and write folding. Subsequently, it smoothly and in batches refreshes the data to the persistence layer based on strict dual thresholds (absolute difference and time latency tolerance). This not only reduces the concurrent write pressure on the underlying storage by several orders of magnitude, eliminating the risk of system crashes under sudden surges, but also ensures extremely high global throughput while taking into account the extreme read / write timeliness of ordinary nodes and the eventual consistency of core business data, giving the distributed architecture extremely strong underlying physical protection and storage robustness.

[0106] The following will combine Figure 4 This specification provides a detailed description of the distributed data processing apparatus provided in the embodiments. It should be noted that... Figure 4 The distributed data processing device shown is used to execute this specification. Figures 1-3The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-3 The example shown.

[0107] Please see Figure 4 This diagram illustrates the structure of a distributed data processing device according to an embodiment of this specification. The distributed data processing device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the distributed data processing device 1 includes a data acquisition module 11, an incremental aggregation module 12, and a data update module 13, specifically used for: Data acquisition module 11 is used to acquire data streaming change events and node baseline dimension data for target business data nodes in the online database of computer platform services through a preset non-intrusive log data capture method; The incremental aggregation module 12 is used to transmit the streaming change event to the integrated batch storage system and use a real-time computing engine to perform dynamic time slicing processing on the streaming change event according to a preset time window strategy. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the baseline dimension data. The data update module 13 is used to superimpose the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain a real-time aggregated state value, and perform node data update processing on the target business node based on the real-time aggregated state value.

[0108] It should be noted that the distributed data processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the distributed data processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the distributed data processing device and the distributed data processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0109] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0110] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-3 The distributed data processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-3 The specific details of the illustrated embodiments will not be elaborated here.

[0111] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-3 The distributed data processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-3 The specific details of the illustrated embodiments will not be elaborated here.

[0112] Please refer to Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected to each other via the bus 1050.

[0113] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or a combination of central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU mainly handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0114] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0115] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 1030 can be a temperature sensor for acquiring the operating temperature of the electronic device. The output device 1040 can be a speaker for outputting audio signals.

[0116] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0117] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0118] exist Figure 5 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the distributed data processing method as described in the various method embodiments of this specification.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0120] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the features, data, and information involved in this specification were all obtained under full authorization.

[0121] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A distributed data processing method applied to a service platform, the method comprising: By using a pre-defined non-intrusive log data capture method, the system obtains streaming change events and node baseline dimension data for the target business data nodes in the online database of the computer platform service. The streaming change event is transmitted to the integrated batch and stream distributed storage system, and the streaming change event is dynamically sliced ​​according to the preset time window strategy in the real-time computing engine. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the benchmark dimension data. The multidimensional incremental aggregated value is superimposed with the underlying baseline state snapshot to obtain the real-time aggregated state value, and the target business node is updated based on the real-time aggregated state value.

2. The method according to claim 1, wherein the online database includes an underlying distributed relational database, and the acquisition of data streaming change events and node baseline dimension data for the target business data node in the online database of the computer platform service includes: From the underlying distributed relational database corresponding to the computer platform service, capture the first type of incremental record, the second type of incremental record, and the third type of incremental record with different structural characteristics or timeliness characteristics as data streaming change events; The historical dimension reference values ​​and node configuration strategies are synchronized to the integrated stream and batch distributed storage system and used as the node baseline dimension data. The first type of incremental record is baseline detailed data written in real time triggered by data interaction between business nodes; the second type of incremental record is delayed detailed data written after asynchronous buffering by a message queue; and the third type of incremental record is summary detailed data written after periodic batch aggregation by a scheduled task.

3. The method according to claim 2, further comprising, before transmitting the data streaming change event to the stream-batch integrated distributed storage system: Parse the binary logs of the underlying distributed relational database to extract streaming change events; Based on the primary key mechanism of the integrated stream and batch distributed storage system, high-frequency repetitive events targeting the same business node within the same time window are deduplicated and intercepted.

4. The method according to claim 1, wherein the dynamic time-slicing processing of the streaming change event according to a preset time window strategy includes: Extract the current operating metrics of the real-time computing engine; When the current operating metrics do not match the preset baseline, the time window span parameter of the sharding is dynamically adjusted.

5. The method according to claim 4, wherein the current operating indicators include operating load indicators and memory level; and the dynamic adjustment of the time window span parameter for sharding includes: When a node event of a target service node is detected to meet the condition of a surge in traffic rate, and the memory level matches the preset tolerance threshold, the time window span parameter is extended. When the traffic rate of the node event is detected to drop back to a preset safe range, the time window span parameter is subjected to time contraction processing.

6. The method according to claim 1, wherein calculating the multidimensional incremental aggregate value of each target service node under the current time window corresponding to the time slice includes: Within the real-time computing engine, dual-stream associated tasks are established based on scrolling windows or cumulative windows; The streaming change events are matched and associated with the node configuration strategy table in the baseline dimension data, and change events with different structural characteristics are uniformly converted into standardized state transition vectors. For multiple standardized state transition vectors belonging to the same target service node within the same current time window, deduplication and differential accumulation operations are performed to obtain the multidimensional incremental aggregated value of the target service node under the current time window corresponding to the time slice.

7. The method according to claim 6, wherein the step of superimposing the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain and update the real-time aggregated state value of the target service node includes: The multidimensional incremental aggregated value is superimposed with the underlying baseline state snapshot to calculate the current real-time aggregated state value of the target service node. When updating the real-time aggregated state value to the persistent layer of the integrated stream and batch distributed storage system, a memory-level multi-level incremental merging buffer pool is constructed. The status update frequency of each target service node within a unit of time is statistically analyzed. When the status update frequency exceeds a preset high-concurrency threshold, the corresponding target service node is identified as a high-concurrency target data object. For the identified high-concurrency target data objects, pre-aggregation and differential accumulation operations are performed in the multi-level incremental merging buffer pool to intercept the update disk writing action. When the differential accumulation value exceeds the absolute value threshold or the buffer residence time reaches the maximum latency tolerance, the latest real-time aggregated state value after merging is refreshed to the persistence layer.

8. The method according to claim 1, further comprising, after superimposing the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain the real-time aggregated state value: Perform negative threshold overflow detection on the target service node to detect whether the real-time aggregated status value is out of bounds. If the real-time aggregated status value is detected as an out-of-bounds type, a blocking signal is triggered for the target service node, and the out-of-bounds status is pushed to the status monitoring terminal.

9. A distributed data processing device applied to a service platform, the device comprising: The data acquisition module is used to acquire streaming change events and node baseline dimension data for the target business data nodes in the online database of the computer platform service through a preset non-intrusive log data capture method. The incremental aggregation module is used to transmit the streaming change event to the integrated batch storage system and use a real-time computing engine to perform dynamic time slicing processing on the streaming change event according to a preset time window strategy. In each time slice, the multi-dimensional incremental aggregation value of each target business node under the current time window corresponding to the time slice is calculated based on the baseline dimension data. The data update module is used to overlay the multidimensional incremental aggregated value with the underlying baseline state snapshot to obtain a real-time aggregated state value, and to perform node data update processing on the target business node based on the real-time aggregated state value.

10. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1 to 8.

11. A computer program product storing at least one instruction, the at least one instruction being loaded by a processor and executing the steps of the method as described in any one of claims 1 to 8.

12. An electronic device, comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as described in any one of claims 1 to 8.

Images