Advertisement data synchronization method

Abstract

This disclosure presents an advertising data synchronization method. One specific implementation of this method includes: an encapsulation system determining an advertising task queue based on a pre-acquired set of advertising request information; a master computing node extracting initial advertising tasks from the advertising task queue determined by the encapsulation system; the master computing node maintaining an advertising task tree based on the initial advertising tasks, and sending advertising data tasks extracted from the advertising task tree to associated slave computing nodes; in response to receiving the advertising data tasks, the slave computing nodes acquiring advertising consumption data based on the target advertising data tasks corresponding to the advertising data tasks; and the slave computing nodes writing the advertising consumption data into a consumption database and updating the advertising task tree and the node states corresponding to the slave computing nodes. This implementation, related to data synchronization, solves the problems of high latency, slow channel coverage, and occasional data gaps in traditional solutions.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of data synchronization, and more specifically to a method for synchronizing advertising data. Background Technology

[0002] Advertising platforms need to periodically retrieve advertising performance data (consumption reports, planning reports, creative reports, etc.) from multiple downstream media channels, facing the following technical problems: (1) Unified scheduling of heterogeneous APIs (Application Programming Interfaces) across multiple channels. The API protocols, authentication methods (Token / AK / SK (Access Key / Secret Key) / Oauth (Open Authorization), data formats, pagination strategies, and frequency limiting thresholds) of each media channel are different. Traditional solutions require developing separate synchronization scripts for each channel, resulting in low code reuse and high maintenance costs. (2) The timeliness of synchronizing massive amounts of data. Advertising data is retrieved daily, and the amount of data grows linearly with the number of accounts and creatives. In a single-node serial processing mode, data synchronization for one channel may take tens of minutes, and serial execution of multiple channels cannot complete full synchronization within an acceptable time. (3) The contradiction between media API frequency control and synchronization efficiency. Each media platform has strict QPS (Queries-per-second) limits for API calls, and API access will be blocked if the limit is exceeded. The lack of a unified frequency control mechanism can easily lead to synchronization interruption, but overly conservative frequency control will reduce synchronization efficiency. (4) Task allocation and fault recovery issues in a distributed environment. In the scenario of multi-node parallel processing, it is necessary to solve the problems of how to distribute tasks evenly and how to automatically discover and reassign unfinished tasks after node failure. Traditional solutions rely on manual monitoring and manual recovery. At present, when synchronizing advertising data, the usual method is to write a timed synchronization task for each media channel (such as CronJob (timed task) + single-threaded HTTP (Hypertext Transfer Protocol) call), and there is no unified scheduling framework between channels, each running independently.

[0003] However, the inventors discovered that when synchronizing advertising data using the above method, the following technical problems often arise: First, there is a lack of a unified task scheduling and routing framework. Each channel develops its own synchronization logic, resulting in different code structures. Adding a new channel requires development from scratch, leading to poor scalability. It is impossible to manage synchronization tasks uniformly by channel and action type.

[0004] Second, single-node serial processing cannot be horizontally scaled. This becomes a system bottleneck when dealing with large amounts of data, and adding more nodes cannot improve synchronous throughput.

[0005] Third, it lacks dynamic pagination parallel processing capabilities. Traditional solutions fetch data in a loop with a fixed page size, which cannot allocate different pages to multiple processing nodes for parallel execution, resulting in high latency when dealing with large amounts of data.

[0006] Fourth, there is a lack of multi-dimensional API frequency control mechanisms. Either the lack of frequency control leads to being blocked, or the global frequency control is too coarse-grained, making it impossible to independently limit traffic between different channels.

[0007] Fifth, it lacks automatic fault recovery capabilities. Tasks are lost after a processing node crashes, requiring manual detection and re-triggering, resulting in long recovery times.

[0008] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0009] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0010] Some embodiments of this disclosure propose an advertising data synchronization method to address the technical problems mentioned in the background section above.

[0011] In a first aspect, some embodiments of this disclosure provide an advertising data synchronization method, which includes: an encapsulation system determining an advertising task queue based on a pre-acquired set of advertising request information; a master computing node extracting initial advertising tasks from the advertising task queue determined by the encapsulation system; the master computing node maintaining an advertising task tree based on the initial advertising tasks, and sending advertising data tasks extracted from the advertising task tree to associated slave computing nodes; the slave computing nodes, in response to receiving advertising data tasks, acquiring advertising consumption data based on target advertising data tasks corresponding to the advertising data tasks; and the slave computing nodes writing the advertising consumption data into a consumption database, and updating the advertising task tree and the node status corresponding to the slave computing nodes.

[0012] The above-described embodiments of this disclosure have the following beneficial effects: First, the data synchronization timeliness is significantly improved: through dynamic pagination and parallel retrieval by multiple Workers (from computing nodes), the synchronization time is reduced from O(N) (time complexity, where N represents the total number of pages of advertising consumption data called via API) to O(N / M) (time complexity, where M represents the number of Workers). Assuming 10 Workers, the synchronization of tens of thousands of data points is reduced from 30 minutes to 3 minutes; with 13 channels processing in parallel, full synchronization can be completed within minutes, allowing advertisers to see near real-time data upon logging into the platform. Second, it offers strong scalability and low maintenance costs: adding a new channel only requires implementing the RunService (business executor interface) and registering, which can be completed within one day with zero framework modifications. It supports horizontal scaling; when data volume increases, only Worker nodes need to be added; automatic fault detection and recovery eliminates the need for manual monitoring, significantly reducing maintenance costs. Next, high synchronization stability and no data loss are ensured: ZK (ZooKeeper, a distributed coordination service) persistently stores task states + Master (primary computing node) automatically detects and redistributes faults + Worker retries. Synchronization is restored within seconds after a node crash, ensuring no data loss or gaps. Channel-level frequency control guarantees APIs will not exceed limits or be blocked, maintaining a consistently stable synchronization link. Then, API call rates closely approximate frequency control: Through frequency control management, hardware is kept at full load speed but only close to, without triggering, media frequency control, reducing call costs and the risk of triggering frequency control while also improving synchronization speed. Therefore, combining these four points, this invention offers faster data updates, a more stable system, supports more channels, and responds faster to new channels. It solves the problems of high latency, slow channel coverage, and occasional data gaps in traditional solutions, thereby improving customer retention and market share. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the advertising data synchronization method according to the present disclosure; Figure 2 This is a timing flowchart of the task of determining target advertising data according to some embodiments of the advertising data synchronization method of this disclosure. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0018] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0019] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0020] Before performing any of the operations involving the collection, storage, and use of user personal information (such as advertising request information and advertising consumption data) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing personal information subjects, and obtaining prior authorization and consent from personal information subjects.

[0021] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of an advertising data synchronization method according to the present disclosure. This advertising data synchronization method includes the following steps: Step 101: The encapsulation system determines the advertising task queue based on the pre-acquired set of advertising request information.

[0023] In some embodiments, the encapsulation system determines an advertising task queue based on a pre-acquired set of advertising request information and sends the advertising task queue to the main computing node. Here, the advertising request information in the advertising request information set can be information representing a request to synchronize advertising consumption data, obtained from a terminal device via a wired or wireless connection. In practice, firstly, the encapsulation system can encapsulate each advertising request information in the advertising request information set using a preset encapsulation algorithm to generate advertising tasks, thus obtaining an advertising task set. Secondly, the encapsulation system can write each advertising task in the advertising task set into the advertising task queue using a preset writing algorithm. The advertising task queue is initially empty. Next, the encapsulation system can send the written advertising task queue to the main computing node. For example, the preset encapsulation algorithm can be a serialization algorithm. The preset writing algorithm can be the RPUSH (Row-to-Tail) command. Here, the advertising request information in the advertising request information set can include, but is not limited to, at least one of the following: channel identifier (channelId), action type (actionType), and user identifier. The channel identifier can uniquely identify a channel. Action types can be, but are not limited to: Synchronizing Consumption Reports (representing ad spending data that needs to be synchronized to the Consumption Report), Synchronizing Plan Reports (representing ad spending data that needs to be synchronized to the Plan Report), and Synchronizing Creative Reports (representing ad spending data that needs to be synchronized to the Creative Report). A user identifier can uniquely identify a user (advertiser). For example, channels can be, but are not limited to: Channel A, Channel B, and Channel C. Here, the ad task queue can include: ad tasks. Ad tasks can include, but are not limited to, at least one of the following: queue identifier, enqueue timestamp, and ad request information. The queue identifier can uniquely identify an ad task included in the ad task queue. The enqueue timestamp can be the timestamp when the ad task is written to the ad task queue. As an example, a consumption report can represent: ad clicks. A plan report can represent: the ad's promotion plan. A creative report can represent: the ad's creative materials. Here, a consumption report can include, but is not limited to, at least one of the following: ad identifier and ad click count. A plan report can include, but is not limited to, at least one of the following: ad identifier, ad name, and ad delivery time period. A creative report can include, but is not limited to, at least one of the following: ad identifier, ad image, and ad video. An ad identifier can uniquely identify an ad. Ad clicks represent the number of times an ad is clicked. Ad time period represents the time period during which an ad is run. Ad image represents the images used in an ad. Ad video represents the videos used in an ad.

[0024] Here, the encapsulation system can be a system that maintains an ad task queue. The master compute node can be an ad compute node that maintains an ad task tree. The slave compute nodes can be ad compute nodes used to write ad consumption data into a consumption database. Ad compute nodes can also be compute nodes used to process ad tasks.

[0025] It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (Ultra Wide Band) connections, and other currently known or future wireless connection methods.

[0026] Optionally, the aforementioned master computing node is elected from the advertising computing node set using a preset master-slave election algorithm. The aforementioned slave computing nodes are the advertising computing nodes in the aforementioned advertising computing node set other than the aforementioned master computing node. For example, the preset master-slave election algorithm could be: ZooKeeper LeaderLatch (distributed automatic master election) algorithm.

[0027] Therefore, this invention achieves automatic node discovery, unified task routing, and parallel collaboration of multiple workers through ZK central coordination.

[0028] Step 102: The main computing node extracts the initial advertising task from the advertising task queue determined by the encapsulation system.

[0029] In some embodiments, the master compute node extracts an initial advertising task from the advertising task queue determined by the encapsulation system. In practice, the master compute node can control an extraction timer to extract an advertising task from the advertising task queue determined by the encapsulation system using a preset extraction algorithm, as the initial advertising task. Here, the extraction timer can be a timer included in the master compute node. For example, the extraction timer can be a JobDistributionTask timer. The preset extraction algorithm can be: retrieving an advertising task from the advertising task queue every 200 milliseconds using a BRPOP (Blocking Pop-up Command) blocking command.

[0030] Step 103: The master computing node maintains an advertising task tree based on the initial advertising task, and sends the advertising data tasks extracted from the advertising task tree to the associated slave computing nodes.

[0031] In some embodiments, the master computing node maintains an advertising task tree based on the initial advertising task and sends advertising data tasks extracted from the advertising task tree to associated slave computing nodes. The advertising data task includes an intermediate state identifier.

[0032] In practice, the aforementioned master computing node can maintain an advertising task tree based on the initial advertising task described above, and send the advertising data extracted from the advertising task tree to the associated slave computing nodes through the following steps: The first step is to write the initial advertising task and preset subtasks into the advertising task tree to maintain the advertising task tree. Here, the preset subtask can be a task representing a child node of the initial advertising task. The preset subtask can include, but is not limited to, at least one of the following: initial advertising task, intermediate state identifier, page number, and creation timestamp. The intermediate state identifier can be an identifier representing "unassigned," "assigned," or "completed." The page number can be 0. The creation timestamp can represent the time when the preset subtask was generated. In practice, the main computing node can use the initial advertising task as the root node representing the root task and the preset subtasks as child nodes representing the subtasks, and write them into the advertising task tree. Here, the advertising task tree is initially empty. The advertising task tree can include: root task and subtasks. Subtasks can include, but are not limited to, at least one of the following: channel identifier, action type, user identifier, intermediate state identifier, page number, and creation timestamp.

[0033] The second step involves controlling the channel thread to scan the aforementioned advertising task tree to obtain a set of advertising data subtasks. Each channel thread can correspond to a single channel. A channel thread can be a thread within the main compute node used to process advertising tasks corresponding to a single channel. In practice, the main compute node can control the channel thread to use each subtask in the aforementioned advertising task tree that meets preset channel task conditions as the set of advertising data subtasks. The preset channel task conditions can be: the channel identifier in the advertising task tree corresponds to the aforementioned channel thread, and the intermediate state representation of the subtasks represents "unassigned" subtasks. Here, the advertising data subtasks in the advertising data subtask set can include: creation timestamps.

[0034] The third step is to prioritize the aforementioned set of advertising data subtasks to generate a sequence of advertising data sorting tasks. In practice, the main computing node can use a preset sorting algorithm to prioritize the advertising data subtasks to generate the sequence of advertising data sorting tasks. For example, the preset sorting algorithm could be to sort the advertising data subtasks according to their creation timestamps from earliest to latest.

[0035] The fourth step involves filtering out the information of each slave compute node that meets the preset node conditions from the local coordination storage component, forming a slave compute node filtering information set. Here, the local coordination storage component can be a storage component used to store the slave compute node information set. For example, the local coordination storage component could be ZooKeeper (a distributed coordination service). The slave compute node information in the slave compute node information set may include, but is not limited to, at least one of the following: slave node identifier, node status, and a channel identifier set (indicating that the slave compute node can obtain data from the channel represented by the channel identifier). The slave node identifier can uniquely identify an advertising compute node in the aforementioned advertising compute node set, excluding the aforementioned main compute node. The node status can represent "idle" or "busy." The preset node conditions can be: slave compute node information with a node status representing "idle," and the existence of a channel identifier in the channel identifier set corresponding to the aforementioned channel thread.

[0036] Fifth, the first advertising data sorting task in the above advertising data sorting task sequence is sent as an advertising data task to the associated slave computing node. Here, the associated slave computing node can be any advertising computing node corresponding to any slave computing node filtering information in the slave computing node filtering information set.

[0037] Optionally, the master computing node may also send each advertising data sorting task in the advertising data sorting task sequence, except for the first advertising data sorting task, to the associated slave computing node.

[0038] Therefore, the advertising data sorting tasks representing N pagination sub-task nodes can be simultaneously sent to M slave computing nodes (the slave computing nodes are the advertising computing nodes that filter the information).

[0039] Step 104: In response to receiving the advertising data task, the computing node obtains advertising consumption data based on the target advertising data task corresponding to the advertising data task.

[0040] In some embodiments, in response to receiving an advertising data task, the computing node obtains advertising consumption data based on a target advertising data task corresponding to the advertising data task. The target advertising data task is determined through the following steps: In the first step, in response to receiving the advertising data task, the aforementioned slave computing node sends a first acknowledgment signal to the aforementioned master computing node. Here, the first acknowledgment signal can be a signal indicating "received".

[0041] In the second step, in response to receiving the first acknowledgment signal, the master computing node sends a second acknowledgment signal to the slave computing node. Here, the second acknowledgment signal can be a signal indicating "received".

[0042] Third, in response to receiving the second acknowledgment signal, the aforementioned slave computing node sends a third acknowledgment signal to the aforementioned master computing node. Here, the third acknowledgment signal can be a signal indicating "received".

[0043] Fourth, in response to receiving the third acknowledgment signal, the master computing node sends a fourth acknowledgment signal to the slave computing node. Here, the fourth acknowledgment signal can be a signal indicating "received".

[0044] Fifth, in response to receiving the fourth confirmation signal, the aforementioned slave computing node updates the intermediate state identifier included in the received advertising data task to update the advertising data task, thereby obtaining the target advertising data task, and also updates the slave computing node information corresponding to the aforementioned slave computing node. In practice, firstly, in response to receiving the fourth confirmation signal, the aforementioned slave computing node updates the intermediate state identifier included in the received advertising data task to an identifier representing "allocated," thereby determining the updated advertising data task as the target advertising data task. Secondly, the node status included in the slave computing node information corresponding to the aforementioned slave computing node is updated to "busy," thereby updating the set of slave computing node information that the local coordination storage component can store.

[0045] As an example, a sequence flowchart of the steps for determining the target advertising data task can be found here. Figure 2 . Figure 2 A timing flowchart illustrating the task of determining target advertising data according to some embodiments of the advertising data synchronization method of this disclosure is shown.

[0046] Therefore, a continuous confirmation anti-jitter mechanism can be used to send advertising data tasks from the master computing node to the slave computing node. This avoids frequent master switching caused by network jitter and solves the problem of duplicate task distribution caused by frequent master switching due to network jitter in existing solutions.

[0047] In practice, the aforementioned computing nodes can obtain advertising consumption data based on the target advertising data task corresponding to the advertising data task through the following steps: The first step is to parse and process the aforementioned target advertising data task to generate parsed information. In practice, the aforementioned computing node can determine the channel identifier, action type, user identifier, and page number included in the target advertising data task as parsed information.

[0048] The second step involves retrieving a task token from a pre-determined task token storage based on the parsed information. Here, the pre-determined task token storage can be a pre-determined storage for tokens. A token can represent a token allowing the current node to communicate. The task token storage may include a task token information set. The task token information in the task token information set may include, but is not limited to, at least one of the following: channel identifier, user identifier, action type, and token count. Here, the token count can represent the number of tokens in the task token storage corresponding to the task token information. In practice, firstly, the aforementioned slave computing node can determine the task token information in the task token storage that meets preset task conditions as the target task token information. The preset task conditions may be: task token information in the task token information set whose channel identifier is the same as the channel identifier included in the parsed information, whose user identifier is the same as the user identifier included in the parsed information, and whose action type is the same as the action type included in the parsed information. Secondly, in response to determining that the number of tokens included in the target task token information is greater than a first preset token count, the aforementioned slave computing node can randomly select one token from the tokens corresponding to the target task token information as the task token. For example, the first preset token count can be 0. Next, the aforementioned slave computing node can update the number of tokens included in the target task token information to a target token number, thereby updating the task token information set stored in the task token memory. Here, the target token number can be the difference between the number of tokens included in the target task token information and a second preset token number. For example, the second preset token number can be 1.

[0049] As an example, tokens can be generated by a task token store using a token bucket algorithm. For instance, the task token store generates a map (a data structure for storing key-value pairs) of task token information every second, using the channel identifier, user identifier, and action type as keys and the number of tokens as values. Here, a value of 0 indicates that all tokens have been allocated to other identical tasks, and the current task cannot be executed.

[0050] The third step involves retrieving initial ad consumption data from the local cache based on the parsed information, in response to determining that the task token meets the preset token normal conditions. The preset token normal conditions can be that the time taken to acquire the task token is less than a preset acquisition time. For example, the preset acquisition time could be 30 seconds. The acquisition time can be the duration from the moment the task token is acquired from the task token storage to the moment it is successfully acquired. Here, the local cache can be a cache used locally to store ad consumption cache data. The ad consumption cache data stored in the local cache can include, but is not limited to, at least one of the following: page number, channel identifier, action type, and consumption data. In practice, the compute node retrieves an ad consumption cache dataset that meets the preset cache conditions from the local cache, based on the parsed information, as the initial ad consumption data, in response to determining that the task token meets the preset token normal conditions. Here, the preset cache conditions can be: the channel identifier included in the ad consumption cache data is the same as the channel identifier included in the parsed information, and the action type included in the ad consumption cache data is the same as the action type included in the parsed information. Here, the consumption data can be, but is not limited to: consumption reports, planning reports, and creative reports.

[0051] The fourth step involves, in response to the determination that the initial ad consumption data is not empty, parsing the initial ad consumption data based on the parsed information to generate ad consumption data. In practice, in response to the determination that the initial ad consumption data is not empty, ad consumption cache data that meets preset parsing conditions included in the initial ad consumption data is identified as ad consumption data. The preset parsing condition may be that the pagination number included in the ad consumption cache data is the same as the pagination number included in the parsed information.

[0052] Fifth, in response to the determination that the initial ad consumption data is empty, the following sub-step is executed: The first sub-step involves calling the consumption interface to obtain the total advertising consumption data based on the parsed information. In practice, firstly, the aforementioned computing node can extract the programming interface, which meets the preset parsing conditions, from the preset configuration file based on the channel identifier and action type included in the parsed information. Here, the preset configuration file may include an interface configuration information set. The interface configuration information in the interface configuration information set may include: channel identifier, action type, and programming interface. The programming interface may be an API (Application Programming Interface) interface used to obtain the total advertising consumption data from the target server. The target server may be a server storing the total advertising consumption data. The preset parsing conditions may be: interface configuration information that is the same as the channel identifier and action type included in the parsed information. Secondly, based on the aforementioned consumption interface, a call request is sent to the target server corresponding to the aforementioned consumption interface to receive the total advertising consumption data sent by the target server. Here, the call request may include: channel identifier and action type. The total advertising consumption data may include, but is not limited to, at least one of the following: channel identifier, action type, and consumption data.

[0053] The second sub-step involves storing the total advertising consumption data in the local cache. In practice, firstly, the aforementioned slave computing node can use a preset partitioning algorithm to partition the total advertising consumption data to generate an advertising consumption partitioning data sequence. The advertising consumption partitioning data in this sequence may include, but is not limited to, at least one of the following: channel identifier, action type, and consumption data. The preset partitioning algorithm may be: assigning a preset number of records from the total advertising consumption data as one advertising consumption partition. For example, the preset number could be 100. Secondly, for each advertising consumption partition in the advertising consumption partitioning data sequence, the position number of the advertising consumption partition in the sequence and the advertising consumption partition data itself are stored as advertising consumption cache data in the local cache. Here, the page number in the advertising consumption cache data corresponds to the position number of the advertising consumption partition in the advertising consumption partitioning data sequence. The channel identifier, action type, and consumption data in the advertising consumption cache data correspond to the channel identifier, action type, and consumption data in the advertising consumption partition data.

[0054] The third sub-step involves updating the advertising task tree based on the total advertising consumption data. In practice, firstly, the aforementioned slave computing node can use a preset sub-task determination algorithm to determine the target sub-task corresponding to each advertising consumption segment in the advertising consumption segment data sequence corresponding to the total advertising consumption data, thus obtaining a target sub-task set. The target sub-tasks in the target sub-task set may include, but are not limited to, at least one of the following: channel identifier, action type, user identifier, intermediate state identifier, page number, and creation timestamp. Next, the aforementioned slave computing node can insert the target sub-task set into the advertising task tree to update the advertising task tree. Here, the algorithm for determining the pre-defined subtask can be as follows: First, the channel identifier and action type included in the advertising consumption segmentation data are determined as the channel identifier and action type included in the target subtask; second, the user identifier included in the parsed information is determined as the user identifier included in the target subtask; next, the identifier representing "unassigned" is determined as the intermediate state identifier; then, the position number of the advertising consumption segmentation data in the above advertising consumption segmentation data sequence is determined as the page number included in the target subtask; finally, the timestamp of the current time is determined as the creation timestamp included in the target subtask.

[0055] The fourth sub-step involves parsing and processing the total advertising consumption data to generate advertising consumption data. In practice, the computing node can identify the first advertising consumption segment in the advertising consumption segment data sequence corresponding to the total advertising consumption data as the advertising consumption data.

[0056] Therefore, when a slave compute node calls the API to retrieve total ad consumption data, it can store it in a local cache. Other slave compute nodes can then retrieve the consumption data directly from the local cache without needing to call the API again. Since the ad data sorting tasks representing N pagination subtask nodes are simultaneously sent to M slave compute nodes (the ad compute nodes corresponding to the filtered information) for parallel processing, and the API calls occur in parallel, compared to the existing technology of serial for loop pagination (where N pages require N sequential API calls, taking tens of minutes), this invention changes the serial process to parallel subtasks, reducing the time spent on API calls from O(N) to O(N / M), and linearly improving throughput by increasing the horizontal scaling of Workers.

[0057] Step 105: Write the advertising consumption data from the computing node to the consumption database, and update the advertising task tree and the node status corresponding to the computing node.

[0058] In some embodiments, the aforementioned slave computing node writes the aforementioned advertising consumption data into the consumption database, and updates the aforementioned advertising task tree and the node status corresponding to the aforementioned slave computing node. The consumption database can be a database used to store advertising consumption data. For example, the consumption database can be a MySQL (relational database) database. In practice, firstly, the slave computing node can write the aforementioned advertising consumption data into the consumption database using a preset data writing algorithm. Secondly, the intermediate state identifiers of the subtasks in the aforementioned advertising task tree corresponding to the aforementioned advertising consumption data are updated to identifiers representing "completed," thereby updating the advertising task tree. Next, the node status corresponding to the aforementioned slave computing node is updated to "idle," thereby updating the slave computing node information set that can be stored by the local coordination storage component. For example, the preset data writing algorithm can be insert.

[0059] Optionally, the above method further includes: The first step involves the master compute node controlling the master timed detection thread to read the slave heartbeat information set and slave compute node information set stored in the local coordination storage component. The slave heartbeat information set includes a heartbeat timestamp, with one slave heartbeat corresponding to one slave compute node. The slave compute node information set includes node status. The slave heartbeat information set may also include a slave node identifier. The master timed detection thread can be a thread within the master compute node used for timed reading. For example, the master timed detection thread can read the slave heartbeat information set and slave compute node information set every 5 seconds.

[0060] The second step involves the primary computing node determining the heartbeat difference for each heartbeat in the aforementioned heartbeat information set based on the current timestamp, thus obtaining a heartbeat difference set. Here, the current timestamp can be a timestamp representing the current time. In practice, firstly, for each heartbeat in the aforementioned heartbeat information set, the primary computing node can determine the heartbeat difference as the difference between the current timestamp and the heartbeat timestamps included in the heartbeat information. Secondly, the primary computing node can define the determined heartbeat differences as a heartbeat difference set.

[0061] Third, for each heartbeat difference in the aforementioned heartbeat difference set, the main computing node, in response to determining that the heartbeat difference is greater than a preset difference, performs the following processing sub-steps: The first sub-step involves deleting the slave compute node information corresponding to the aforementioned heartbeat difference from the aforementioned slave compute node information set. For example, the preset difference could be 30 seconds. In practice, the aforementioned master compute node can delete the slave compute node information corresponding to the aforementioned heartbeat difference from the aforementioned slave compute node information set to update the slave compute node information set stored in the aforementioned local coordination storage component.

[0062] The second sub-step involves updating the advertising task tree based on the aforementioned heartbeat difference. In practice, the main computing node can update the intermediate state identifier of each subtask in the advertising task tree corresponding to the aforementioned heartbeat difference with an identifier representing "unassigned" to update the advertising task tree.

[0063] Optionally, the above method further includes: The first step involves the target compute node controlling a timed monitoring thread to monitor the set of slave compute node information stored in the local coordination storage component. The target slave compute node is any advertising compute node in the advertising compute node set other than the primary compute node. Here, the timed monitoring thread can be a thread within the slave compute node used to periodically monitor the slave compute node information set. For example, the timed monitoring thread can read the slave compute node information set every 5 seconds.

[0064] In the second step, in response to determining that the aforementioned slave computing node information set meets a preset missing condition, the aforementioned target slave computing node restarts the aforementioned target slave computing node and adds the target slave computing node information corresponding to the aforementioned target slave computing node to the aforementioned slave computing node information set. The preset missing condition may be: the slave computing node information set does not include slave computing node information corresponding to the aforementioned target slave computing node. The target slave computing node information corresponding to the aforementioned target slave computing node may include, but is not limited to, at least one of the following: target slave node identifier, target node status, and target channel identifier set. The target slave node identifier can uniquely identify the target slave computing node. The target node status can represent "idle". The target channel identifier set can uniquely identify a channel.

[0065] Therefore, in existing technologies, task loss occurs after a Worker crash, requiring manual detection and recovery, which can take minutes to hours. This invention achieves a two-layer fault tolerance system of automatic detection, reallocation, and retries within seconds, with Zookeeper persistence ensuring no task loss, all without requiring manual intervention.

[0066] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0067] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for synchronizing advertising data, characterized in that, include: The encapsulation system determines the advertising task queue based on a pre-acquired set of advertising request information; The main computing node extracts the initial advertising task from the advertising task queue determined by the encapsulation system; The master computing node maintains an advertising task tree based on the initial advertising task, and sends advertising data extracted from the advertising task tree to the associated slave computing nodes. In response to receiving an advertising data task, the computing node obtains advertising consumption data based on the target advertising data task corresponding to the advertising data task. The slave computing node writes the advertising consumption data into the consumption database and updates the advertising task tree and the node status corresponding to the slave computing node.

2. The method according to claim 1, characterized in that, The master computing node is elected from the advertising computing nodes through a preset master-slave election algorithm. The slave computing node is the advertising computing node in the advertising computing node set other than the master computing node.

3. The method according to claim 2, characterized in that, The step of maintaining an advertising task tree based on the initial advertising task, and sending advertising data extracted from the advertising task tree to the associated slave computing node, includes: The initial advertising task and the preset subtasks are written into the advertising task tree to maintain the advertising task tree; The control channel thread scans the advertising task tree to obtain a set of advertising data subtasks; The advertising data sub-task set is prioritized to generate an advertising data sorting task sequence; Information on each slave compute node that meets the preset node conditions is selected from the local coordinated storage component and used as the slave compute node selection information set. The first advertising data sorting task in the advertising data sorting task sequence is sent as an advertising data task to the associated slave computing node. Here, the associated slave computing node can be any advertising computing node corresponding to any slave computing node filtering information in the slave computing node filtering information set.

4. The method according to claim 3, characterized in that, The advertising data task includes: an intermediate state identifier; and the target advertising data task is determined through the following steps: In response to receiving the advertising data task, the slave computing node sends a first confirmation signal to the master computing node; In response to receiving the first confirmation signal, the master computing node sends a second confirmation signal to the slave computing node; In response to receiving the second confirmation signal, the slave computing node sends a third confirmation signal to the master computing node; In response to receiving the third confirmation signal, the master computing node sends a fourth confirmation signal to the slave computing node; In response to receiving the fourth confirmation signal, the slave computing node updates the intermediate state identifier included in the received advertising data task to update the advertising data task, obtain the target advertising data task, and update the slave computing node information corresponding to the slave computing node.

5. The method according to claim 1, characterized in that, The step of obtaining advertising consumption data based on the target advertising data task corresponding to the advertising data task includes: The target advertising data task is parsed and processed to generate parsed information; Based on the parsed information, a task token is retrieved from a pre-determined task token storage. In response to determining that the task token meets the preset token normal conditions, initial advertising consumption data is extracted from the local cache based on the parsed information; In response to determining that the initial ad consumption data is not empty, the initial ad consumption data is parsed based on the parsing information to generate ad consumption data; In response to determining that the initial ad consumption data is empty, the following invocation steps are performed: Based on the parsed information, the consumption interface is called to obtain the total advertising consumption data; The total advertising consumption data is stored in the local cache. The advertising task tree is updated based on the total advertising consumption data. The total advertising consumption data is parsed and processed to generate advertising consumption data.

6. The method according to claim 3, characterized in that, The method further includes: The master computing node controls the master timer detection thread to read the slave heartbeat information set and slave computing node information set stored in the local coordination storage component. The slave heartbeat information set includes: heartbeat timestamp, one slave heartbeat information corresponds to one slave computing node information, and the slave computing node information set includes: node status. The master computing node determines the heartbeat difference value corresponding to each heartbeat information in the heartbeat information set based on the current timestamp, thus obtaining the heartbeat difference value set; For each heartbeat difference in the heartbeat difference set, in response to determining that the heartbeat difference is greater than a preset difference, the main computing node performs the following processing steps: Remove the slave node information corresponding to the heartbeat difference from the slave node information set; The advertising task tree is updated based on the heartbeat difference.

7. The method according to claim 6, characterized in that, The method further includes: The target slave computing node controls the timed detection thread to monitor the slave computing node information set stored in the local coordination storage component, wherein the target slave computing node is any advertising computing node in the advertising computing node set other than the master computing node; In response to determining that the slave computing node information set meets a preset missing condition, the target slave computing node restarts the target slave computing node and adds the target slave computing node information corresponding to the target slave computing node to the slave computing node information set.

Images