User information synchronization method and system based on multiple queues

Abstract

The invention discloses a user information synchronization method and system based on multiple queues, and belongs to the technical field of data synchronization, and the method comprises the steps: obtaining a synchronization information request of a user, and generating a to-be-processed event; calculating a priority score of the to-be-processed event; dividing to-be-processed event types according to a threshold range in which the priority scores of the to-be-processed events are located; according to the type of the to-be-processed event, if the to-be-processed event is a high-priority event, entering a high-priority queue for processing; if the priority event is a common priority event, entering a domain exclusive queue for processing; and if the event is a low-priority event, entering a low-priority queue for processing. According to the method provided by the invention, differentiated processing is carried out on event synchronization requirements of different priorities, and the processing efficiency, the stability and the data security of user information synchronization are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, and in particular to a user information synchronization method and system based on multiple queues. Background Technology

[0002] In modern information systems, user information is usually stored in multiple different databases or data storage units. In order to ensure that the user information obtained by users is consistent and real-time when using the system in different scenarios, it is necessary to synchronize these scattered user information.

[0003] Traditional user information synchronization methods often use a single queue to handle all user information synchronization tasks. With this approach, when the volume of user information data is large, or when there are various types and priorities of user information synchronization needs within the system, a single queue is prone to task backlog and low processing efficiency. Because all tasks are processed sequentially according to the order they enter the queue, it's impossible to differentiate between user information synchronization tasks of different urgency or importance. This results in some critical, urgently needed user information potentially waiting a long time to be processed, thus impacting overall system performance and user experience.

[0004] Furthermore, a single queue also faces performance bottlenecks when handling high-concurrency user information synchronization requests. As the number of system users continues to increase, operations such as updating, adding, and deleting user information become more frequent. A single queue struggles to efficiently handle a large number of concurrent requests, easily leading to queue congestion and further exacerbating the delay in user information synchronization. Summary of the Invention

[0005] The purpose of this invention is to propose a user information synchronization method based on multiple queues, which aims to solve the problems in existing information systems where user information synchronization uses a single queue to process all tasks, resulting in the inability to differentiate synchronization needs of different priorities, easy task backlog and processing delays, and difficulty in coping with queue congestion caused by high concurrency requests.

[0006] This invention is implemented as follows: a user information synchronization method based on multiple queues, the method comprising: Obtain user synchronization information requests and generate pending events; Calculate the priority score of the events to be processed; The types of events to be processed are classified according to the threshold range in which the priority scores of the events to be processed fall; Depending on the type of event to be processed If it is a high-priority event, it will be processed in the high-priority queue. If it is a normal priority event, it will be processed in a domain-specific queue; If it is a low-priority event, it will be processed in the low-priority queue.

[0007] Another objective of this invention is to propose a user information synchronization system based on multiple queues.

[0008] The system includes: The pending event generation module is used to obtain users' synchronization information requests and generate pending events; The pending event priority score calculation module is used to calculate the priority score of pending events; The pending event type classification module is used to classify pending events into types based on the threshold range of the priority scores of the pending events; The routing module processes events based on their type: high-priority events are processed in a high-priority queue; normal-priority events are processed in a domain-specific queue; and low-priority events are processed in a low-priority queue.

[0009] Beneficial effects of the present invention This invention discloses a user information synchronization method and system based on multiple queues. The method comprises: acquiring user synchronization information requests and generating pending events; calculating the priority score of the pending events; classifying the pending events into types according to the threshold range of their priority scores; and, based on the type of the pending event, processing high-priority events in a high-priority queue, processing ordinary-priority events in a domain-specific queue, and processing low-priority events in a low-priority queue. This invention enables differentiated processing of synchronization requests for events of different priorities, effectively improving the processing efficiency, stability, and data security of user information synchronization. Attached Figure Description

[0010] Figure 1 This is a flowchart of a user information synchronization method based on multiple queues according to a preferred embodiment of the present invention; Figure 2 This is a structural diagram of a user information synchronization system based on multiple queues, according to a preferred embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. For ease of explanation, only the parts related to the embodiments of this invention are shown. It should be understood that the specific embodiments described herein are merely for explaining this invention and are not intended to limit this invention.

[0012] This invention proposes a user information synchronization method and system based on multiple queues. The method involves: acquiring user synchronization requests and generating pending events; calculating the priority score of each pending event; classifying pending events into types based on the threshold range of their priority scores; and, depending on the type of the pending event, processing high-priority events in a high-priority queue, processing ordinary-priority events in a domain-specific queue, and processing low-priority events in a low-priority queue. This invention enables differentiated processing of synchronization requests for events of different priorities, effectively improving the processing efficiency, stability, and data security of user information synchronization.

[0013] Figure 1 This is a flowchart of a user information synchronization method based on multiple queues according to a preferred embodiment of the present invention; the method includes: S1, obtain the user's synchronization information request and generate pending events; The metadata of the event body to be processed includes a globally unique transaction ID (GTID), timestamp, event status, etc., and the associated information of the event to be processed includes user level, business type, data sensitivity, event urgency, etc. User levels include regular users, VIP users, diamond users, etc. Business types include core businesses (such as payment, orders, password modification), non-core businesses (such as information query, logs, login, data update), and peripheral businesses (such as advertising push). Data sensitivity includes low sensitivity (such as basic information), medium sensitivity, high sensitivity (such as privacy information), and top secret. Event urgency is categorized into real-time (e.g., flash sale orders), near real-time (e.g., user information updates), and non-real-time (e.g., monthly statistics). S2, calculate the priority score of the event to be processed; In this embodiment of the invention, calculating the priority of the event to be processed includes the following steps (S21-S23): S21, Select at least one piece of related information as the dimension of the calculation; In this embodiment of the invention, the selected associated information is user level, business type, data sensitivity, and event urgency. S22, score each dimension according to the preset scoring rules and set the weight coefficient of the dimension; In this embodiment of the invention, the scores for each dimension are determined according to preset scoring rules; the coefficients for each dimension are preset according to their importance.

[0014] For example, the scoring rules for each dimension are as follows: User levels: Regular users: 1-3 points; VIP1-VIP3: 4-6 points; VIP4-VIP5: 7-8 points; Diamond users: 9-10 points; Business type: Core business (payment, order processing, password modification): 8-10 points; Non-core business (information query / log): 4-7 points; Peripheral business (ad push): 1-3 points; Data sensitivity: Low sensitivity (e.g., basic information) 1-3 points; Medium sensitivity 4-6 points; High sensitivity 7-8 points; Top secret, etc. 9-10 points; Event urgency: Real-time (e.g., flash sale orders): 9-10 points; Near real-time (e.g., user information updates): 5-8 points; Non-real-time (e.g., monthly statistics): 1-4 points; The user coefficient is 40%, the business coefficient is 30%, the urgency coefficient is 20%, and the additional coefficient is 10%.

[0015] In actual implementation, the specific values ​​for the score range of each dimension need to be refined by combining the details of the business scenario and data characteristics, and further divided by interval + subdivision rules; For example: User level dimensional classification The user is divided into three categories: regular users (1-3 points), VIP1-VIP3 (4-6 points), VIP4-VIP5 (7-8 points), and diamond users (9-10 points).

[0016] Segmentation rules: Within each interval, specific scores are determined based on secondary characteristics such as user activity and historical contributions. Regular users: Low-frequency activity (less than 3 logins per month) → 1 point; Medium-frequency activity (3-10 logins per month) → 2 points; High-frequency activity (more than 10 logins per month) → 3 points; For example: Business type dimension classification Division of areas: Core business (8-10 points), Non-core business (4-7 points), Peripheral business (1-3 points); Segmentation rules: Segment by scope of business impact, operational complexity, or revenue relevance: Core business (payment / orders): Order creation (triggering the entire subsequent process) → 10 minutes; Payment completed (funds received) → 9 points; Order status update (e.g., shipped) → 8 points; In practical implementation, the specific values ​​of the score range for each dimension can also be achieved through dynamic rules. The flexible adaptation of scores can be achieved through "rule engine + real-time data source + dynamic parameters". This technology is a commonly used technical means in this field and will not be elaborated here. In practice, the coefficients of each dimension can be preset or dynamically configured. The method of dynamically configuring the coefficients of each dimension is a common technique in this field and will not be elaborated here. S23, calculate the priority score of the event to be processed by weighting the dimension scores; In this embodiment of the invention, the priority score calculation formula is as follows: Priority score = (User level score × User level coefficient) + (Business type score × Business type coefficient) + Data sensitivity score × Data sensitivity coefficient + (Event urgency score × Event urgency coefficient); S3, classify the types of events to be processed according to the threshold range of the priority scores of the events to be processed; specifically: If the priority score is greater than or equal to the first threshold, it is determined to be a high-priority event; If the second threshold is less than the priority score and less than the first threshold, then it is determined to be a normal priority event; If the priority score is less than the second threshold, it is determined to be a low-priority event; For example: High-priority events: Priority score ≥ 30; Normal priority events: 15 ≤ priority score < 30; Low priority events: priority score < 15; S4, based on the type of event to be processed. If it is a high-priority event, it will be processed in the high-priority queue. If it is a normal priority event, it will be processed in a domain-specific queue; If it is a low-priority event, it will be processed in the low-priority queue. The domain-specific queues are isolated according to the target domain (such as basic information class, privacy information class, etc.), and different information is placed in each category, which also facilitates independent expansion; The processing steps after entering the high-priority queue include: S4a1 verifies the integrity of the event body metadata (including GTID, timestamp, etc.), filters invalid or duplicate events, and writes them to a high-priority queue after passing the verification. The method for filtering invalid or duplicate events is as follows: Based on the business domain to which the event belongs (such as order domain or payment domain), select the local idempotent table of the corresponding storage medium (Redis cache is preferred for order domain, and MySQL database is used for payment domain). Establish an idempotent index structure with "GTID + timestamp" as the deduplication key; check the target domain idempotent index through atomic query operations (such as database INSERT ... ON DUPLICATE KEY UPDATE or cached SETNX command), and set a short query timeout (such as 100ms). If there is no corresponding record in the index, the thread directly executes the event handling logic and writes it to the index record after completion (status is "completed"). If a record exists in the index and its status is "completed", the thread determines it as a duplicate event, returns the processing result directly, and does not execute the business logic. If a record exists in the index and its status is "processing", the thread obtains the final processing result through a spin-wait mechanism (non-blocking); In this embodiment of the invention, the idempotent index is independent of the main business data index and is used only for event deduplication determination. The index record contains event processing status identifiers (such as "not processed", "processing", "completed").

[0017] S4a2 enables an independently configured high-priority consumer thread pool (e.g., fixed 4 core threads) to listen for event enqueue signals in high-priority queues in real time. S4a3: When a new event is detected in the high-priority queue, the event is retrieved from the high-priority queue according to the FIFO + priority sorting rule (high-priority events in the same queue are retrieved first); S4a4: After the consumer thread extracts the event, it initiates a request to call the target domain interface. The target domain interface call request uses transport protocols such as HTTP / gRPC, and the target domain interface request parameters carry the core data of the event body to be processed (including GTID, business parameters, context information, etc.). S4a5 allows setting a short timeout threshold (e.g., 500ms) when making API calls. If the timeout threshold is reached, the current request is terminated. S4a6, if the consumer thread receives a successful ACK response from the target domain, it submits the message offset of the event in the queue, updates the consumption progress, and marks the "processing completed" status in the event log. If the interface call times out or a 5xx error is received in the target domain (such as 500 Internal Service Error or 503 Service Unavailable), the consumer thread will mark the event as "retryable", carry the reason for failure (such as "Network timeout on 2025-11-11 10:00:01"), and transfer it to the retry queue. If the interface call receives a 4xx error in the target domain (such as 400 invalid parameter, 404 interface does not exist), the consumer thread determines it as a "business logic error" and directly transfers the event with the error code and error description to the dead letter queue (DLQ). If a high-priority event fails to trigger, an alarm message (including the event GTID, failure type, target domain, and failure time) is generated and pushed to the relevant personnel via SMS, WeChat Work, or other channels.

[0018] The processing steps after entering the domain-specific queue include: S4b1 configures an independent consumer group for each domain-specific queue (such as the basic information domain queue and the privacy information domain queue); each group supports a maximum of 16 threads of concurrent consumption capability. S4b2 collects the current capacity metrics (such as QPS, response time, and error rate) of the target domain interface in real time, and dynamically adjusts the number of consumer threads in the corresponding domain queue according to the maximum capacity of the interface. S4b3, for high-throughput domain queues (such as basic information domain queues), the consumer thread pulls events in batches (e.g., according to the rule of 10 items / batch), encapsulates them into HTTP / gRPC requests and sends them to the target domain interface; for sensitive domain queues (such as privacy information domain queues), the consumer thread processes events by pulling them one by one and sending them serially. S4b4, after receiving an event in the target domain, extracts the globally unique event ID (GTID) from the event body and queries the corresponding status of the GTID in a local idempotent table (such as Redis) using atomic operations: If the status corresponding to the GTID is "completed", it is determined to be a duplicate event, and a success response is returned directly; If the status corresponding to the GTID is "processing", choose to wait or reject according to the business strategy; If it does not exist, the status corresponding to the GTID will be set to "Processing" and subsequent business logic will be executed. If successful, it will be updated to "Completed".

[0019] Once the target domain (for the entire batch or a single event) returns a successful response, the domain queue consumer thread commits the corresponding message offset, updates the consumption progress, and marks the event processing as complete. If the target domain returns a non-successful response For network timeouts, 5xx errors, or retryable 4xx errors (such as 429), mark the event as "retryable" and transfer it to the retry queue; for unrecoverable 4xx errors (such as 400, 404), determine it as a "business logic error" and transfer it to the dead letter queue.

[0020] Real-time monitoring of queue depth (number of backlogged events) for each domain. When the backlog exceeds the dynamically configured threshold, the rate at which events are pulled from the event source for that domain is automatically reduced (instead of being completely paused), achieving smooth backpressure control and preventing resource exhaustion or event loss.

[0021] The processing steps after entering the low-priority queue include: S4c1 verifies the integrity of the event body metadata fields (including GTID, target domain identifier, and business type), filters invalid or duplicate events, and writes them to a low-priority queue after passing the verification. The method for filtering invalid or duplicate events is the same as that in the high-priority queue, and will not be described again here. S4c2 determines whether to enable delayed batch processing mode based on the event business type (such as advertising push and log summary marked as batch processing, and low-frequency user data supplementation marked as single processing). Batch processing events enter the batch processing candidate pool, and single processing events directly enter the ordinary consumption queue (the queue of single processing events in the low priority queue). S4c3 configures a delayed processing time window for events entering the batch processing candidate pool and marks the delayed trigger time in the event metadata; S4c4 monitors the batch processing candidate pool in real time. When the delay time window is reached or the cumulative number of events reaches the batch processing threshold (e.g., 1000 items / batch), the deferred batch processing process is triggered; otherwise, it continues to wait. S4c5 groups candidate events by business domain and data type (e.g., advertising domain - e-commerce category ads, log domain - user access logs), batch filters expired events (exceeding the 24-hour timeout threshold) and duplicate events (based on GTID deduplication), and generates batch processing tasks; S4c6 calls the minimum resource pool of the low-priority queue (e.g., fixed 1-2 consumer threads, occupying ≤10% of the system CPU / memory); if the system has idle resources during off-peak hours (high-priority queue idle rate >60%), it can be temporarily expanded to a specific number (e.g., 4 threads) to execute batch processing tasks. S4c7 (via HTTP / gRPC batch interface) sends grouped events to the target domain (such as the batch push interface for the advertising domain or the batch storage interface for the log domain), sets a timeout threshold (such as 5000ms), and adapts to the non-real-time characteristics of low-priority events. S4c8 receives batch ACK responses (such as {"code":200, "ack":true, "successCount":980, "failCount":20}) returned by the target domain. Successful events commit message offsets and clean up the queue, while failure events (such as user logout causing ad push failure) are directly marked as discarded. S4c9 uses a FIFO + sleep-wake consumption mode for single events that are not batch-processed, calls the target domain's single interface for processing, and cleans up the event immediately after processing is complete; The FIFO + sleep-wake consumption mode refers to a consumption mode that determines the processing order of single events in a low-priority queue that have not enabled batch processing according to the first-in-first-out (FIFO) rule, while controlling the consumer thread to adopt a periodic scheduling mechanism of sleeping for a preset duration (such as 5 seconds) and then waking up and executing the event retrieval and processing for a second preset duration (such as 1 second) after waking up, so as to complete the processing of single events in a low-resource-consumption manner.

[0022] S4c10: After the batch processing task is completed, the temporarily expanded resources will be returned to the minimum resource pool. Furthermore, batch processing logs (such as trigger time, processing quantity, and time consumption) can be written to a low-quality event audit table and retained for a specific period (such as 30 days) for subsequent traceability. The steps for entering the retry queue include: S4d1 integrates delayed queue components (such as the RocketMQ delayed message module or the Redis ZSet data structure) to provide delayed storage and triggering capabilities for retry events; The delayed queue is physically isolated from the main queue (priority queue, domain-specific queue) to prevent retry events from consuming main queue resources. S4d2, update the retry count field; The current number of retries is recorded in the retry_count field of the event header. When the event first enters the retry queue, this field is set to 1, and it is automatically incremented by 1 before each subsequent retry, ensuring that each retry operation can be traced back to the current retry round. S4d3 calculates the delay time according to the exponential backoff strategy based on the number of times the event has failed for the first time or has been retried. The exponential backoff strategy is as follows: after a failed event enters the retry queue, it is exponentially backed off at 2ⁿ seconds, where n is the number of retries, and ±10% random jitter is added. The maximum backoff time (e.g., 300 seconds) and the maximum number of retries (e.g., 5 times) are set. If the maximum backoff time is exceeded, it is transferred to the dead letter queue.

[0023] For example, the first failure is delayed by 2 seconds, the second failure by 4 seconds, the third failure by 8 seconds, the fourth failure by 16 seconds, and the fifth failure by 32 seconds, and the maximum delay for all retry times shall not exceed 300 seconds. S4d4 stores events with calculated delay durations into a delay queue, configures the delay queue to automatically trigger consumption after the preset delay duration is reached, and the consumption process is executed through an independent thread pool. In S4d5, when the delay time is reached, the independent consumer thread of the retry queue pulls the event and immediately calls the target domain interface to initiate a retry; at the same time, it records the trigger time of this retry, the reason for failure (such as network timeout, 5xx error) and the actual delay time. S4d6 performs a retry count limit check; After each retry, the value of the retry_count field in the event header is read. If the value is less than the preset threshold (e.g., 5), it is determined that the maximum number of retries has not been reached, the next round of exponential backoff delay time is recalculated, and the device is returned to the retry queue to wait for the next retry. If the value is greater than or equal to the preset threshold (e.g., 5), it is determined that the limit has been exceeded. S4d7 If it is an over-limit event, the retry process is terminated and the event is directly moved to the dead letter queue (DLQ); at the same time, the reason for the transfer of "retry over-limit" is marked in the event metadata; Full-process retry logging: Every retry operation (delay calculation, consumption trigger, interface call, count update, over-limit transfer) is recorded in the audit log. The log content includes the event GTID, retry time, failure reason, delay duration, and retry_count value. The log data is persistently stored (e.g., written to the ELK log system) to meet the needs of subsequent auditing and problem investigation.

[0024] The steps for entering the dead letter queue include: S4e1 writes dead-letter events to a pre-defined persistent storage medium (such as a MySQL database or a Kafka standalone dead-letter topic). S4e2: When a new dead letter event is written to the dead letter queue, an immediate alarm is triggered. In this embodiment of the invention, alarm information can be pushed to the operation and maintenance or business manager through DingTalk, Lark, WeChat Work or SMS channels; the alarm information includes core information such as event GTID, failure type, and transfer time. Furthermore, in this embodiment of the invention, the step of entering the dead letter queue for processing further includes: marking the reason for the dead letter failure and associating it with event metadata; That is, each dead letter event is automatically or manually marked with a specific failure reason and associated with the event metadata to form a structured failure reason record; the failure reasons include network timeout, interface change, data format error, target system crash, etc. Furthermore, embodiments of the present invention also provide a Web visualization console and a standardized API interface, which support inputting event GTID or filtering conditions (such as time range, failure type) to query the complete details of dead letter events (including event body, retry records, and failure logs). Through the Web console or API interface, users can select a single, partial, or all dead-letter events, specify the target resubmission queue (such as a domain-specific queue or a priority queue), trigger a one-click resubmission operation, and re-inject the dead-letter events into the event handling process. Furthermore, in this embodiment of the invention, a daily summary report of dead letters is generated; Specifically, the system automatically compiles daily DLQ event data at fixed times (e.g., 2 AM), generates a summary report, and includes information such as the total number of dead letters, the percentage of each failure type (e.g., network timeout accounts for 30%, interface change accounts for 20%), and the target domains of high-frequency failures, and pushes the report to designated personnel. Furthermore, in this embodiment of the invention, the failure reason data marked in the dead letter queue is periodically summarized and analyzed, and the analysis results are fed back to the technical team for optimizing event handling rules (such as adjusting retry strategies) or fixing bugs in the target domain interface. In this embodiment of the invention, dead letter event lifecycle management rules are pre-configured, specifically: dead letter data is set to be retained in the dead letter queue for a specific period of time (e.g., 30 days). After the expiration, the system automatically performs preset operations to transfer dead letter events that meet the conditions to archive storage media (e.g., cold backup database) or directly delete them physically, thereby releasing storage resources. Corresponding to the user information synchronization method based on multiple queues described in the above embodiments, Figure 2 This diagram illustrates a user information synchronization system architecture based on a multi-queue system according to an embodiment of this application. For ease of explanation, only the parts relevant to this embodiment are shown. The system includes: The pending event generation module is used to obtain users' synchronization information requests and generate pending events; The pending event priority score calculation module is used to calculate the priority score of pending events; The pending event type classification module is used to classify pending events into types based on the threshold range of the priority scores of the pending events; The routing module processes events based on their type: high-priority events are processed in a high-priority queue; normal-priority events are processed in a domain-specific queue; and low-priority events are processed in a low-priority queue.

[0025] The metadata of the event body to be processed includes a globally unique transaction ID (GTID), timestamp, event status, etc., and the event association information includes user level, business type, data sensitivity, event urgency, etc. The process involves classifying the events to be processed into different types based on the threshold range of their priority scores; specifically: If the priority score is greater than or equal to the first threshold, it is determined to be a high-priority event; If the second threshold is less than the priority score and less than the first threshold, then it is determined to be a normal priority event; If the priority score is less than the second threshold, it is determined to be a low-priority event; Furthermore, the pending event priority score calculation module includes... The dimension selection module is used to select at least one piece of related information as the dimension for calculation. The dimension scoring and weight setting module is used to score each dimension according to preset scoring rules and set the weight coefficient of the dimension. The priority calculation module is used to calculate the priority score of the event to be processed based on the weighted average of the dimension scores. In this embodiment of the invention, the priority score calculation formula is as follows: Priority score = (User level score × User level coefficient) + (Business type score × Business type coefficient) + Data sensitivity score × Data sensitivity coefficient + (Event urgency score × Event urgency coefficient); Furthermore, the domain-specific queues are isolated according to the target domain (such as basic information class, privacy information class, etc.), and different information is placed in each category. Furthermore, the high-priority queue includes The event enqueue verification unit is used to verify the integrity of the event body metadata (including fields such as GTID and timestamp), filter invalid or duplicate events, and write them into a high-priority queue after passing the verification. The high-priority consumer thread pool startup unit is used to enable an independently configured high-priority consumer thread pool and listen for event enqueue signals in the high-priority queue in real time. The event real-time retrieval unit is used to retrieve events from the high-priority queue when a new event is detected, according to the FIFO + priority sorting rule (higher priority events in the same queue are retrieved first). The target domain interface immediate invocation unit is used by the consumer thread to initiate a target domain interface invocation request after retrieving the event; The short timeout control configuration unit is used to set a short timeout threshold when making an interface call. If the timeout threshold is reached, the current request will be terminated. The first event processing unit is used to start the consumer thread to submit the message offset of the event in the queue, update the consumption progress, and mark the "processing completed" status in the event log after receiving a successful ACK response from the target domain. If the API call times out or a 5xx error is received in the target domain, the consumer thread is started to mark the event as "retryable", carrying the reason for failure, and transfer it to the retry queue. If the API call receives a 4xx error in the target domain, the consumer thread is started to determine it as a "business logic error" and the event is directly transferred to the dead letter queue along with the error code and error description. If a high-priority event fails to trigger, an alarm message is generated and pushed to the relevant responsible person.

[0026] Furthermore, the domain-specific queue includes: The independent consumer group initialization unit is used to configure an independent consumer group for each domain-specific queue (such as the basic information domain queue and the privacy information domain queue); each group supports a maximum of 16 threads of concurrent consumption capability; The dynamic thread count adjustment unit is used to collect the current capacity metrics (such as QPS, response time, and error rate) of the target domain interface in real time, and dynamically adjust the number of consumer threads in the corresponding domain queue according to the maximum capacity of the interface. The high-throughput domain batch sending processing unit is used to batch pull events from high-throughput domain queues (such as basic information domain queues) by consumer threads, and batch encapsulate them into HTTP / gRPC requests to send to the target domain interface. The sensitive field single-serial processing unit is used for the queue of sensitive characteristics (such as the privacy information field queue). The consumer thread processes events by pulling one by one and sending them serially. The target domain local idempotency verification unit is used to extract the globally unique event ID (GTID) from the event body after receiving an event in the target domain, and query the status corresponding to the GTID in the local idempotency table (such as Redis) with an atomic operation. If the status corresponding to the GTID is "completed", it is determined to be a duplicate event and a success response is returned directly. If the status corresponding to the GTID is "processing", the system chooses to wait or reject according to the business strategy. If it does not exist, the status corresponding to the GTID is set to "processing" and subsequent business logic is executed. After success, it is updated to "completed".

[0027] The second event processing unit is used to submit the corresponding message offset, update the consumption progress, and mark the event processing as complete when the target domain (for the whole batch or a single event) returns a successful response. If the target domain returns a non-successful response: For network timeouts, 5xx errors, or retryable 4xx errors (such as 429), mark the event as "retryable" and transfer it to the retry queue; for unrecoverable 4xx errors (such as 400, 404), determine it as a "business logic error" and transfer it to the dead letter queue.

[0028] The queue backpressure triggering and execution unit is used to monitor the queue depth (number of events backlogged) of each domain in real time. When the backlog exceeds the dynamically configured threshold, it automatically reduces the rate at which events are pulled from the event source for that domain (instead of pausing completely), thus achieving smooth backpressure control and preventing resource exhaustion or event loss.

[0029] Furthermore, the low-priority queue includes: The low-priority event enqueue verification unit is used to verify the integrity of the event body metadata fields (including GTID, target domain identifier, and business type), filter invalid or duplicate events, and write them into the low-priority queue after passing the verification. The delayed batch processing mode determination unit is used to determine whether to enable the delayed batch processing mode based on the event business type (such as advertising push and log summary marked as batch processing, and low-frequency user data supplementation marked as single processing). Batch processing events enter the batch processing candidate pool, and single processing events directly enter the ordinary consumption queue (the queue of single processing events in the low priority queue). The delay time window configuration unit is used to configure a delay processing time window for events entering the batch processing candidate pool and mark the delay trigger time in the event metadata; The batch processing trigger condition judgment unit is used to monitor the batch processing candidate pool in real time. When the delay time window is reached or the cumulative number of events reaches the batch processing threshold, the delayable batch processing process is triggered; otherwise, it continues to wait. The batch event aggregation and filtering unit is used to group candidate events by business domain and data type, filter expired and duplicate events in batches, and generate batch processing tasks. The low-priority resource pool invocation unit is used to invoke the minimum resource pool of the low-priority queue; if resources are idle during off-peak periods, it can be temporarily expanded to a specific number of threads to execute batch processing tasks.

[0030] The target domain batch interface call unit is used to send grouped events to the target domain, set timeout thresholds, and adapt to the non-real-time characteristics of low-priority events. The batch processing result ACK processing unit is used to receive batch ACK responses returned by the target domain. Successful events commit the message offset and clean up the queue, while failure events are directly marked as discarded. The single-event normal processing unit is used to process single events that are not batch processing enabled by using a FIFO + sleep-wake consumption mode, calling the target domain's single interface for processing, and immediately cleaning up the event after processing is completed. The FIFO + sleep-wake consumption mode refers to a consumption mode that determines the processing order of single events in low-priority queues that have not yet enabled batch processing, according to the first-in-first-out (FIFO) rule. At the same time, it controls the consumer thread to adopt a periodic scheduling mechanism that wakes up after a preset sleep time and then performs event retrieval and processing for a second preset time after wake-up, so as to complete the processing of single events in a low-resource-consumption manner.

[0031] The batch processing resource reclamation unit is used to reclaim temporarily expanded resources to the minimum resource pool after the batch processing task is completed. Furthermore, the retry queue includes: The delayed queue initialization configuration unit is used to integrate delayed queue components (such as the RocketMQ delayed message module or the Redis ZSet data structure) to provide delayed storage and triggering capabilities for retry events; the delayed queue is physically isolated from the main queue (priority queue, domain-specific queue) to avoid retry events occupying main queue resources. The retry count field update unit is used to update the retry count field; The current number of retries is recorded in the retry_count field of the event header. When the event first enters the retry queue, this field is set to 1, and it is automatically incremented by 1 before each subsequent retry, ensuring that each retry operation can be traced back to the current retry round. The Exponential Backoff Delay Calculation Unit is used to calculate the delay time based on the first failure or the number of retries of an event, according to the exponential backoff strategy. The exponential backoff strategy is as follows: after a failed event enters the retry queue, it is exponentially backed off at 2ⁿ seconds, where n is the number of retries, and ±10% random jitter is added. The maximum backoff time (e.g., 300 seconds) and the maximum number of retries (e.g., 5 times) are set. If the maximum backoff time is exceeded, it is transferred to the dead letter queue.

[0032] The delayed consumption trigger configuration unit is used to store events with calculated delay durations into a delay queue, configure the delay queue to automatically trigger consumption after the preset delay duration is reached, and the consumption process is executed through an independent thread pool; The retry event consumption execution unit is used to retrieve the event from the independent consumption thread of the retry queue when the delay time is reached, immediately call the target domain interface to initiate a retry, and record the trigger time, failure reason and actual delay time of this retry. The retry count limit judgment unit is used to judge whether the retry count limit has been exceeded. After each retry, the value of the retry_count field in the event header is read. If the value is less than the preset threshold, it is determined that the maximum number of retries has not been reached, the next round of exponential backoff delay time is recalculated, and the device is returned to the retry queue to wait for the next retry. If the value is greater than or equal to the preset threshold, it is determined that the limit has been exceeded. The dead-letter transfer unit for over-limit events is used to determine if an over-limit event is occurring. If so, the retry process is terminated, the event is directly moved into the dead-letter queue (DLQ), and the transfer reason of "retry over-limit" is marked in the event metadata. The retry full-process log recording unit is used to record each retry operation (delay calculation, consumption trigger, interface call, count update, over-limit transfer) to the audit log. The log content includes event GTID, retry time, failure reason, delay duration, retry_count value, and the log data is persistently stored. Furthermore, the dead-letter queue includes: The dead-letter event persistent storage unit is used to write dead-letter events to a preset persistent storage medium; The single dead letter instant alarm triggering unit is used to immediately trigger an instant alarm when a new dead letter event is written to the dead letter queue; Furthermore, the dead-letter queue also includes: The dead letter failure reason marking unit is used for marking dead letter failure reasons: automatically or manually marking the specific failure reason for each dead letter event and associating it with the event metadata to form a structured failure reason record; The Dead Letter Details Query Support Unit provides a web-based visual console and standardized API interface, allowing users to input the event GTID or filter conditions to query the complete details of dead letter events (including the event body, retry records, and failure logs). The One-Click Resend Operation Unit for Dead Letters is used for one-click resend operations: Through the Web console or API interface, it supports selecting a single, partial, or all dead letter events, specifying the target resend queue, triggering the one-click resend operation, and re-injecting the dead letter events into the event handling process. The reasons for failure include network timeout, interface change, data format error, target system crash, etc. The Dead Letter Daily Summary Report Generation Unit is used to generate daily summary reports of dead letters: the system automatically counts the DLQ event data of the day at fixed time periods every day, generates a summary report, which includes information such as the total number of dead letters, the proportion of each failure type, and the target domain of high-frequency failures, and pushes it to designated personnel. The Dead Letter Data Optimization and Analysis Application Unit is used to periodically summarize and analyze the failure reason data marked in the dead letter queue, and feed the analysis results back to the technical team to optimize event handling rules or fix bugs in the target domain interface. The Dead Letter Event Lifecycle Management Unit is used to configure the lifecycle rules for dead letter events: set the retention time of dead letter data in the dead letter queue, and after the expiration, the system will automatically perform preset operations to transfer dead letter events that meet the conditions to archive storage media or directly delete them physically, thereby releasing storage resources. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by program instructions and related hardware. The program can be stored in a computer-readable storage medium, such as ROM, RAM, disk, optical disk, etc.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for synchronizing user information based on multiple queues, the method comprising: The method includes: Obtain user synchronization information requests and generate pending events; Calculate the priority score of the events to be processed; The types of events to be processed are classified according to the threshold range in which the priority scores of the events to be processed fall; Depending on the type of event to be processed If it is a high-priority event, it will be processed in the high-priority queue. If it is a normal priority event, it will be processed in a domain-specific queue; If it is a low-priority event, it will be processed in the low-priority queue.

2. The user information synchronization method with multiple queues as described in claim 1, characterized in that, The metadata of the event body to be processed includes a globally unique transaction ID, timestamp, and event status; the associated information of the event to be processed includes user level, business type, data sensitivity, and event urgency.

3. The multi-queue user information synchronization method of claim 1, wherein, The calculation of the priority score for the events to be processed includes: Select at least one piece of related information as a dimension for calculation; Scoring is assigned to each dimension according to the preset scoring rules, and the weight coefficients of the dimensions are set. The priority score of the event to be processed is calculated by weighting the dimensional scores. The priority score calculation formula is as follows: Priority score = (User level score × User level coefficient) + (Business type score × Business type coefficient) + Data sensitivity score × Data sensitivity coefficient + (Event urgency score × Event urgency coefficient).

4. The multi-queue user information synchronization method of claim 1, wherein, The processing steps after entering the high-priority queue include: Verify the integrity of the event body metadata, filter invalid or duplicate events, and write events that pass the verification to a high-priority queue; Enable a separately configured high-priority consumer thread pool to listen for event enqueue signals in high-priority queues in real time; When a new event is detected in the high-priority queue, the event is retrieved from the high-priority queue according to the FIFO + priority sorting rule; After the consumer thread extracts the event, it initiates a request to call the target domain interface. Set a short timeout threshold when making an API call. If the timeout threshold is reached, terminate the current request. If the consumer thread receives a successful ACK response from the target domain, it submits the message offset of the event in the queue, updates the consumption progress, and marks the processing completion status in the event log. If the interface call times out or a 5xx error is received in the target domain, the consumer thread will mark the event as retryable, carry the reason for failure, and transfer it to the retry queue. If the interface call receives a 4xx error in the target domain, the consumer thread determines it to be a business logic error and directly transfers the event with the error code and error description to the dead letter queue. If a high-priority event fails to trigger, an alarm message will be generated and pushed to the relevant person in charge.

5. The multi-queue user information synchronization method of claim 1, wherein, The processing steps after entering the domain-specific queue include: Configure a separate consumer group for each domain-specific queue; Real-time collection of current capacity metrics of the target domain interface, and dynamic adjustment of the number of consumer threads in the corresponding domain queue based on the maximum capacity of the interface; For domain queues with high throughput, the consumer thread pulls events in batches, encapsulates them into HTTP or gRPC requests, and sends them to the target domain interface. For domain queues with sensitive characteristics, the consumer thread processes events by pulling them one by one and sending them serially. After receiving an event, the target domain extracts the GTID from the event body and queries the local idempotency table using atomic operations to determine the state corresponding to that GTID. If the status is "completed", it is determined to be a duplicate event, and a success response is returned directly; If the status is "Processing", choose to wait or reject according to the business strategy; If it does not exist, set the status of the GTID to "processing" and execute the subsequent business logic. If successful, update it to "completed". When the target domain returns a successful response, the domain queue consumer thread submits the corresponding message offset, updates the consumption progress, and marks the event processing as complete. If the target domain returns a non-successful response: For network timeouts, 5xx errors, or retryable 4xx errors, mark the event as "retryable" and move it to the retry queue; For unrecoverable 4xx errors, they are classified as "business logic errors" and transferred to the dead letter queue; Real-time monitoring of queue depth for each domain; when the backlog exceeds a dynamically configured threshold, the rate at which events for that domain are retrieved from the event source is automatically reduced.

6. The multi-queue user information synchronization method of claim 1, wherein, The processing steps after entering the low-priority queue include: Verify the integrity of the event body metadata fields, filter invalid or duplicate events, and write events that pass the verification to a low-priority queue. Based on the event business type, determine whether to enable the delayed batch processing mode. Batch-processable events enter the batch processing candidate pool, while single-processing events directly enter the normal consumption queue. Configure a delayed processing time window for events entering the batch processing candidate pool, and mark the delayed trigger time in the event metadata; The batch processing candidate pool is monitored in real time. When the delay time window is reached or the cumulative number of events reaches the batch processing threshold, the delayable batch processing process is triggered; otherwise, the process continues to wait. Candidate events are grouped by business domain and data type, expired and duplicate events are filtered in batches, and batch processing tasks are generated. Call the minimum resource pool of the low-priority queue; if resources are idle during off-peak hours, temporarily expand to a specific number of threads to execute batch processing tasks; Send the grouped events to the target domain, set a timeout threshold, and adapt to the non-real-time characteristics of low-priority events; Receive the batch ACK response returned by the target domain. On success, commit the message offset and clean up the queue. On failure, mark the message as discarded. For single events that are not batch processing enabled, a FIFO + sleep-wake consumption mode is used to call the target domain's single interface for processing, and the event is cleaned up immediately after processing is completed. The FIFO+sleep-wake consumption mode refers to determining the event processing order for single events in low-priority queues that have not enabled batch processing according to the first-in-first-out rule, while controlling the consumer thread to adopt a periodic scheduling mechanism of sleeping for a preset duration and then waking up and executing event retrieval and processing for a second preset duration after waking up. After the batch processing task is completed, the temporarily expanded resources will be returned to the minimum resource pool.

7. The multi-queue user information synchronization method as described in claim 4 or 6, characterized in that, The method for filtering invalid or duplicate events is as follows: Select the local idempotency table for the corresponding storage medium based on the business domain to which the event belongs; Establish an idempotent index structure with GTID + timestamp as the deduplication key; Use atomic query operations to search for idempotent indexes on the target domain, while setting a short query timeout; If there is no corresponding record in the index, the thread directly executes the event handling logic and writes it to the index record after completion. If a record exists in the index and its status is "completed", the thread determines it as a duplicate event, returns the processing result directly, and does not execute the business logic. If a record exists in the index and its status is "processing", the thread obtains the final processing result through a spin-wait mechanism.

8. The multi-queue user information synchronization method of claim 1, wherein, The steps for entering the retry queue include: Integrate delayed queue components; Update the retry count field; The delay time is calculated based on the number of initial failures or retries for an event, using an exponential backoff strategy. Events with calculated delay durations are stored in a delay queue. The delay queue is configured to automatically trigger consumption after the preset delay duration is reached, and the consumption process is executed through an independent thread pool. When the delay time is reached, the independent consumer thread of the retry queue pulls the event, calls the target domain interface to initiate a retry; at the same time, it records the trigger time, failure reason and actual delay time of this retry. Perform a retry limit check; If it is an over-limit event, the retry process will be terminated, the event will be moved directly to the dead letter queue, and the reason for the transfer "retry over-limit" will be marked in the event metadata. Each retry operation is recorded in the audit log, and the log data is persistently stored.

9. The multi-queue user information synchronization method of claim 1, wherein, The steps for entering the dead letter queue include: Write the dead letter event to the preset persistent storage medium; When a new dead letter event is written to the dead letter queue, an immediate alert is triggered.

10. A multi-queue user information synchronization system, characterized by, The system includes: The pending event generation module is used to obtain users' synchronization information requests and generate pending events; The pending event priority score calculation module is used to calculate the priority score of pending events; The pending event type classification module is used to classify pending events into types based on the threshold range of the priority scores of the pending events; The routing module processes events based on their type: high-priority events are processed in a high-priority queue; normal-priority events are processed in a domain-specific queue; and low-priority events are processed in a low-priority queue.

Images