User tag calculation method and system, electronic equipment and storage medium

Abstract

The invention relates to the field of user tag calculation, and provides a user tag calculation method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining a task request of user tag calculation; grouping and circling to-be-calculated user tag data under the triggering of the task request; and independently completing user label calculation on a plurality of groups obtained by group circling selection. In order to solve the defects of memory overflow, non-uniform resource utilization rate, weak fault-tolerant capability, complex system architecture and the like in related technologies, the scheme of the invention provides a high fault-tolerant distributed computing method and system based on grouping, circling and event driving, and related problems in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of user tag calculation technology, and in particular to a user tag calculation method and system, electronic device, and storage medium. Background Technology

[0002] In some sophisticated operational systems with large user bases, user tagging systems are critical infrastructure supporting core business functions such as personalized recommendations, precision marketing, and risk control. These systems need to provide calculation services for hundreds of tags for over 60 million users daily, processing over one billion tag calculation requests per day.

[0003] The related technologies have the following problems when performing tag calculations, which prevent them from fully meeting user needs: Memory overflow leads to service unavailability: Traditional tag computing loads all user data at the tag granularity, which easily exceeds the memory capacity of a single node, causing JVM memory overflow (OOM). Actual tests show that the computing node restart probability for high-coverage tag tasks exceeds 20%, and system availability is less than 80%, severely hindering the implementation of personalized marketing and other business applications. Uneven resource utilization and limited scalability: The number of users covered by different tags varies significantly (e.g., "newly registered users" only number in the tens of thousands, while the user base reaches tens of millions). The existing system still adopts a static task allocation strategy, which causes high-load nodes to become bottlenecks and low-load nodes to be idle. Simply expanding horizontally is difficult to improve overall efficiency and cannot meet the daily demand of one billion tag calculations. Weak fault tolerance: In a distributed architecture, cross-service collaboration increases, but there is a lack of breakpoint resumption mechanisms. Once a computing node fails, the tasks it is processing are lost, requiring manual intervention to restart, resulting in high maintenance costs and impacting the timeliness of business activities such as marketing campaigns.

[0004] Complex system architecture and high operation and maintenance costs: Existing solutions often rely on centralized coordination components such as ZooKeeper, which not only increases deployment complexity but also contradicts the actual needs of rapid iteration of business requirements and low-cost operation and maintenance. Summary of the Invention

[0005] This invention provides a user tag calculation method and system, electronic device, and storage medium to solve the defects in related technologies such as memory overflow, uneven resource utilization, weak fault tolerance, and complex system architecture. The solution of this application provides a highly fault-tolerant distributed computing method and system based on group selection and event-driven methods, which can solve the above problems.

[0006] This invention provides a user tag calculation method, comprising: Get the task request for user tag calculation; Upon triggering the task request, the user tag data to be calculated is grouped and selected; For each of the selected groups, user tags are calculated independently.

[0007] According to the user tag calculation method provided by the present invention, the step of grouping and selecting the user tag data to be calculated under the triggering of the task request includes: If a group already exists, continue grouping based on the largest group; If no grouping exists, then the user tag data will be grouped.

[0008] According to the user tag calculation method provided by the present invention, the step of grouping and selecting the user tag data to be calculated under the triggering of the task request includes: The user tag data in several groups obtained by group selection is the same.

[0009] According to the user tag calculation method provided by the present invention, the step of grouping and selecting the user tag data to be calculated under the triggering of the task request further includes: Context information is generated for several groups selected by grouping.

[0010] According to the user tag calculation method provided by the present invention, the step of grouping and selecting the user tag data to be calculated under the triggering of the task request includes: If the group selection process is interrupted, the grouping will restart from the point of interruption.

[0011] According to the user tag calculation method provided by the present invention, the step of grouping and selecting the user tag data to be calculated under the triggering of the task request includes: If all user tag data has been grouped and each group is normal, the group selection is complete. If all user tag data has been grouped, but any grouping is abnormal, the grouping selection is deemed to have failed.

[0012] This invention also provides a user tag calculation system, applied to a user tag calculation method, characterized in that it includes: The request retrieval module is used to retrieve task requests for user tag calculation; The group selection module is used to group and select user tag data to be calculated when the task request is triggered. The tag calculation module is used to independently calculate user tags for several groups obtained from the group selection.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the user tag calculation methods described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the user tag calculation methods described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the user tag calculation methods described above.

[0016] The user tag calculation method provided by this invention has at least the following beneficial effects: High fault tolerance: In the event of node failure, network interruption or other failures, the task will automatically recover without manual intervention or additional monitoring components; Resource efficiency: Failed tasks are only retried to the corresponding group, avoiding a full rerun; Memory safety: Data loading is controlled at the group level to prevent OutOfMemoryError (OOM). Simple architecture: It does not rely on heavy components such as ZooKeeper and YARN, making it suitable for rapid deployment; Clear semantics: It utilizes Kafka's native consumption model, and the logic conforms to best practices for distributed systems. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the user tag calculation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the user tag calculation system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] In related technologies, when calculating user tags, most adopt a tag-granular full-scale calculation mode and a distributed framework based on centralized scheduling. Specifically, each tag task loads all the user data it covers into the memory of a single computing node to execute the tag logic (such as determining whether "newly registered lottery buyers have completed access to the five core functions"), and relies on ZooKeeper for task allocation and state synchronization, combined with YARN / K8s for resource management.

[0021] However, the existing technologies also have many drawbacks, such as uncontrollable memory: high-coverage tags are prone to causing OOM (Out of Memory) errors, service availability is less than 80%, and they cannot stably support the calculation of 60 million user tags; unbalanced load: tasks are not dynamically allocated according to the number of users, resulting in resource waste and performance bottlenecks; poor fault tolerance: tasks are directly lost after node failure, requiring manual rerun of the entire task, affecting business continuity; heavy architecture: it strongly depends on components such as ZooKeeper, resulting in high deployment and maintenance costs, which does not conform to the lightweight operation and maintenance orientation of sports lottery; no ability to resume calculations without interruption: failed tasks cannot accurately locate the interruption point, and can only be recalculated for the entire tag, resulting in serious waste of computing resources.

[0022] Based on this, the solution proposed in this application provides a new method and system for calculating user tags, as detailed below.

[0023] Figure 1 This is a flowchart illustrating the user tag calculation method provided in an embodiment of the present invention.

[0024] like Figure 1 As shown, this embodiment provides a user tag calculation method, which can be applied to any scenario that requires user tag calculation. For example, it can be a sports lottery management system. The method includes: Step 101: Obtain the task request for user tag calculation; In practical applications, when a tag computation task is triggered, a task record can be created in the task metadata table. The record's fields can include the task ID, tag ID, and status. Then, a task request can be sent to the task queue for tag computation.

[0025] Step 102: Upon triggering the task request, group and select the user tag data to be calculated; After receiving a task request, the task queue can call the task management interface to update the corresponding task status to "computing". Then, it can query the group status table to check if a group exists. If a group already exists, it can continue to group based on the largest group. If no grouping exists, then the user tag data will be grouped.

[0026] Specifically, if the maximum group ID exists, it means that the previous segmentation was partially interrupted. The segmentation can continue from after the maximum group ID, with each group containing 10,000 users (e.g., min=20001, max=30000). If the maximum group ID does not exist, that is, it is the first calculation, it can start from the first one of the user instances. The first group is min=1, max=10000, and group number=1.

[0027] In practical applications, a group context can also be generated for each new group. The context information includes: tag ID, group ID, group number, maximum user instance ID, minimum user instance ID, tag calculation logic, and whether it is the last group (initially set to "no"). Write the grouping context to the "Pending Calculation Group Queue" and insert a record in the grouping status table, setting the status to "Pending Calculation"; When the segmentation reaches the last group (e.g., if the total number of users is 123,456, then the last group is min=120001, max=123456), mark "Is it the last group = yes"; After all groups are split, submit a manual ACK to Kafka in the "Pending Tasks Queue" to continue processing the next task.

[0028] In practice, a fault tolerance mechanism can also be set up. For example, if a group selection node crashes during the splitting process, the task messages that have not been submitted with ACK will be automatically resent to the new node by Kafka. The new node repeats the steps of querying the breakpoint and continues to split from the breakpoint. For example, if the original node splits to group number 5, the new node starts from number 6, avoiding execution from the beginning and achieving efficient breakpoint continuation.

[0029] Step 103: For the several groups obtained by group selection, complete the user tag calculation independently.

[0030] In practice, when performing group calculations, each group can be processed independently. Specifically, for each group, the group status table can be queried to update the group status to "calculating". Then, the tag calculation logic can be executed (e.g., "new registered users who have accessed ≥1 core function") to generate a new tag result set for the current group. After comparing the new tag results with the old tag results, three types of operation lists can be generated: Add to list: New if existing, no existing → Execute INSERT; Update list: Old and new values ​​do not match → Execute UPDATE; Delete from list: Existing items, new items, none → Execute DELETE.

[0031] During implementation, database operations can be executed in batches in a safe order, ensuring no query window period; If the calculation process is successful, update the group status to "completed", send a completion message to the "group calculation completion queue", and submit a KafkaACK. If it fails, retry; If the retry process is less than 3 times: keep the status "compiling" and do not submit ACK (Kafka will resubmit); If the retry process is ≥3 times: update status = "failed", send a failure message, and submit ACK to terminate the retry.

[0032] In the calculation process of the user tag data above, a fault tolerance mechanism is also set up. Specifically, unacknowledged group messages are re-delivered to a new node by Kafka; when the new node queries the group status as "calculating", it directly re-enters the calculation process (reloading data based on min / maxID) without having to reselect or initialize, thus achieving precise breakpoint continuation calculation.

[0033] In the exemplary embodiment, after grouping is completed, a final state determination can be performed. If there is a group with is_last_group='Yes', it can be determined that all user tag data has been grouped and all group statuses are "completed". Then, the task status is updated to "completed" and a "task completion notification" is pushed to the business system. If a group with is_last_group='Yes' exists, it can be determined that all user tag data has been grouped, but some group status is "Failed" (retries exhausted). In this case, the task status is updated to "Failed", and a "Task Failure Notification" is pushed (including the failed group ID for easy troubleshooting by operations and maintenance). Submit an ACK to the "Group Calculation Completed Queue" to complete the loop closure.

[0034] The user tag calculation system provided by the present invention is described below. The user tag calculation system described below can be referred to in correspondence with the user tag calculation method described above.

[0035] Figure 2 This is a schematic diagram of the user tag calculation system provided in an embodiment of the present invention.

[0036] like Figure 2 As shown, the user tag calculation system provided in this embodiment includes: Request retrieval module 201 is used to retrieve the task request for user tag calculation; The group selection module 202 is used to group and select user tag data to be calculated when the task request is triggered. The tag calculation module 203 is used to independently calculate user tags for several groups obtained by group selection.

[0037] Specifically, the user tag calculation system provided in this embodiment may include five core modules, each of which is deployed independently and supports dynamic scaling: (1) Tag Calculation Task Management Module, responsible for the full lifecycle management of tag calculation tasks, with main functions including: Task initialization: Create a new task record in the task metadata table, and set its initial state to "initialization"; Task triggering: Send a task message to the "Pending Tasks Queue" to initiate the subsequent process; Final state determination: Continuously monitor the "Group Calculation Completion Queue". When the completion or failure message of all groups (including the group marked as "last group") under a certain task is received, determine the overall status of the task and update the task record to "completed" or "failed".

[0038] This module acts as the task scheduling hub, ensuring the integrity and observability of each tag computing task from start to finish.

[0039] (2) The group selection module consumes task messages from the "task queue to be executed", executes the user grouping and splitting logic, and supports breakpoint resume capability: First, update the status of the corresponding task to "compiling"; Query the group status table to obtain the maximum existing group ID under the current tag and its corresponding max_user_id: If a maximum group ID exists, then starting from max_user_id+1, continue to divide the data at a granularity of "10,000 users per group" (to achieve breakpoint continuation of the division). If it does not exist (first calculation), then start splitting from the beginning starting with user_id=1; For each newly generated group, construct a complete group context, including: tag ID, group ID, group number (auto-incrementing in natural order within the task), min_user_id, max_user_id, tag calculation logic (such as a regular expression or script), and whether it is the "last group" (used for task final state determination). Persist the group state to the group state table; Sending the group context to the "Pending Computation Group Queue" must ensure that writing to the status table and sending messages are controlled by a single transaction. After all groups are segmented, submit an ACK to the "Tasks to be Executed Queue" to continue processing the next task.

[0040] This module significantly improves the recoverability and execution efficiency of large-scale tagging tasks through range-based segmentation and breakpoint continuation mechanisms.

[0041] (3) The group calculation module consumes group messages in the "Pending Calculation Group Queue" and performs end-to-end tag calculation and update process. The specific steps are as follows: Status Lock: Query the group status table and update the current group status to "Calculating"; Data loading and calculation: Based on the range of min_user_id and max_user_id, query the corresponding user data from the user master database and execute the pre-set tag calculation logic (e.g., "Vigor value ≥ 100 is judged as a highly active user"). Call the label Diff update submodule: compare the new calculation result with the historical label value and generate incremental operation instructions; Status write-back and notification: If the calculation is successful, update the group status to "completed"; If the attempt fails and the number of retries is ≥3, update the status to "failed". Send the grouping result message to the "Grouping Calculation Complete Queue"; Submit a manual ACK to Kafka to the "Pending Computation Group Queue" to confirm that message processing is complete.

[0042] This module is the core execution unit for tag computing, possessing high concurrency and high fault tolerance capabilities, and supporting dynamic scaling up and down.

[0043] (4) The label Diff update submodule, as a key sub-component of the grouping calculation module, is responsible for implementing incremental label updates without a window period: Query the set of old tag values ​​for users in the current group from a sports lottery tag repository (such as HBase or MySQL); By comparing the new label value set obtained from this calculation, three types of operation lists are generated: Adding to the list: If the new value exists but the old value does not exist, execute INSERT; Update list: Both old and new values ​​exist but their contents are inconsistent → Execute UPDATE; Deleting from a list: If the old value exists but the new value does not exist, execute DELETE. Database operations are executed in batches according to the business security order (usually INSERT / UPDATE first, then DELETE) to ensure that users can query valid tag values ​​at any time, completely avoiding the data gap problem caused by the traditional "delete first, insert later" mode.

[0044] (5) Group status management module, which maintains a unified group status table, provides status coordination services for the entire system, and supports the following key functions: The group selection module can query the "maximum group ID" to enable breakpoint continuation. Provides the group calculation module with the ability to update and query the real-time status of groups; The label calculation task management module determines whether the task has reached its final state (it needs to be confirmed that the "last group" has been processed).

[0045] This table only needs to store lightweight fields such as group ID, tag ID, min_user_id, max_user_id, status, number of retries, and whether it is the last group, resulting in high storage efficiency and excellent query performance.

[0046] The five modules mentioned above work together to form a highly fault-tolerant, highly elastic, and highly consistent distributed user tag computing system. It has been successfully applied in the sports lottery production environment, stably supporting an average of 1 billion tag computing requests per day, and meeting the requirements of core businesses such as refined operation and search recommendation for the timeliness and accuracy of tag computing.

[0047] In summary, the user tag calculation method and system provided in this application have the following characteristics: Based on the collaborative fault tolerance mechanism of Kafka manual ACK and group status table: the compute node only submits ACK after the task is successful or the maximum number of retries is reached. Unfinished tasks are automatically resubmitted by Kafka. Combined with the status table, accurate re-entry is achieved to avoid duplicate calculations or task loss. The "Calculating in progress" status supports breakpoint resumption: after a node restarts, it can identify unfinished groups and continue execution without having to start from the beginning; Decentralized automatic retry: It abandons external monitoring scans or scheduler intervention and relies entirely on message queue semantics to achieve high fault tolerance; Group selection + group parallel computing: ensures controllable memory usage for a single task and supports horizontal scaling; Lightweight ZooKeeper-free architecture: relies only on Kafka and state storage (such as MySQL / HBase), resulting in low deployment costs.

[0048] Task-Group Two-Level Status Collaborative Management Mechanism: The task status (initialization / calculation in progress / completed / failed) and group status (pending calculation / calculation in progress / completed / failed) are linked through a group status table. The task status is updated only when "there is a last group and the final status of all groups is clear", which solves the problem of progress misjudgment caused by the traditional scattered status.

[0049] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a user tag calculation method, which includes: Get the task request for user tag calculation; Upon triggering the task request, the user tag data to be calculated is grouped and selected; For each of the selected groups, user tags are calculated independently.

[0050] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the user tag calculation method provided by the above methods, the method including: Get the task request for user tag calculation; Upon triggering the task request, the user tag data to be calculated is grouped and selected; For each of the selected groups, user tags are calculated independently.

[0052] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the user tag calculation method provided by the methods described above, the method comprising: Get the task request for user tag calculation; Upon triggering the task request, the user tag data to be calculated is grouped and selected; For each of the selected groups, user tags are calculated independently.

[0053] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0054] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A user tag calculation method, characterized by, The method comprises the following steps: acquiring a task request for user label calculation; grouping and selecting user label data to be calculated under the triggering of the task request; independently completing user label calculation for a plurality of groups obtained through grouping and selection.

2. The user tag calculation method according to claim 1, wherein, The step of grouping and selecting user label data to be calculated under the triggering of the task request comprises the following steps: if there are groups, continue grouping based on the largest group; if there are no groups, start grouping user label data.

3. The user tag calculation method according to claim 1, wherein, The step of grouping and selecting user label data to be calculated under the triggering of the task request comprises the following steps: user label data in the plurality of groups obtained through grouping and selection is the same. 4.The user tag calculation method according to claim 2 or 3, characterized in that, The step of grouping and selecting user label data to be calculated under the triggering of the task request further comprises the following steps: generating context information for the plurality of groups obtained through grouping and selection.

5. The user tag calculation method of claim 1, wherein, The step of grouping and selecting user label data to be calculated under the triggering of the task request comprises the following steps: if interruption occurs during the grouping and selection process, start grouping again from the interruption. 6.The user tag calculation method of claim 1, wherein, The step of grouping and selecting user label data to be calculated under the triggering of the task request comprises the following steps: if all user label data is grouped and each group is normal, it is determined that the grouping and selection is completed; if all user label data is grouped but any group is abnormal, it is determined that the grouping and selection fails.

7. A user tag calculation system for use in the user tag calculation method according to any one of claims 1 to 6, characterized by The method comprises the following steps: a request acquisition module, configured to acquire a task request for user label calculation; a grouping and selection module, configured to group and select user label data to be calculated under the triggering of the task request; a label calculation module, configured to independently complete user label calculation for a plurality of groups obtained through grouping and selection.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the user label calculation method according to any one of claims 1-6 when executing the program.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that, The computer program implements the user label calculation method according to any one of claims 1-6 when executed by the processor.

10. A computer program product comprising a computer program, characterized in that, The computer program implements the user label calculation method according to any one of claims 1-6 when executed by the processor.

Images