A user data cleaning method and related apparatus

Abstract

The application discloses a user data cleaning method and related device, and relates to the field of data management. Specifically, a deletion message consumer pulls a deletion message from a deletion message queue, determines a user corresponding to the present consumption and a data cleaning range, and deletes database data in batches according to the user and the data cleaning range. The data amount of each batch deletion does not exceed a batch deletion threshold, so that the time consumption and log amount of a single deletion transaction can be reduced. After the batch deletion is completed, if the actual deletion data amount is equal to the batch deletion threshold, a new deletion message is written into the deletion message queue based on the obtained deletion message. Through the progressive batch deletion and queue rearrangement mechanism, the application solves the database performance problems caused by a large batch deletion operation, such as database lock competition, deadlock, transaction timeout, master-slave delay and the like, and provides a basis for realizing a business data record cleaning task under the premise of guaranteeing database performance.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a user data cleaning method and related apparatus. Background Technology

[0002] With the development of Internet / IoT technologies and the widespread adoption of enterprise-level applications, business data records from various business systems, such as user behavior logs and business operation logs, exhibit characteristics such as high write frequency, large data volume, and rapid growth over time, posing a severe challenge to database storage and management. Taking user behavior logs as an example, a medium-sized Internet application may generate hundreds of millions of user behavior records daily. Storing all user behavior records without control would lead to a rapid increase in database storage costs over time, while database query performance would also decline due to the excessive data volume.

[0003] To control storage costs and ensure query performance, it is often necessary to clean up business data records in the database. Current technologies typically employ a method of uniformly scanning all user data and uniformly deleting user data exceeding a fixed threshold to achieve this purpose. However, large-scale deletion operations can easily lead to database performance disasters, resulting in problems such as database lock contention, deadlocks, and transaction timeouts. Summary of the Invention

[0004] In view of the above problems, this application provides a user data cleaning method and related apparatus. By using a separate consumer to serialize the data deletion requests of each user based on a deletion message queue, the lock contention problem between deletion tasks is eliminated. Through progressive batch deletion and queue requeuing mechanism, the automatic splitting of large batch deletion requests is realized, reducing the possibility of deadlock and transaction timeout, and providing a foundation for ensuring database performance.

[0005] The specific plan is as follows: The first aspect of this application provides a user data cleanup method for deleting message consumers, the method comprising: Retrieve deletion messages from the deletion message queue. The deletion messages in the deletion message queue carry cleanup task information, which includes user information and data cleanup scope indication information. Based on the obtained deletion messages, determine the target users and the scope of target data cleanup for this cleanup task; Batch deletion of business data records in the database corresponding to the target user and the target data cleanup scope, wherein the total amount of data deleted in the batch does not exceed a preset batch deletion threshold; If the actual amount of data deleted is equal to the batch deletion threshold, then a new deletion message is written to the deletion message queue based on the obtained deletion message; otherwise, no message is written. The obtained deletion message is either a message written by the first processing unit when a first condition is met, or a message written by the deletion message consumer when a second condition is met; the first condition is that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold, and the second condition is that the actual amount of data deleted when executing the historical cleanup task of the target user is equal to the batch deletion threshold.

[0006] A second aspect of this application provides a user data cleaning method, the method comprising: When the first processing unit determines that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold, it writes a deletion message to the deletion message queue. The deleted message carries the user information of the target user and data cleanup range indication information configured based on the data over-limit situation of the target user. The deletion message consumer retrieves deletion messages from the deletion message queue. Based on the retrieved deletion messages, it determines the users and data cleanup scope of the current cleanup task. It then performs batch deletion on the business data records in the database corresponding to the users and data cleanup scope of the current cleanup task. The total amount of data deleted in the batch does not exceed a preset batch deletion threshold. If the actual amount of data deleted is equal to the batch deletion threshold, a new deletion message is written to the deletion message queue based on the retrieved deletion messages; otherwise, no new message is written. The retrieved deletion messages are either the deletion messages written by the first processing unit or the messages written by the deletion message consumer when a second condition is met. The second condition is that the actual amount of data deleted during the execution of a historical cleanup task is equal to the batch deletion threshold, and the historical cleanup task has the same users as the current cleanup task.

[0007] A third aspect of this application provides a user data cleaning apparatus, comprising: a first processing unit, a message deletion queue, and a message deletion consumer; The first processing unit is used to write a deletion message to the deletion message queue when it is determined that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold. The deletion message written by the first processing unit carries the user information of the target user and data cleanup range indication information configured based on the data over-limit situation of the target user. The deletion message consumer is used to retrieve deletion messages from the deletion message queue, determine the users and data cleanup scope of the current cleanup task based on the retrieved deletion messages, and perform batch deletion of business data records in the database corresponding to the users and data cleanup scope of the current cleanup task. The total amount of data deleted in the batch does not exceed a preset batch deletion threshold. Furthermore, a new deletion message is written to the deletion message queue based on the retrieved deletion messages only when the actual amount of data deleted equals the batch deletion threshold. The retrieved deletion messages are either: deletion messages written by the first processing unit, or messages written by the deletion message consumer when a second condition is met, whereby the actual amount of data deleted during the execution of a historical cleanup task equals the batch deletion threshold, and the historical cleanup task has the same users as the current cleanup task. The deletion message queue is used to carry the written deletion messages.

[0008] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the user data cleanup method described in the first aspect above, or to implement the user data cleanup method described in the second aspect above.

[0009] The fifth aspect of this application provides a storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the user data cleanup method described in the first aspect or the user data cleanup method described in the second aspect.

[0010] The sixth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the user data cleanup method described in the first aspect above, or to implement the user data cleanup method described in the second aspect above.

[0011] By employing the aforementioned technical solution, this application achieves complete decoupling of data cleanup operations from other operations (such as cleanup detection and business data record writing) in terms of time and resources through a separate delete message consumer asynchronously and serially executing deletion tasks based on a delete message queue. This solves, to some extent, the problem of resource competition between data cleanup and other operations, helping to ensure stable system throughput and avoid database performance fluctuations caused by deletion operations. Furthermore, by using a single consumer to consume the delete message queue, the lock contention problem between deletion tasks is resolved (i.e., the problem of a single transaction holding row locks and gap locks for a long time, blocking read and write operations of other transactions). Based on this, the delete message consumer pulls deletion messages from the delete message queue, determines the user and data cleanup scope corresponding to this deletion task, and performs batch deletion on the corresponding data stored in the database within the limit of the batch deletion threshold. Deletion is performed by batch deleting data in a way that does not exceed the batch deletion threshold and only deletes a fixed number of business data records per batch. This reduces the time each deletion transaction holds locks, thereby reducing the probability of database deadlock and the risk of transaction timeout (i.e., the risk of transaction failure or rollback due to long transaction execution time triggering the database transaction timeout mechanism). At the same time, it reduces the amount of logs generated by each deletion transaction, providing a basis for controlling master-slave synchronization latency. After batch deletion is completed, the deletion message consumer compares the actual amount of data deleted with the batch deletion threshold. When the two are equal, a new deletion message is written to the deletion message queue based on the obtained deletion message, so that the user data can continue to be cleaned by consuming the message again. This application achieves the goal of meeting the needs of large-scale deletion by repeatedly delivering deletion messages while data cleanup is not complete.

[0012] This application solves database performance problems caused by large-scale deletion operations, such as database lock contention, deadlock, transaction timeout, and master-slave latency (i.e., the large amount of logs generated by large-scale deletion leads to latency in slave database replay, affecting query performance under the read-write separation architecture), by using a progressive batch deletion and queue requeuing mechanism. It provides a foundation for implementing business data record cleaning tasks while ensuring database performance. Attached Figure Description

[0013] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a user data cleaning method provided in this application; Figure 2 An example is provided illustrating the user data cleanup process. Figure 3 A schematic diagram of a user data cleaning device provided in this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0014] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0015] This application provides a user data cleaning method and related apparatus, which can be applied to solve the database data cleaning task of business systems, especially the cleaning task of business data records (user behavior logs, business operation logs, etc.).

[0016] First, let me explain the concepts involved in this application.

[0017] A message queue is middleware used for asynchronous communication between modules. The aforementioned modules may specifically include existing modules of the business system, newly added modules, or external modules integrated with the business system / database. The modules or units described in this application refer to code units used to implement specific functions. Any module / unit can be implemented through independent threads, processes, or other forms; this application does not limit the specific implementation method of the modules / units.

[0018] Optionally, the message queue described in this application may specifically be a distributed stream processing platform such as Kafka, the open-source message broker software RabbitMQ, the distributed message and streaming data platform RocketMQ, or the message queue toolkit Redis Stream, or other middleware systems or tools capable of implementing queue functionality. This application does not limit the specific implementation method of the message queue; the selection can be based on the database cleanup requirements of the business system and the performance of the message queue middleware (such as throughput, reliability, and latency).

[0019] A deletion message queue is used to cache deleted messages. These messages are consumed by consumers of the deletion message queue (referred to as deletion message consumers). The process of a deletion message consumer consuming a message is equivalent to executing the corresponding deletion task. In other words, there is a one-to-one correspondence between deleted messages and deletion tasks. Furthermore, the deletion message queue can decouple the data cleanup process from normal business processes.

[0020] The producer of the deletion message queue is used to write deletion messages to the queue. Specifically, it includes a first processing unit, which writes a corresponding deletion message when a user's data exceeds the limit. The first processing unit and the deletion message consumer are independent of each other, and both can execute their respective processing logic simultaneously. Furthermore, the producer of the deletion message queue also includes a deletion message consumer, which rewrites deletion messages to the queue when the current small-batch deletion does not meet the actual deletion requirements, thereby achieving the re-enqueueing of deletion messages and the cyclical execution of deletion tasks.

[0021] Figure 1 This is a flowchart illustrating a user data cleanup method according to an embodiment of this application, which is applied to deleting message consumers. Combined with... Figure 1 As shown, the method may include the following steps: S101. Retrieve the deletion message from the deletion message queue.

[0022] The deletion message in the message queue carries cleanup task information, which may specifically include user information and data cleanup scope indication information. For example, a message in the deletion message queue can be represented as: (user information, data cleanup scope indication information).

[0023] The user information indicates the user to whom the data to be cleaned belongs; in other words, whose business data records need to be cleaned. The data cleanup scope indication information indicates the scope of data to be cleaned; in other words, which data needs to be cleaned / deleted. This information may specifically include at least one of the following: the identifier set of the data to be cleaned, the total number, the creation time range, and data characteristics that only the data to be cleaned must meet. This embodiment does not limit the specific form of the data cleanup scope indication information.

[0024] It should be noted that the message consumer can consume messages in the message queue in a preset order. For example, the preset order can be a chronological order, i.e., a first-in, first-out (FIFO) order.

[0025] S102. Based on the obtained deletion message, determine the target user and target data cleanup scope for this cleanup task.

[0026] Specifically, the process involves parsing the obtained deletion message, identifying the target user based on the parsed user information, and determining the target data cleanup scope based on the parsed data cleanup scope indication information.

[0027] S103. Perform batch deletion of business data records in the database that correspond to the target user and the target data cleanup scope.

[0028] The batch deletion described in S103 refers to small-batch deletion, where the total amount of data to be deleted does not exceed a preset batch deletion threshold. When the total amount of data to be deleted exceeds the batch deletion threshold, this batch deletion cannot meet the data cleanup objective. That is, after executing S103, the total number of business data records for the target user still exceeds the limit. However, this consumption will not continue to clean up the excess data or continuously occupy database resources, thereby resolving the database anomaly risk caused by large-batch deletion.

[0029] The batch deletion threshold can be set based on factors such as database hardware capabilities, data characteristics of business data records, and business requirements. For example, the threshold can be set to a value that ensures the execution time of a single deletion task meets the database timeout threshold and the amount of Binlog logs generated by a single batch deletion meets the master-slave synchronization performance limits. For instance, the batch deletion threshold could be set to 500.

[0030] S104. Determine whether the actual amount of data deleted is equal to the batch deletion threshold.

[0031] If the value equals the batch deletion threshold, it indicates that the data cleanup for the target user is incomplete and the total number of business data records for the target user still exceeds the limit. In this case, S105 is executed to re-deliver the deletion message for cleaning up the business data records of the target user.

[0032] If the value is not equal to the batch deletion threshold, it means that the batch deletion operation has deleted all the data to be cleaned up, and the data cleanup for the target user has been completed. In this case, there is no need to redeliver the deletion message, that is, no new deletion message is written to the deletion message queue.

[0033] S105. Write a new deletion message to the deletion message queue based on the obtained deletion message.

[0034] It should be noted that the new deletion message carries cleanup task information, specifically including: the target user's user information, and cleanup scope indication information used to indicate the scope of data cleanup after executing S103. By executing S105, the unfinished data cleanup tasks for the target user can be continuously delivered and executed through the message queue. In this way, multiple small-batch deletion tasks executed serially can meet the target user's large-batch (i.e., data cleanup needs exceeding the batch deletion threshold) data cleanup needs, achieving a gradual batch deletion task.

[0035] Based on the foregoing, the deletion message obtained by the deletion message consumer in S102 can be: a message written by the first processing unit when the first condition is met, or a message written by the deletion message consumer when the second condition is met. The first condition is that the first processing unit determines that the total number of business data records of the target user in the database exceeds a pre-configured data retention threshold. It should be noted that the first processing unit can be pre-configured to perform cleanup checks on the business data records of the target user stored in the database when the preset cleanup detection conditions are met. For example, the cleanup detection conditions may include: the current time reaching the cleanup detection execution time specified by a preset timed cleanup cycle; or, the first processing unit detecting that a new business data record belonging to the target user has been written to the database; or, the first processing unit detecting that the total amount of data in the database has reached a preset quantity threshold; or, the first processing unit obtaining an indication message indicating that a new business data record belonging to the target user has been written to the database / the total amount of data in the database has reached a preset quantity threshold. The indication message can be actively obtained by the first processing unit or passively received by the first processing unit. It should be noted that the aforementioned cleanup detection conditions are only examples, and this embodiment does not limit the specific form of the cleanup detection conditions.

[0036] Depending on the specific cleanup detection conditions, the range of data to be cleaned can take different forms. For example, the range of data to be cleaned could refer to business data records whose creation time is more than a preset time interval from the current time. Here, creation time refers to the moment the business data record is generated or written to, representing the execution time of the corresponding user behavior or operation. By using the indication information representing the aforementioned range, the cleanup task of historical business data records far removed from the current time can be implemented, alleviating the database's data storage pressure and improving database query performance. In other possible implementations, the range of data to be cleaned could also be the amount of data exceeding a preset limit (let's say N). By using the indication information representing the aforementioned range, at least N business data records with the earliest creation time can be deleted, achieving a targeted cleanup task for data exceeding the limit, thus meeting the database's storage limit requirements for the target user's business data records.

[0037] It should be noted that, to retain business data records with greater business value, data created earlier is typically prioritized for deletion. In other scenarios, the data types or characteristics to be deleted can also be set as needed.

[0038] The data retention threshold refers to the maximum number of data records that can be retained for each user (or each user class).

[0039] The second condition is that the actual amount of data deleted by the message consumer when performing the historical cleanup task for the target user is equal to the batch deletion threshold. That is, when the message consumer last consumed the deletion message corresponding to the target user, it determined that the data cleanup for the target user was not completed. In this case, the deleted message obtained is the deleted message re-delivered by the message consumer.

[0040] Based on the above, this application achieves complete decoupling of data cleanup operations from other operations (such as cleanup detection and business data record writing) in terms of time and resources by using a separate delete message consumer to asynchronously and serially execute delete tasks based on a delete message queue. This solves the problem of resource competition between data cleanup and other operations to a certain extent, helps ensure stable system throughput, and avoids database performance fluctuations caused by delete operations. In addition, by using a single consumer to consume the delete message queue, the lock contention problem between delete tasks is solved (i.e., the problem of a single transaction holding row locks and gap locks for a long time, blocking read and write operations of other transactions). On this basis, the delete message consumer pulls delete messages from the delete message queue, determines the user and data cleanup scope corresponding to this delete task, and performs batch deletion of the corresponding data stored in the database under the limitation of the batch deletion threshold. In addition, since the amount of data deleted in a batch does not exceed the batch deletion threshold, and only a fixed number of business data records are deleted in each batch, the time that each deletion transaction holds the lock is reduced, thereby reducing the probability of database deadlock and the risk of transaction timeout (i.e., the risk of the database transaction timeout mechanism being triggered due to the long execution time of the transaction, leading to transaction failure or rollback). At the same time, the amount of logs generated by each deletion transaction is reduced, providing a basis for controlling master-slave synchronization latency. After the batch deletion is completed, the deletion message consumer compares the actual amount of data deleted with the batch deletion threshold. When the two are equal, a new deletion message is written to the deletion message queue based on the obtained deletion message, so that the user data can continue to be cleaned by consuming the message again. This application achieves the purpose of meeting the needs of large batch deletion by repeatedly delivering deletion messages while the data cleanup is not completed.

[0041] This application solves database performance problems caused by large-scale deletion operations, such as database lock contention, deadlock, transaction timeout, and master-slave latency (i.e., the large amount of logs generated by large-scale deletion leads to latency in slave database replay, affecting query performance under the read-write separation architecture), by using a progressive batch deletion and queue requeuing mechanism. It provides a foundation for implementing business data record cleaning tasks while ensuring database performance.

[0042] In one or more embodiments provided in this application, the first processing unit may be a message consumer that triggers the message queue.

[0043] The trigger message queue is used to cache trigger messages, which are consumed by consumers (i.e., deletion message consumers) that determine whether a trigger is needed for the deletion message queue. Correspondingly, the producer of the trigger message queue is a unit / module distinct from the trigger message consumer and the deletion message consumer, used to write trigger messages to the trigger message queue when the corresponding triggering conditions are met. The trigger messages in the trigger message queue carry user information, which indicates the user requiring data cleanup checks; that is, the message can instruct the trigger message consumer to perform data cleanup checks on the user corresponding to the carried user information.

[0044] Based on this, the user corresponding to the user information carried in the deletion message obtained by the deletion message consumer, i.e., the target user, can be the user determined by the first processing unit based on the trigger message obtained from the trigger message queue.

[0045] Based on this, the first processing unit (i.e., the trigger message consumer) can be configured to: after determining the target user based on the obtained trigger message, detect whether the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold, so that if it is determined that the total number of business data records of the target user exceeds the pre-configured data retention threshold, write a deletion message carrying user information for indicating the target user to the deletion message queue.

[0046] It should be noted that the message deletion consumer and the message triggering consumer are independent of each other. They can consume messages in the message deletion queue and the message triggering queue respectively, and perform data cleaning operations and cleaning detection operations in parallel.

[0047] By employing a dual-message queue architecture with independent messages, this embodiment achieves complete decoupling between cleanup detection and data deletion. Trigger messages and deletion messages are processed asynchronously through independent queues, reducing the risk of resource contention and helping to improve data cleanup efficiency.

[0048] In one possible implementation, the trigger message obtained by the first processing unit may be: a message written to the trigger message queue by the second processing unit in response to the new business data record of the target user being written to the database, or a message written to the trigger message queue by the third processing unit after configuring the activity status identifier for the target user.

[0049] The second and third processing units are distinct from the trigger message consumer and the delete message consumer; both belong to the producers of the trigger message queue. The two cases are explained below.

[0050] The second processing unit will be described below.

[0051] In one possible implementation, the second processing unit may be a data writing module or a business interface thread. This module is used to write business data records of each user to the database. Based on this, after successfully writing the business data records belonging to the target user to the database, the data writing module writes a trigger message carrying user information indicating the target user to the trigger message queue.

[0052] In another possible implementation, the second processing unit can be a database-integrated insert trigger. In this case, after the target user's business data record is written to the data record table (i.e., the database), the trigger can automatically execute a predefined stored procedure or function and send a trigger message carrying user information indicating the target user to the trigger message queue.

[0053] Based on this, the trigger message consumer can perform cleanup detection based on the trigger message, and when the total amount of data is detected to exceed the limit, write a deletion message carrying user information indicating the target user to the deletion message queue, so as to trigger the deletion message consumer to clean up the business data records of the target user.

[0054] In summary, this application can implement the writing of trigger messages to the trigger message queue in response to new data writes, based on database insert triggers or the application layer. This application does not limit the specific method for generating the trigger messages.

[0055] This application decouples data writing operations from data cleanup tasks by using a trigger message queue. The write chain completes the operation immediately after sending the trigger message, without waiting for cleanup detection or data cleanup results. This eliminates resource contention during data cleanup, helping to maintain stable system throughput and meet the data lifecycle management needs in high-concurrency write scenarios. Based on the above, this application adopts a new message-driven real-time trigger mechanism for cleanup detection. A trigger message is generated after each new data write to initiate a check on the total data volume for that user. If the limit is exceeded, a deletion message triggers the subsequent cleanup process. Furthermore, because the deletion message consumer adopts a progressive deletion mechanism, each deletion transaction performs only small-batch deletions with short processing time, compressing the time window between exceeding the data volume limit and cleanup execution to the millisecond level. Based on this, the cleanup detection trigger mechanism based on new data writes enables real-time reclamation of storage space, solving the problem of data cleanup lag in traditional scheduled cleanup schemes. In addition, this application can automatically adapt to business changes, such as a surge in user data, without requiring manual adjustment of the scheduled cleanup cycle, helping to reduce the frequency of database administrator (DBA) intervention and save labor costs.

[0056] Figure 2Taking the second processing unit as the data writing module and the first processing unit as the trigger message consumer as an example, a schematic diagram of a user data cleanup process is illustrated. Figure 2 As shown, the user data cleanup process may include: The first step is for the data writing module to write a trigger message to the trigger message queue in response to the new business data record being written to the database.

[0057] Specifically, upon receiving user data from the business application, the data writing module writes the corresponding new business data record to the data record table in the database. After successful writing, it generates a trigger message and sends it to the trigger message queue, or simultaneously generates a trigger message and sends it to the trigger message queue while writing the corresponding new business data record to the data record table in the database. The trigger message generated / written by the second processing unit carries user information indicating the user to whom the new business data record belongs.

[0058] The second step involves the trigger message consumer consuming each trigger message in the trigger message queue one by one.

[0059] Specifically, each triggered message consumption process may include: retrieving a trigger message from the trigger message queue, determining the first user based on the retrieved trigger message, checking whether the total amount of business data records of the first user stored in the database exceeds the corresponding data retention threshold. If it exceeds the data retention threshold, a deletion message is generated and sent to the deletion message queue, and then the consumption is completed; if it does not exceed the data retention threshold, the consumption is completed directly.

[0060] In this context, the first user refers to the user indicated by the user information carried in the trigger message obtained by the trigger message consumer. In other words, the trigger message obtained by the trigger message consumer carries the first user's identification information, such as an ID. The deletion message generated / written by the trigger message consumer carries user information indicating the first user, as well as information indicating the scope of data cleanup for the first user. This information can be configured based on the detection results of the trigger message consumer on the business data records of the first user stored in the database.

[0061] The third step involves the consumer consuming each deleted message in the message queue one by one.

[0062] Specifically, each deletion message consumption process may include: retrieving deletion messages from the deletion message queue; determining the second user and the second data cleanup scope based on the retrieved deletion messages; performing batch deletion of business data records in the database (specifically, the data record table) corresponding to the second user and the second data cleanup scope according to the batch deletion threshold, such as sending the corresponding batch deletion command to the database (specifically, the data record table); comparing the number of successfully deleted records returned by the database with the batch deletion threshold; if it equals the batch deletion threshold, it indicates that the data cleanup task for the second user is not completed, a new deletion message is generated and sent to the deletion message queue to wait for the next consumption, and then the current consumption is completed; if it is less than the batch deletion threshold, it indicates that the data cleanup task for the second user is completed, no deletion message is written, and the current consumption is completed directly.

[0063] In this context, the second user is the deletion message consumer, and the user information carried in the deletion message obtained during this consumption process indicates the user. In other words, the deletion message consumer obtains the second user's identification information, such as an ID, carried in the deletion message. New deletion messages generated or written by the deletion message consumer carry user information indicating the second user, as well as information indicating the scope of data cleanup for the second user. The information indicating the scope of data cleanup for the second user can be set at least based on the cleanup scope indication information carried in the deletion message obtained by the deletion message consumer. In one possible implementation, this information can be consistent with the cleanup scope indication information carried in the deletion message obtained by the deletion message consumer; in another possible implementation, this information can also be obtained by updating the cleanup scope indication information carried in the deletion message obtained by the deletion message consumer based on the current batch deletion situation.

[0064] Based on the above, this embodiment generates a trigger message immediately after new data is written to initiate a data volume check for the user, thereby enabling real-time triggering and on-demand execution of data cleanup tasks.

[0065] The third processing unit will be described below.

[0066] The third processing unit is used to configure a corresponding activity status identifier for each user based on the business data records of each user within a preset time window. The configured activity status identifier is used by the first processing unit to determine the number retention threshold for the target user.

[0067] In one possible implementation, the aforementioned preset time window can refer to a time window of duration T preceding the current moment, where the current moment can refer to the moment when the activity status identifier is configured, and T represents the preset time window duration. For example, T can be 30 days or 15 days, which can be configured according to the business scenario. The process by which the third processing unit configures the target user's activity status identifier can then include: Scan the target user's business data records stored in the database (i.e., scan the target user's data record table), obtain the target user's last activity time, calculate the time difference between the current time and the target user's last activity time, and if the time difference is not greater than the preset time window length T, then configure an activity status identifier representing that the target user is in an active state; if the time difference is greater than the time window length T, then configure an activity status identifier representing that the target user is in an inactive state.

[0068] Based on this, the data retention threshold can be configured based on the target user's activity status identifier, with the data retention threshold being higher when the target user is in an active state than when the target user is in an inactive state.

[0069] This embodiment categorizes users into active and inactive users based on whether they have any business data records (including but not limited to page visits, new messages, new operations, etc.) within a preset time window. Inactive users are typically churned, dormant, or low-frequency users. The historical user data of inactive users has low business value, and retaining only a small amount of recent data is sufficient to meet users' backtracking needs. Therefore, this application prioritizes retaining the historical user data of active users; that is, the data retention threshold for the same user when in an active state is greater than the data retention threshold when the user is in an inactive state, in order to meet the query needs of active users for historical data.

[0070] Optionally, the third processing unit can be a business scanning module. For example, the business scanning module can be started as a background thread when the business system starts to initialize the activity status identifier of each user, and then trigger the cleanup detection, i.e., the initial detection, when the system starts.

[0071] To avoid repeated database scanning by the third processing unit and the impact on database performance, the activity status information of each user can be stored in the storage medium. In other words, the aforementioned configuration of the activity status identifier for the target user may include: storing the user activity status information corresponding to the target user in the storage medium. This information may include: user information used to indicate the target user, and an activity status identifier that represents whether the target user is in an active or inactive state.

[0072] Based on this, the third processing unit can be configured to periodically perform activity status flag configuration operations according to a preset period in order to maintain the activity status flags of each user in the storage medium.

[0073] In another possible implementation, the aforementioned storage medium can be a cache medium, such as a Redis cache. In this case, when the third processing unit stores user activity status information corresponding to the target user in the cache medium, it can also configure an expiration time for this information to automatically clean up expired information, reducing cache pressure and unnecessary cache overhead. For example, the aforementioned expiration time can be T, i.e., a preset time window duration.

[0074] Building upon the above, the process of a message consumer consuming a triggered message each time can also include: Before detecting whether the total number of business data records of the first user stored in the database exceeds the corresponding data retention threshold, the user activity status identifier of the first user is obtained from the cache medium. Based on the obtained result, the data retention threshold is configured for the first user and the user activity status information corresponding to the first user is configured.

[0075] For example, the threshold for retaining configuration data for the first user based on the obtained results may include: If the obtained user activity status identifier indicates that the first user is in an active state, then the first preset threshold is configured as the data retention threshold. If the obtained user activity status identifier indicates that the first user is inactive, or if no user activity status identifier is obtained, then the second preset threshold is configured as the data retention threshold.

[0076] The first preset threshold is greater than the second preset threshold. For example, the first preset threshold is 3000 and the second preset threshold is 100.

[0077] In addition, user activity status information corresponding to the target user is configured based on the obtained results, including: If the obtained user activity status identifier indicates that the first user is in an active state, update the expiration time of the activity status information corresponding to the first user in the cache medium. If the obtained user activity status identifier indicates that the first user is inactive, the activity status identifier of the first user in the cache medium is updated to the activity status identifier indicating that the first user is active, and the expiration time corresponding to the activity status information of the first user is updated. If the activity status identifier of the first user is not obtained, the user activity status information corresponding to the first user (including the activity status identifier representing that the first user is in an active state) is stored in the cache medium, and an expiration time is configured for the information.

[0078] Specifically, updating the expiration time means delaying the expiration time of the first user's activity status information; that is, the updated expiration time is later than the original expiration time. For example, the expiration time can be updated to a time T (denoted as t+T) after the current time t (i.e., the time when the first user's activity status identifier is obtained). In other words, regardless of whether a status identifier is obtained or what type of status identifier is obtained, after the consumer completes this consumption, the first user's activity status information in the cache medium will be the same (including the identifier type and expiration time).

[0079] Based on the above, under the premise of triggering cleanup checks in response to new data writing, the activity status of each user is initialized through the third processing unit. The trigger message consumer configures the activity status of each user while consuming the trigger message and performing cleanup checks. This allows for the maintenance of user activity status during use without the need for the third processing unit to perform full user activity status configuration periodically, saving system resources. Furthermore, the trigger message consumer can determine the corresponding data retention threshold based on the actual activity status of users, and perform differentiated data cleanup for active and inactive users accordingly, balancing business needs and storage resource cleanup requirements.

[0080] In one or more embodiments provided in this application, the process by which the first processing unit (such as a trigger message consumer) detects whether the total amount of business data records of the first user (or target user) stored in the database exceeds the data retention threshold may include: Query the creation time of the target data in the database.

[0081] The target data is the Nth or N+1th business data record in the business data records corresponding to the first user (or target user) and sorted in descending order of creation time. N is the data retention threshold. The Nth business data record can be called the retention boundary data, which is the earliest created data in the business data records of the first user (or target user) that needs to be retained. The N+1th business data record can be called the cleanup boundary data, which is the latest created data in the business data records of the first user (or target user) that needs to be cleaned up.

[0082] If the query fails to find the target user, it is determined that the total number of business data records for the first user (or target user) in the database does not exceed the data retention threshold for that user; if the query succeeds, it is determined that the total number of business data records for the first user (or target user) in the database exceeds the data retention threshold for that user.

[0083] Based on this, the data cleanup range indication information carried in the trigger message generated / written by the first processing unit (such as the trigger message consumer), that is, the data cleanup range indication information carried in the deletion message obtained by the deletion message consumer, may include: the creation time of the retained boundary data, or the creation time of the cleanup boundary data, so that the deletion message consumer can determine the creation time range of the data to be cleaned.

[0084] The message consumer then determines the target data cleanup scope for this cleanup task based on the obtained deletion messages, which may include: Based on the data cleanup scope indication information carried in the obtained deletion message, business data records whose creation time is earlier than the creation time of the retained boundary data, or whose creation time is no later than the creation time of the cleanup boundary data, are determined as the target data cleanup scope for this cleanup task.

[0085] In this case, the deleted message newly written by the consumer during a certain consumption process can carry the same information as the deleted message obtained.

[0086] Based on the above, the total amount of business data records for each user in the database can be kept near their respective data retention thresholds, effectively controlling system storage costs while ensuring query performance.

[0087] This application employs a real-time data write-triggered cleanup mechanism, combined with message queue decoupling and a gradual batch deletion mechanism, to solve the database performance, resource contention, and cleanup delay issues caused by scheduled batch deletion in existing technologies. This achieves a balance between storage costs and business needs, providing a foundation for ensuring high-performance and stable database operation. Furthermore, this application utilizes a user activity stratification mechanism, dynamically classifying users into active and inactive users based on their activity within a preset time window. Differentiated data retention thresholds are set for users with different activity levels, effectively reducing storage usage by inactive users while ensuring that active users' historical data meets their frequent query needs. In short, this application significantly reduces the overall database storage while guaranteeing a good business experience for active users, achieving a precise match between storage resources and business requirements.

[0088] The user data cleaning apparatus provided in the embodiments of this application is described below. The user data cleaning apparatus described below can be referred to in correspondence with the user data cleaning method described above.

[0089] Figure 3 This is a schematic diagram of the structure of a user data cleaning device disclosed in an embodiment of this application. Figure 3 As shown, the device may include: a first processing unit 31, a message deletion queue 32, and a message deletion consumer 33.

[0090] Specifically, the first processing unit is used to write a deletion message to the deletion message queue when it is determined that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold. The deletion message written by the first processing unit carries the user information of the target user and data cleanup range indication information configured based on the data exceeding the limit of the target user.

[0091] The deletion message consumer is used to retrieve deletion messages from the deletion message queue, determine the users and data cleanup scope of the current cleanup task based on the retrieved deletion messages, and perform batch deletion of business data records in the database corresponding to the users and data cleanup scope of the current cleanup task. The total amount of data deleted in the batch does not exceed a preset batch deletion threshold. Furthermore, a new deletion message is written to the deletion message queue based on the retrieved deletion messages only when the actual amount of data deleted equals the batch deletion threshold. The retrieved deletion messages are either the deletion messages written by the first processing unit or the messages written by the deletion message consumer when a second condition is met. The second condition is that the actual amount of data deleted during the execution of a historical cleanup task equals the batch deletion threshold, and the historical cleanup task has the same users as the current cleanup task.

[0092] The deletion message queue is used to carry the written deletion messages.

[0093] In one or more embodiments provided in this application, the apparatus may further include: a trigger message queue 34, and a producer 35 of the trigger message queue 34.

[0094] The producer 35 of the trigger message queue 34 is used to write a trigger message to the trigger message queue 34 when the corresponding trigger condition is met. The trigger message queue 34 is used to carry the written trigger messages, and the trigger messages in the trigger message queue carry information for instructing the user to trigger the cleanup detection.

[0095] Based on the above, the process by which the first processing unit 31 determines whether the total number of business data records of the target user in the database exceeds the data retention threshold may include: Retrieve the trigger message from the trigger message queue 34, and determine the user to be cleaned up and detected based on the retrieved trigger message, and select the target user. The system detects whether the total number of business data records for the target user in the database exceeds the data retention threshold.

[0096] For example, the process by which the first processing unit 31 detects whether the total number of business data records of the target user in the database exceeds the data retention threshold may include: The system queries the creation time of the target data in the database. If the query does not find the target data, it determines that the total number of business data records of the target user in the database does not exceed the data retention threshold; otherwise, it determines that the total number of business data records of the target user in the database exceeds the data retention threshold.

[0097] The target data is either the Nth business data record (i.e., the retained boundary data) or the N+1th business data record (i.e., the cleaned boundary data) in the business data records corresponding to the target user and arranged in descending order of creation time, where N is the data retention threshold.

[0098] Based on this, the data cleanup scope indication information carried in the deletion message obtained by the deletion message consumer 33 may include: the creation time of the data to be retained at the boundary, or the creation time of the data to be cleaned at the boundary. Therefore, the process by which the deletion message consumer 33 determines the target data cleanup scope for this cleanup task based on the obtained deletion message may include: Based on the data cleanup range indication information carried in the obtained deletion message, business data records whose creation time is earlier than the creation time of the retained boundary data, or whose creation time is no later than the creation time of the cleanup boundary data, are determined as the target data cleanup range for this cleanup task.

[0099] Optionally, the producer 35 of the trigger message queue 34 can be a second processing unit (such as a data writing module or a database insert trigger). Then, the process by which the second processing unit writes a trigger message to the trigger message queue when the corresponding trigger condition is met can include: In response to the writing of new business data records to the database, a trigger message is written to the trigger message queue.

[0100] The trigger message written by the second processing unit carries information indicating the user to whom the new business data record belongs.

[0101] Optionally, the producer 35 of the trigger message queue 34 can be a third processing unit, which is used to configure a corresponding activity status identifier for each user based on the business data records of each user within a preset time window. The process by which the third processing unit writes a trigger message to the trigger message queue when the corresponding trigger condition is met can include: After configuring an activity status identifier for the current user, a trigger message is written to the trigger message queue.

[0102] The trigger message written by the third processing unit carries information indicating the current user. The current user is the user among all users in the business system whose activity status is being configured by the third processing unit.

[0103] Based on this, the data retention threshold can be configured based on the target user's activity status identifier, with the data retention threshold being higher when the target user is in an active state than when the target user is in an inactive state.

[0104] In one or more embodiments provided in this application, the device may further include: a fourth processing unit, such as a business configuration manager, which can be used to pull configuration parameters (such as batch deletion threshold, data retention threshold for different types of users / the same user in different states, time window length, etc.) from the configuration center when the business system starts up and store them in local memory.

[0105] The data cleaning solution provided in this application is illustrated below with examples.

[0106] Assume that the data retention threshold for user a in the inactive state is M. In the initial inspection stage, or in the service stage (when new service data records of user a are written into the database), the third processing unit or the second processing unit writes a trigger message carrying the identifier of user a into the trigger message queue. The trigger message consumer performs cleanup detection on user a based on this message, and queries the creation time time_a of the Mth service data set in the service data records of user a sorted in descending order by creation time. If the number of data records in the database is less than M, the query misses, no deletion is triggered, and the current consumption is directly ended. If the number of data records in the database is not less than M, the query hits, deletion is triggered, and a deletion message is written into the deletion message queue, where the written deletion message can be expressed as (a, time_a). The deletion message consumer performs data cleanup based on this message, determines that the target user is user a, and the data range to be cleaned up is service data records whose creation time is earlier than time_a. Assume that the current moment is T0, a batch deletion (with a batch deletion threshold of K) is performed once on the service data records of user a stored in the database whose creation time is earlier than time_a. If the actual amount of deleted data returned by this deletion operation is K, it can indicate that the total amount of service data records of user a in the database at moment T0 is not less than M+K. After this deletion operation, the total amount of service data records of user a in the database is still not less than M, and the data cleanup task for user a is not completed. The deletion message consumer writes a new deletion message into the deletion message queue, where the written deletion message can be expressed as (a, time_a), and then ends the current consumption. The newly written deletion message is located at the end of the deletion message queue, and waits for the deletion message consumer to continue cleaning up the service data records of user a through the message re-queuing mechanism. If the actual amount of deleted data returned by this deletion operation is k (k<K), it can indicate that the total amount of service data records of user a in the database at moment T0 is M+k. After this deletion operation, the total amount of service data records of user a in the database is M, which meets the data retention restriction requirement, the data cleanup task for user a has been completed, and the deletion message consumer directly ends the current consumption.

[0107] In combination with the above content, when the first processing unit performs cleanup detection, it does not split the full cleanup requirement of the user, writes the data cleanup requirement into the deletion message queue as a whole message. When the deletion message consumer consumes this deletion message, the full cleanup task is implicitly split into multiple physical executions based on the limit of the batch deletion threshold, and the full data cleanup requirement is finally met by cyclically delivering deletion messages. Since there is no need to actively split a large-batch deletion task into multiple subtasks for separate delivery, the processing cost and message cost in the data cleanup detection stage are saved.

[0108] The user data cleaning device provided in this application embodiment can be applied to electronic devices, such as terminals with data processing capabilities: mobile phones, computers, servers, etc. Optionally, Figure 4 A hardware block diagram of the electronic device is shown, with reference to... Figure 4 The hardware structure of an electronic device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4; In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4; Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device; The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement any of the user data cleaning methods described above.

[0109] This application also provides a storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the user data cleanup methods provided in this application.

[0110] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the user data cleanup methods provided in this application.

[0111] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0113] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A user data cleaning method, characterized in that, Applied to deleting message consumers, the method includes: Retrieve deletion messages from the deletion message queue. The deletion messages in the deletion message queue carry cleanup task information, which includes user information and data cleanup scope indication information. Based on the obtained deletion messages, determine the target users and the scope of target data cleanup for this cleanup task; Batch deletion of business data records in the database corresponding to the target user and the target data cleanup scope, wherein the total amount of data deleted in the batch does not exceed a preset batch deletion threshold; If the actual amount of data deleted is equal to the batch deletion threshold, then a new deletion message is written to the deletion message queue based on the obtained deletion message; otherwise, no message is written. The obtained deletion message is either a message written by the first processing unit when a first condition is met, or a message written by the deletion message consumer when a second condition is met; the first condition is that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold, and the second condition is that the actual amount of data deleted when executing the historical cleanup task of the target user is equal to the batch deletion threshold.

2. The user data cleaning method according to claim 1, characterized in that, The data cleanup scope indication information carried in the obtained deletion message includes: the creation time of the data to be retained at the boundary, or the creation time of the data to be cleaned at the boundary; Based on the obtained deletion messages, the target data cleanup scope for this cleanup task is determined to include: Based on the data cleanup range indication information carried in the obtained deletion message, business data records whose creation time is earlier than the creation time of the retained boundary data, or business data records whose creation time is no later than the creation time of the cleanup boundary data, are determined as the target data cleanup range for this cleanup task.

3. The user data cleaning method according to claim 1 or 2, characterized in that, The target user is the user determined by the first processing unit based on the trigger message obtained from the trigger message queue. The first processing unit is configured to: after determining the target user based on the obtained trigger message, detect whether the total number of business data records of the target user in the database exceeds the data retention threshold. The obtained trigger message is either: a message written by the second processing unit to the trigger message queue in response to the new business data record of the target user being written to the database, or a message written by the third processing unit to the trigger message queue after configuring an activity status identifier for the target user, wherein the activity status identifier of the target user is set based on the business data record status of the target user within a preset time window.

4. The user data cleaning method according to claim 3, characterized in that, The data retention threshold is configured based on the activity status identifier of the target user. The data retention threshold when the target user is in an active state is higher than the data retention threshold when the target user is in an inactive state.

5. A user data cleaning method, characterized in that, The method includes: When the first processing unit determines that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold, it writes a deletion message to the deletion message queue. The deleted message carries the user information of the target user and data cleanup range indication information configured based on the data over-limit situation of the target user. The deletion message consumer retrieves deletion messages from the deletion message queue. Based on the retrieved deletion messages, it determines the users and data cleanup scope of the current cleanup task. It then performs batch deletion on the business data records in the database corresponding to the users and data cleanup scope of the current cleanup task. The total amount of data deleted in the batch does not exceed a preset batch deletion threshold. If the actual amount of data deleted is equal to the batch deletion threshold, a new deletion message is written to the deletion message queue based on the retrieved deletion messages; otherwise, no new message is written. The retrieved deletion messages are either the deletion messages written by the first processing unit or the messages written by the deletion message consumer when a second condition is met. The second condition is that the actual amount of data deleted during the execution of a historical cleanup task is equal to the batch deletion threshold, and the historical cleanup task has the same users as the current cleanup task.

6. A user data cleaning device, characterized in that, The device includes: a first processing unit, a message deletion queue, and a message deletion consumer; The first processing unit is used to write a deletion message to the deletion message queue when it is determined that the total number of business data records of the target user in the database exceeds the pre-configured data retention threshold. The deletion message written by the first processing unit carries the user information of the target user and data cleanup range indication information configured based on the data over-limit situation of the target user. The deletion message consumer is used to retrieve deletion messages from the deletion message queue, determine the users and data cleanup scope of the current cleanup task based on the retrieved deletion messages, and perform batch deletion of business data records in the database corresponding to the users and data cleanup scope of the current cleanup task. The total amount of data deleted in the batch does not exceed a preset batch deletion threshold. Furthermore, a new deletion message is written to the deletion message queue based on the retrieved deletion messages only when the actual amount of data deleted equals the batch deletion threshold. The retrieved deletion messages are either: deletion messages written by the first processing unit, or messages written by the deletion message consumer when a second condition is met, whereby the actual amount of data deleted during the execution of a historical cleanup task equals the batch deletion threshold, and the historical cleanup task has the same users as the current cleanup task. The deletion message queue is used to carry the written deletion messages.

7. The user data cleaning device according to claim 6, characterized in that, The device also includes: a trigger message queue, and a producer of the trigger message queue; The producer of the trigger message queue is used to write a trigger message to the trigger message queue when the corresponding trigger condition is met; The trigger message queue is used to carry the written trigger messages, and the trigger messages in the trigger message queue carry information for instructing the user to trigger the cleanup detection; The process by which the first processing unit determines whether the total number of business data records of the target user in the database exceeds the data retention threshold includes: The trigger message is obtained from the trigger message queue, and the user to be cleaned up and detected is determined based on the obtained trigger message, which is then used as the target user. The system detects whether the total number of business data records of the target user in the database exceeds the data retention threshold.

8. The user data cleaning device according to claim 7, characterized in that, The process by which the first processing unit detects whether the total number of business data records of the target user in the database exceeds the data retention threshold includes: Query the creation time of the target data in the database. The target data is the Nth or N+1th business data record in the business data records corresponding to the target user and sorted in descending order of creation time. N is the data retention threshold. If the query fails to find the target user, it is determined that the total number of business data records of the target user in the database does not exceed the data retention threshold; otherwise, it is determined that the total number of business data records of the target user in the database exceeds the data retention threshold.

9. The user data cleaning device according to claim 8, characterized in that, The producer that triggers the message queue includes: a second processing unit; The process by which the second processing unit writes a trigger message to the trigger message queue when the corresponding trigger condition is met includes: In response to a new business data record being written to the database, a trigger message is written to the trigger message queue; the trigger message written by the second processing unit carries information indicating the user to whom the new business data record belongs.

10. The user data cleaning device according to claim 8, characterized in that, The producer of the trigger message queue includes a third processing unit; the third processing unit is used to configure a corresponding activity status identifier for each user according to the business data record of each user within a preset time window. The data retention threshold is configured based on the activity status identifier of the target user, and the data retention threshold when the target user is in an active state is higher than the data retention threshold when the target user is in an inactive state. The process by which the third processing unit writes a trigger message to the trigger message queue when the corresponding triggering condition is met includes: After configuring an activity status identifier for the current user, a trigger message is written to the trigger message queue; the trigger message written by the third processing unit carries information for indicating the current user.

Images