Log data batch updating method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a log data batch updating method and device, equipment and a medium, and the method comprises the steps: obtaining a full-amount historical log in a preset database in real time, converting the full-amount historical log into an event stream, and writing the event stream into preset message middleware, obtaining a partition ID of the total historical log in the message middleware, generating a global unique identifier bound with a partition according to the partition ID to obtain a distributed ID, generating a consumer ID associated with the user identity information according to the distributed ID by calling a pre-designed distributed algorithm, and sending the consumer ID to the message middleware. And carrying out lock-free parallel updating on the database according to the consumer ID based on connection pool preheating and asynchronous submission to obtain an updating result, carrying out consistency verification on the updating result to obtain a verification result, and confirming the updating result according to the verification result. And the batch updating efficiency of the log data is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for batch updating log data. Background Technology

[0002] In the sharded architecture of internet applications, when it is necessary to supplement existing data with additional fields, existing technologies typically employ the following methods: Full traversal processing, i.e., using paginated queries to traverse all physical tables and execute SELECT-APPLY-UPDATE operations row by row; reliance on strong transactions, using FOR UPDATE row locks or optimistic locking version number mechanisms to ensure data consistency, resulting in prolonged occupation of the database connection pool; reliance on single-point ID generation, where a centralized Snowflake (a distributed ID generation algorithm) service generates CustomerIDs, with ID generation speed limited by the performance of a single node; and offline batch processing, relying on scheduled tasks to execute updates in batches, which carries the risk of data version drift and omissions between batches.

[0003] However, these existing technologies have the following drawbacks: low processing efficiency, averaging over 20 hours with 150 million data entries, and an 80% drop in database QPS (Queries Per Second) during peak periods; uncontrolled resource consumption, with full table scans generating 5 million random I / O operations and peak Java database connection pool occupancy reaching 95%; significant system vulnerability, with a single point of failure in Snowflake causing a complete system outage, and uncommitted long transactions blocking online business update operations; and data quality defects, with the pagination offset mechanism causing 0.15% of data to be repeatedly updated or missed, making it difficult to meet the needs of efficient and reliable processing of large-scale existing data. Summary of the Invention

[0004] This invention provides a method, apparatus, computer device, and medium for batch updating log data, in order to solve the problem of low efficiency in existing batch updating methods for log data.

[0005] Firstly, a method for batch updating log data is provided, including: The system acquires all historical logs from a preset database in real time and converts the full historical logs into an event stream, which is then written into a preset message middleware. Obtain the partition ID of the full historical log in the message middleware, and generate a globally unique identifier bound to the partition based on the partition ID to obtain a distributed ID; A consumer ID associated with the user's identity information is generated by invoking a pre-designed distributed algorithm based on the distributed ID; Based on connection pool preheating and asynchronous commit, the database is updated in a lock-free parallel manner according to the consumer ID to obtain the update result; The update result is subjected to a consistency check to obtain a check result, and the update result is confirmed based on the check result.

[0006] Secondly, a batch update device for log data is provided, including: Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described batch update method for log data.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described batch update method for log data.

[0008] In the aforementioned solution implemented by the batch update method, device, computer equipment, and storage medium for log data, the Xuanwu component can be used to obtain the full historical logs of MySQL in real time, extract the mapping relationship between user ID numbers and uids, and convert it into an INSERT event stream. This stream is then distributed to the corresponding Kafka partition according to the uid % 10000 rule. Based on the Kafka partition ID, a WorkerID is dynamically bound, and a globally unique distributed ID (i.e., consumer ID customerId) is generated by combining a hybrid logical timestamp (44 bits) and a 9-bit auto-incrementing sequence (supporting 512 IDs per millisecond per node), meeting the business requirement of "one customerId corresponding to multiple uids." Through connection pool preheating and allocation of dedicated threads according to the number of Kafka partitions, a "partition-thread-connection" mapping is formed. After parsing the log fields, an update statement with optimistic locking is constructed. Every 5000 log entries... The update is submitted and executed via an asynchronous thread pool, achieving lock-free parallel updates through serial processing within partitions. The number of affected rows is identified by version number verification; if it is 0, a version conflict alarm is triggered, otherwise the update is confirmed as successful. Consistency verification is performed on the update results. First, the unique index of customerId is checked to ensure no duplicates. Then, the partition difference between the consumed points of each Kafka partition and the corresponding updated rows in the database is calculated. Combined with the partition weights (core 1.5, normal 1.0, edge 0.8), the overall deviation rate is obtained. If the deviation rate is ≤0.001%, the update is confirmed as valid; otherwise, an alarm is triggered, thus improving the efficiency of batch updates of log data. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a batch update method for log data in one embodiment of the present invention; Figure 2 This is a flowchart illustrating a batch update method for log data according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a log data batch update device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The batch update method for log data provided in this embodiment of the invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via the network. The server can use the Xuanwu component on the client to obtain the full historical logs of MySQL in real time, extract the mapping relationship between user ID numbers and UIDs, and convert it into an INSERT event stream. This stream is then distributed to the corresponding Kafka partition according to the UID%10000 rule. Based on the Kafka partition ID, a WorkerID is dynamically bound, and a globally unique distributed ID (i.e., consumer ID customerId) is generated by combining a hybrid logical timestamp (44 bits) and a 9-bit auto-incrementing sequence (supporting 512 IDs per millisecond per node), meeting the business requirement of "one customerId corresponding to multiple UIDs". Through connection pool preheating and allocation of dedicated threads according to the number of Kafka partitions, a "partition-thread-connection" mapping is formed. After parsing the log fields, an update statement with optimistic locking is constructed, and updates are performed every 5000 connections. The update is submitted and executed via an asynchronous thread pool, utilizing serial processing within partitions to achieve lock-free parallel updates. A version number check identifies the number of affected rows; if the number is 0, a version conflict alarm is triggered; otherwise, the update is confirmed successful. Consistency checks are performed on the update results. First, the unique index of `customerId` is checked to ensure no duplicates. Then, the partition difference between the consumed points in each Kafka partition and the corresponding updated rows in the database is calculated. Combined with partition weights (core 1.5, normal 1.0, edge 0.8), the overall deviation rate is obtained. If the deviation rate is ≤0.001%, the update is confirmed valid; otherwise, an alarm is triggered, improving the efficiency of batch updates of log data. Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0013] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a batch log data update method provided in an embodiment of the present invention includes the following steps: S1. Obtain the full historical logs from the preset database in real time, and convert the full historical logs into an event stream and write it into the preset message middleware.

[0014] In this embodiment of the invention, the preset database may be a MySQL database.

[0015] In this embodiment of the invention, the full historical logs in the preset database can be obtained in real time using the preset Xuanwu component.

[0016] In this embodiment of the invention, the preset message middleware may refer to Kafka.

[0017] In this embodiment of the invention, the step of converting the full historical log into an event stream and writing it into a preset message middleware may involve obtaining the existing data in the full historical log, converting the existing data (the original uid-ID number mapping records of the sharded database and tables) into an INSERT event stream, and distributing the data to the corresponding partition of the message middleware (Kafka) according to the preset sharding rules (such as uid % 10000).

[0018] In this embodiment of the invention, the step of converting the full historical log into an event stream and writing it to a preset message middleware includes: Obtain the mapping records between user identity information and user unique identifiers from the full historical logs to obtain the mapping record data; The mapped record data is converted into an event stream; Based on preset database sharding and table partitioning rules, the event stream is sent to the corresponding partition of the preset message middleware.

[0019] In detail, obtaining the mapping records of user identity information and user unique identifiers in the full historical logs means accurately extracting the correspondence between user ID number and UID (user unique identifier) ​​from the full historical logs. These mapping records are the basis for subsequent data processing.

[0020] In detail, the mapping record data is converted into an event stream. According to a preset event format, each mapping record is transformed into a standardized event stream. Each event contains key information such as credit_id, uid, and event type (e.g., INSERT). This conversion makes the data more suitable for transmission and processing in message brokers.

[0021] In detail, the event stream is sent to the corresponding partition of the preset message middleware based on the preset database sharding and table partitioning rules. The partition to which each event belongs is determined according to the database sharding and table partitioning rules, for example, using the rule of uid % 10000. The partition number corresponding to the uid in each event is calculated, and then the event is sent to that partition in the message middleware. This ensures that related data under the same database sharding and table partitioning rules enter the same partition.

[0022] S2. Obtain the partition ID of the full historical log in the message middleware, generate a globally unique identifier bound to the partition based on the partition ID, and obtain a distributed ID.

[0023] In this embodiment of the invention, after the full historical log is written to the message middleware through partitions, each partition of the message middleware has a partition ID.

[0024] In this embodiment of the invention, the step of generating a globally unique identifier bound to the partition based on the partition ID to obtain a distributed ID includes: Use the partition ID as the worker ID; Generate a hybrid logical timestamp based on local physical time; Generate an auto-incrementing sequence based on the worker ID and the hybrid logical timestamp; The worker ID, the HLC hybrid logical timestamp, and the auto-incrementing sequence are concatenated based on a preset fixed format to obtain a distributed ID.

[0025] In detail, using the partition ID as the worker ID means directly using the obtained Kafka partition ID as the worker ID, establishing a strong mapping relationship of "partition ID ↔ worker ID". For example, if the Kafka partition ID is 3, the corresponding worker ID is also 3. This ensures that the ID generation of each partition is identified by a unique worker ID, making the ID generation process of each partition independent and unique. This avoids different partitions generating duplicate IDs at the same timestamp, thus solving the problem of ID conflicts among multiple nodes in a distributed system.

[0026] In detail, the generation of hybrid logical timestamps based on local physical time involves using an HLC (Hybrid Logical Clock) to replace the traditional physical clock, generating a 44-bit timestamp to solve the ID duplication problem caused by clock rollback in distributed systems. Each node of the HLC maintains a local logical counter, and when physical time has not advanced, the timestamp is monotonically increased by incrementing the counter.

[0027] In detail, the step of generating an auto-incrementing sequence based on the worker ID and the hybrid logical timestamp involves generating a 9-bit auto-incrementing sequence (within the range of 0 to 511) within the same WorkerID and HLC timestamp, ensuring that a maximum of 512 unique IDs can be generated within a single millisecond on a single node.

[0028] S3. Generate a consumer ID associated with the user's identity information by calling a pre-designed distributed algorithm based on the distributed ID.

[0029] In this embodiment of the invention, the step of generating a consumer ID associated with user identity information based on the distributed ID using a pre-designed distributed algorithm involves an application thread parsing binlog events in a Kafka message, extracting fields such as uid and credit_id, and then calling the Snowflake algorithm (i.e., the distributed ID generation logic based on partition ID in S2) to generate a unique customerId. This ID corresponds one-to-one with an ID card number, serving as a substitute identifier for sensitive information and meeting the business requirement of "one customerId corresponding to multiple uids".

[0030] In this embodiment of the invention, the step of generating a consumer ID associated with the user's identity information by calling a pre-designed distributed algorithm based on the distributed ID is achieved by using the application layer to subscribe to Kafka partitions in the form of consumer groups, with each thread consuming one partition independently.

[0031] In this embodiment of the invention, the user identity information may refer to the user's ID card number.

[0032] In this embodiment of the invention, a pre-designed distributed algorithm is invoked to generate a consumer ID associated with the user's identity information based on the distributed ID. This enables one consumer ID to correspond to multiple unique user identifiers (uids), meaning that different uids of the same user (same ID number) will be mapped to the same customerId. This mapping of one consumer ID to multiple unique user identifiers ensures that the CustomerID generation logic for the same UID is always executed by the same physical node, thus eliminating ID conflicts at the source.

[0033] S4. Based on connection pool preheating and asynchronous commit, perform lock-free parallel update on the database according to the consumer ID to obtain the update result.

[0034] In this embodiment of the invention, when performing lock-free parallel updates to the database based on the consumer ID using connection pool preheating and asynchronous commit, and obtaining the update result, optimistic locking can be implemented by setting the WHERE customer_id IS NULL condition to reduce database lock contention.

[0035] In detail, the optimistic locking implemented by setting the WHERE customer_id IS NULL condition to reduce database lock contention is based on the fact that in the scenario of batch update of log data, the initial null value of the consumer ID (customer_id) is used as an indicator that the data has not been updated. During the update, the operation is only performed on the records where customer_id is still NULL. This method does not require the addition of row locks or table locks in advance. Instead, it confirms that the data has not been modified by other threads by checking the condition at the moment of update. This ensures that the same row of data will only be successfully updated by one thread (after the first update, customer_id becomes non-null, and the update condition of subsequent threads is not met). It also avoids the thread blocking and resource occupation caused by traditional locking mechanisms, thereby reducing database lock contention, adapting to the requirements of lock-free parallel updates, and improving the efficiency of batch update.

[0036] In this embodiment of the invention, the optimistic locking is a concurrency control mechanism. Its core idea is to "optimistically assume that the data will not be modified by other threads during the update process." Therefore, it does not lock in advance (such as row locks or table locks), but instead determines whether the data has been tampered with "at the moment the update operation is executed" through specific logic, thereby ensuring data consistency.

[0037] In this embodiment of the invention, the lock-free approach is a macro-architectural concept. Its core is to "not rely on traditional locking mechanisms (such as row locks, table locks, and distributed locks) to control concurrent access." Instead, it uses designs such as "architectural isolation" and "logical judgment" to allow multiple threads / nodes to naturally avoid conflicts when processing data in parallel, thereby achieving "safe concurrency in a lock-free state."

[0038] In this embodiment of the invention, the lock-free parallel update of the database based on the consumer ID, obtained by connection pool preheating and asynchronous commit, includes: Preheat the connection pool based on preset connection pool configuration parameters; Based on the number of partitions in the message middleware, threads are allocated to the preheated connection pool to obtain a thread mapping table; Based on the thread mapping table, the full historical log is parsed by thread calls to obtain a list of parsed fields; Construct a database update statement based on the list of parsed fields and the consumer ID; The number of database update statements generated in real time is obtained; When the number of generated data reaches a preset threshold, the database is updated asynchronously according to the database update statement to obtain the update result.

[0039] In detail, the connection pool preheating based on preset connection pool configuration parameters can involve creating 200 database connections in advance and verifying their validity using a SELECT 1 statement to ensure all connections are active. This step avoids the performance overhead of frequently creating connections during real-time processing and provides ready-to-use connection resources for subsequent batch updates.

[0040] In detail, the thread allocation to the preheated connection pool based on the number of partitions of the message middleware is used to obtain a thread mapping table. This can be done by creating 10,000 threads according to the number of Kafka partitions, with each thread obtaining one dedicated database connection from the preheated connection pool, forming a mapping relationship of "partition ID-thread-connection" (e.g., partition 3 → thread 3 → connection 3).

[0041] In detail, the process involves using the thread mapping table to parse the full historical logs via thread calls, resulting in a list of parsed fields. This parsing process extracts non-sensitive information according to field rules: extracting the uid (unique user identifier), calculating the shard ID (e.g., uid % 10000), and filtering sensitive fields such as credit_id (identity ID). The parsing process is parallelized through thread isolation, improving log parsing efficiency.

[0042] Specifically, the database update statement can be "UPDATE table_{N} SET customer_id ={snowflake_id} WHERE uid = {uid}".

[0043] Specifically, the preset threshold can be 5000. That is, it is executed once every 5000 accumulated update statements.

[0044] In this embodiment of the invention, the core of the lock-free update lies in the fact that data within the same Kafka partition is processed by a single thread, ensuring that the update order is consistent with the shards, without the need for distributed locks or row locks. In this embodiment of the invention, the step of asynchronously executing a lock-free update of the database based on the constructed database update statement to obtain the update result includes: Submit the database update statement to a preset asynchronous thread pool; Identify the idle threads in the asynchronous thread pool; The idle thread is used to execute the database update statement to obtain the version number corresponding to the updated consumer ID in the database. The version number corresponding to the consumer ID is compared with the preset expected version number to obtain the comparison result. Based on the comparison result, the number of data rows in the database that have been updated is determined, and the number of affected rows is obtained. If the number of affected lines is 0, it is determined to be a version conflict, and a version conflict warning is sent to the preset developer client; If the number of affected rows is not 0, the update is confirmed to be successful. All updated data in the database is then aggregated to obtain the update result.

[0045] In detail, submitting the database update statement to the preset asynchronous thread pool involves encapsulating the update statement into a Runnable or Callable task and submitting it to the task queue of the thread pool for scheduling and execution by the thread pool.

[0046] In detail, the version number verification based on the version number refers to verifying whether the version number meets expectations (such as whether it is the next version of the current transaction).

[0047] S5. Perform a consistency check on the update result to obtain the check result, and confirm the update result based on the check result.

[0048] In this embodiment of the invention, the step of performing a consistency check on the update result to obtain a check result includes: Obtain the updated consumer ID contained in the update result; Get the unique index for each updated consumer ID; Determine whether there are duplicate consumer IDs based on the unique index; If duplicate consumer IDs exist, the verification result is confirmed as a consistency check failure. If there are no duplicate consumer IDs, then based on the update result, obtain the cumulative number of message location markers that have been successfully processed and submitted maintained by the consumer group of the message middleware for each partition, and obtain the number of consumed points. Calculate the deviation rate between the number of consumed points and the number of affected rows; If the deviation rate is greater than the preset deviation rate threshold, then the verification result is confirmed as a failure of consistency verification. If the deviation rate is less than or equal to the deviation rate threshold, then the verification result is confirmed as a successful consistency verification.

[0049] In detail, obtaining the unique index for each updated consumer ID refers to obtaining the unique index for the consumer ID field from the MySQL table.

[0050] Specifically, calculating the deviation rate between the number of consumed points and the number of affected rows includes: Calculate the difference between the consumed data points and the number of updated rows corresponding to different partitions in the message middleware to obtain the partition difference; Calculate the ratio of the partition difference to the consumed point data corresponding to different partitions to obtain the partition deviation rate of different partitions; The deviation rate is obtained by weighting and fusing the partition deviation rates based on preset partition weights.

[0051] Specifically, the different partitions in the message middleware may include core business partitions, ordinary partitions, and edge partitions, and further, the partition weights are 1.5, 1, and 0.8, respectively.

[0052] Specifically, the deviation rate threshold can be 0.001.

[0053] In this embodiment of the invention, the consumer group of the message middleware (Kafka) maintains the consumption progress independently for each partition. This progress is based on the position marker (Offset) corresponding to the messages that have been successfully processed and committed. During the process of obtaining the update result, the committed Offset value recorded by the consumer group for each partition is read by calling the built-in interface of the message middleware (such as Kafka's consumer.position() or offsetsForTimes()). The cumulative count of this value is the number of consumed positions in the corresponding partition, which directly reflects the total number of messages that have been successfully processed in the partition.

[0054] In this embodiment of the invention, confirming the update result based on the verification result means that if the verification result is a pass for consistency verification, the update result is confirmed to be valid; if the verification result is a fail for consistency verification, the update result is confirmed to be unsuccessful, and an error message is sent to the preset developer user terminal.

[0055] As can be seen, in the above solution, the Xuanwu component is used to obtain the full historical logs of MySQL in real time, extract the mapping relationship between user ID numbers and UIDs, and convert it into an INSERT event stream. This stream is then distributed to the corresponding Kafka partition according to the UID % 10000 rule. Based on the Kafka partition ID, a WorkerID is dynamically bound, and a globally unique distributed ID (i.e., the consumer ID customerId) is generated by combining a hybrid logical timestamp (44 bits) and a 9-bit auto-incrementing sequence (supporting 512 IDs per millisecond per node), meeting the business requirement of "one customerId corresponding to multiple UIDs". A "partition-thread-connection" mapping is formed through connection pool preheating and allocation of dedicated threads according to the number of Kafka partitions. After parsing the log fields, an update statement with optimistic locking is constructed, and every 5000 connections... The update is submitted and executed via an asynchronous thread pool, utilizing serial processing within the partition to achieve lock-free parallel updates. The number of affected rows is identified by version number verification; if it is 0, a version conflict alarm is triggered; otherwise, the update is confirmed as successful. Consistency verification is performed on the update results. First, the unique index of customerId is checked to ensure no duplicates. Then, the partition difference between the consumed points of each Kafka partition and the corresponding updated rows in the database is calculated. Combined with the partition weights (core 1.5, normal 1.0, edge 0.8), the overall deviation rate is obtained. If the deviation rate is ≤0.001%, the update is confirmed as valid; otherwise, an alarm is triggered, thus improving the efficiency of log data processing.

[0056] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0057] In one embodiment, a log data batch update apparatus is provided, which corresponds one-to-one with the log data batch update method in the above embodiments. For example... Figure 3 As shown, the log data batch update device includes a data acquisition module 101, a distributed ID generation module 102, a consumer ID generation module 103, a parallel update module 104, and a consistency verification module 105. Detailed descriptions of each functional module are as follows: The data acquisition module 101 is used to acquire the full historical logs in the preset database in real time, and convert the full historical logs into an event stream and write it into the preset message middleware. The distributed ID generation module 102 is used to obtain the partition ID of the full historical log in the message middleware, generate a globally unique identifier bound to the partition based on the partition ID, and obtain a distributed ID. The consumer ID generation module 103 is used to generate a consumer ID associated with the user's identity information by calling a pre-designed distributed algorithm based on the distributed ID; Parallel update module 104 is used to perform lock-free parallel update on the database based on the consumer ID according to the connection pool preheating and asynchronous commit, and obtain the update result; The consistency verification module 105 is used to perform consistency verification on the update result, obtain the verification result, and confirm the update result based on the verification result.

[0058] In one embodiment, when the data acquisition module 101 performs the step of converting the full historical log into an event stream and writing it to a preset message middleware, it is specifically used for: Obtain the mapping records between user identity information and user unique identifiers from the full historical logs to obtain the mapping record data; The mapped record data is converted into an event stream; Based on preset database sharding and table partitioning rules, the event stream is sent to the corresponding partition of the preset message middleware.

[0059] In one embodiment, the distributed ID generation module 102, when performing the step of generating a globally unique identifier bound to the partition based on the partition ID to obtain a distributed ID, is specifically used for: Use the partition ID as the worker ID; Generate a hybrid logical timestamp based on local physical time; Generate an auto-incrementing sequence based on the worker ID and the hybrid logical timestamp; The worker ID, the HLC hybrid logical timestamp, and the auto-incrementing sequence are concatenated based on a preset fixed format to obtain a distributed ID.

[0060] In one embodiment, the parallel update module 104, when performing the lock-free parallel update of the database based on the consumer ID according to the connection pool preheating and asynchronous commit to obtain the update result, is specifically used for: Preheat the connection pool based on preset connection pool configuration parameters; Based on the number of partitions in the message middleware, threads are allocated to the preheated connection pool to obtain a thread mapping table; Based on the thread mapping table, the full historical log is parsed by thread calls to obtain a list of parsed fields; Construct a database update statement based on the list of parsed fields and the consumer ID; The number of database update statements generated in real time is obtained; When the number of generated data reaches a preset threshold, the database is updated asynchronously according to the database update statement to obtain the update result.

[0061] In one embodiment, the parallel update module 104, when executing the lock-free update of the database based on the constructed database update statement using asynchronous execution and obtaining the update result, is specifically used for: Submit the database update statement to a preset asynchronous thread pool; Identify the idle threads in the asynchronous thread pool; The idle thread is used to execute the database update statement to obtain the version number corresponding to the updated consumer ID in the database. The version number corresponding to the consumer ID is compared with the preset expected version number to obtain the comparison result. Based on the comparison result, the number of data rows in the database that have been updated is determined, and the number of affected rows is obtained. If the number of affected lines is 0, it is determined to be a version conflict, and a version conflict warning is sent to the preset developer client; If the number of affected rows is not 0, the update is confirmed to be successful. All updated data in the database is then aggregated to obtain the update result.

[0062] In one embodiment, when performing the consistency verification on the update result and obtaining the verification result, the consistency verification module 105 is specifically used for: Obtain the updated consumer ID contained in the update result; Get the unique index for each updated consumer ID; Determine whether there are duplicate consumer IDs based on the unique index; If duplicate consumer IDs exist, the verification result is confirmed as a consistency check failure. If there are no duplicate consumer IDs, then based on the update result, obtain the cumulative number of message location markers that have been successfully processed and submitted maintained by the consumer group of the message middleware for each partition, and obtain the number of consumed points. Calculate the deviation rate between the number of consumed points and the number of affected rows; If the deviation rate is greater than the preset deviation rate threshold, then the verification result is confirmed as a failure of consistency verification. If the deviation rate is less than or equal to the deviation rate threshold, then the verification result is confirmed as a successful consistency verification.

[0063] In one embodiment, the consistency verification module 105, when performing the calculation of the deviation rate between the number of consumed points and the number of affected rows, is specifically used for: Calculate the difference between the consumed data points and the number of updated rows corresponding to different partitions in the message middleware to obtain the partition difference; Calculate the ratio of the partition difference to the consumed point data corresponding to different partitions to obtain the partition deviation rate of different partitions; The deviation rate is obtained by weighting and fusing the partition deviation rates based on preset partition weights.

[0064] This invention provides a batch log data update device. It utilizes the Xuanwu component to acquire the full historical logs of MySQL in real time, extracts the mapping relationship between user ID numbers and UIDs, and converts it into an INSERT event stream. This stream is then distributed to the corresponding Kafka partitions according to the UID % 10000 rule. Based on the Kafka partition ID, a WorkerID is dynamically bound, and a globally unique distributed ID (i.e., consumer ID customerId) is generated by combining a hybrid logical timestamp (44 bits) and a 9-bit auto-incrementing sequence (supporting 512 IDs per millisecond per node), meeting the business requirement of "one customerId corresponding to multiple UIDs." Through connection pool preheating and allocation of dedicated threads according to the number of Kafka partitions, a "partition-thread-connection" mapping is formed. After parsing log fields, an update statement with optimistic locking is constructed. Every 5000 log entries... The update is submitted and executed via an asynchronous thread pool, achieving lock-free parallel updates through serial processing within partitions. The number of affected rows is identified by version number verification; if it is 0, a version conflict alarm is triggered, otherwise the update is confirmed as successful. Consistency verification is performed on the update results. First, the unique index of customerId is checked to ensure no duplicates. Then, the partition difference between the consumed points of each Kafka partition and the corresponding updated rows in the database is calculated. Combined with the partition weights (core 1.5, normal 1.0, edge 0.8), the overall deviation rate is obtained. If the deviation rate is ≤0.001%, the update is confirmed as valid; otherwise, an alarm is triggered, thus improving the efficiency of batch updates of log data.

[0065] Specific limitations regarding the log data batch update device can be found in the limitations of the log data batch update method described above, and will not be repeated here. Each module in the aforementioned log data batch update device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0066] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a log data batch update method on the server side.

[0067] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a batch log data update method. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the user's bank card status information; Determine whether there is a card on the preset card reader based on the status information; If there is no card on the preset card reader, return to the step of obtaining the user's bank card status information; If there is a card on the preset card reader, the bank card information is obtained, and the user's ID card is read to obtain the ID card data; Generate images of the front and back of the document based on the ID card data; Obtain the key information of the preset symmetric key and obtain the plaintext key input by the user; Determine whether the key information is the same as the plaintext key; If the key information is different from the plaintext key, then the user authentication is deemed to have failed. If the key information is the same as the plaintext key, the user authentication is deemed successful, and service access is granted based on the bank card information and the front and back images of the identification document.

[0068] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the user's bank card status information; Determine whether there is a card on the preset card reader based on the status information; If there is no card on the preset card reader, return to the step of obtaining the user's bank card status information; If there is a card on the preset card reader, the bank card information is obtained, and the user's ID card is read to obtain the ID card data; Generate images of the front and back of the document based on the ID card data; Obtain the key information of the preset symmetric key and obtain the plaintext key input by the user; Determine whether the key information is the same as the plaintext key; If the key information is different from the plaintext key, then the user authentication is deemed to have failed. If the key information is the same as the plaintext key, the user authentication is deemed successful, and service access is granted based on the bank card information and the front and back images of the identification document.

[0069] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0070] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0072] Finally, it should be noted that if any software tools or components not belonging to this company appear in the embodiments of the application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A log data batch update method, characterized by, The application comprises the following steps: Real-time acquisition of full-amount historical logs in a preset database, conversion of the full-amount historical logs into an event stream and writing of the event stream into a preset message middleware; Acquisition of partition ID of the full-amount historical logs in the message middleware, generation of a globally unique identification bound with a partition according to the partition ID and obtaining of a distributed ID; Generation of a consumer ID associated with user identity information according to the distributed ID by calling a pre-designed distributed algorithm; Non-locking parallel update of the database according to the consumer ID based on connection pool preheating and asynchronous submission, obtaining of an update result; Consistency check of the update result, obtaining of a check result and confirmation of the update result according to the check result.

2. The log data bulk update method of claim 1, wherein, The conversion of the full-amount historical logs into an event stream and writing of the event stream into a preset message middleware comprises the following steps: Obtaining of mapping record data by acquiring mapping records of user identity information and user unique identification in the full-amount historical logs; Conversion of the mapping record data into an event stream; Sending of the event stream to a corresponding partition of the preset message middleware based on a preset database and table splitting rule.

3. The log data bulk update method of claim 1, wherein, The generation of a globally unique identification bound with a partition according to the partition ID and obtaining of a distributed ID comprises the following steps: Taking the partition ID as a worker ID; Generation of a hybrid logical timestamp based on a local physical time; Generation of an auto-increment sequence according to the worker ID and the hybrid logical timestamp; Splicing of the worker ID, the hybrid logical timestamp and the auto-increment sequence based on a preset fixed format to obtain a distributed ID.

4. The log data bulk update method of claim 1, wherein, The non-locking parallel update of the database according to the consumer ID based on connection pool preheating and asynchronous submission and obtaining of an update result comprise the following steps: Connection pool preheating based on a preset connection pool configuration parameter; Thread allocation to the preheated connection pool based on the number of partitions of the message middleware to obtain a thread mapping table; Field parsing of the full-amount historical logs based on the thread mapping table by thread calling to obtain a parsed field list; Construction of a database update statement according to the parsed field list and the consumer ID; Real-time acquisition of the number of generated database update statements; When the number of generated database update statements reaches a preset threshold, non-locking update of the database according to the constructed database update statement based on asynchronous execution to obtain an update result.

5. The log data bulk update method of claim 4, wherein, The non-locking update of the database according to the constructed database update statement based on asynchronous execution to obtain an update result comprises the following steps: Submission of the database update statement to a preset asynchronous thread pool; Identification of an idle thread of the asynchronous thread pool; Execution of the database update statement by the idle thread to acquire a version number corresponding to the updated consumer ID in the database; Comparison of the version number corresponding to the consumer ID with a preset expected version number to obtain a comparison result, confirmation of the number of data rows completed in the update in the database according to the comparison result to obtain an affected row number; If the affected row number is 0, it is determined that there is a version conflict and a version conflict warning is sent to a preset developer user end. If the affected row number is not 0, it is confirmed that the update is successful, all updated data in the database is summarized to obtain an update result.

6. The log data bulk update method of claim 5, wherein, The consistency check is performed on the update result to obtain a check result, comprising: An updated consumer ID contained in the update result is obtained. A unique index of each updated consumer ID is obtained. It is judged whether there is a repeated consumer ID according to the unique index. If there is a repeated consumer ID, it is confirmed that the check result is not passed in the consistency check. If there is no repeated consumer ID, the number of consumed point positions is obtained according to the update result, which is the cumulative number of message middleware consumer group maintained for each partition, successfully processed and submitted message position marks. The deviation rate between the number of consumed point positions and the affected row number is calculated. If the deviation rate is greater than a preset deviation rate threshold, it is confirmed that the check result is not passed in the consistency check. If the deviation rate is less than or equal to the deviation rate threshold, it is confirmed that the check result is passed in the consistency check.

7. The log data bulk update method of claim 1, wherein, The deviation rate between the number of consumed point positions and the affected row number is calculated, comprising: The difference value between the consumed point position data corresponding to different partitions in the message middleware and the update row number is calculated to obtain a partition difference value. The partition deviation rate of different partitions is obtained by calculating the ratio of the partition difference value to the consumed point position data corresponding to different partitions. The partition deviation rate is weighted and fused based on a preset partition weight to obtain the deviation rate.

8. A log data batch update apparatus characterized by comprising: Comprising: A data acquisition module is configured to acquire full-amount historical logs in a preset database in real time, and convert the full-amount historical logs into event streams to write into a preset message middleware. A distributed ID generation module is configured to acquire partition IDs of the full-amount historical logs in the message middleware, and generate globally unique identifiers bound to partitions according to the partition IDs to obtain distributed IDs. A consumer ID generation module is configured to generate consumer IDs associated with user identity information according to the distributed IDs by calling a pre-designed distributed algorithm. A parallel update module is configured to perform lock-free parallel update on the database according to the consumer IDs based on connection pool preheating and asynchronous submission to obtain an update result. A consistency check module is configured to perform consistency check on the update result to obtain a check result, and confirm the update result according to the check result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the log data batch update method in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the log data batch update method in any one of claims 1 to 7.

Images