Message disorder solving method and device for real-time data collection and storage medium

Abstract

The invention belongs to the technical field of change data capture message processing, and provides a message disorder solving method and device for real-time data collection and a storage medium, and the method comprises the steps: obtaining a change data capture message generated by a producer; determining identification information according to the change data capture message; determining a target partition based on a logic planning strategy according to the identification information; the altered data capture message is distributed to the target partition. In a concurrent execution scene, the change data capture messages in the plurality of message partitions are executed according to time sequence consumption, so that the problem of disorder can be avoided, and meanwhile, the concurrent execution efficiency is not influenced. Meanwhile, local sorting in the message partition replaces global sorting, the change data capture messages needing to be processed according to the time sequence are limited in the same message partition through a logic planning strategy, only the local sequence in the message partition needs to be maintained, global management or central global sequence number generation guarantee time sequence is not needed, and calculation overhead can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of change data capture message processing, and particularly to a method, device, and storage medium for solving out-of-order messages in real-time data acquisition. Background Art

[0002] Real-time data acquisition serves scenarios of real-time feedback and decision-making. By continuously capturing dynamic changes in data sources, data is transmitted to downstream systems such as databases, stream processing, etc., thereby enabling real-time analysis, event response, and data synchronization. In this process, change data capture (CDC) is a technology that captures insert, update, and delete operations on data tables and converts them into event messages, used to track data changes in databases in real-time and can efficiently meet the requirements of real-time data acquisition.

[0003] However, although the CDC technology is efficient, in a high-concurrency execution environment, out-of-order problems are likely to occur. When multiple transactions modify the same data entity in parallel, the recording order of database logs may be inconsistent with the logical order of actual business operations, that is, there is an out-of-order message problem. For example, when updating the user status and deleting the user are executed concurrently, if the delete user transaction is committed first, its delete operation may be captured earlier than the update transaction operation, that is, out-of-order occurs, resulting in an error in the update transaction.

[0004] Therefore, in the process of real-time data acquisition, how to avoid out-of-order messages is an urgent problem to be solved. Summary of the Invention

[0005] The present invention provides a method, device, and storage medium for solving out-of-order messages in real-time data acquisition to solve the defect of out-of-order messages in the prior art during the process of real-time data acquisition.

[0006] The present invention provides a method for solving out-of-order messages in real-time data acquisition, including: Obtaining change data capture messages generated by a producer; Determining identification information according to the change data capture messages; Determining a target partition based on a logical planning strategy according to the identification information; Distributing the change data capture messages to the target partition; Wherein, the change data capture messages in the target partition are arranged in chronological order and consumed in chronological order, and the change data capture messages corresponding to the same data table are located in the same partition.

[0007] A method for solving out-of-order messages in real-time data collection provided by the present invention, after obtaining the change data capture message generated by the producer, further includes: Use a Bloom filter to perform preliminary screening on the change data capture message to form a preliminarily screened message; Based on cache information, perform secondary screening on the preliminarily screened message to remove duplicate data and form the screened change data capture message; Wherein, the cache information includes the identification information for historically obtaining the change data capture message.

[0008] A method for solving out-of-order messages in real-time data collection provided by the present invention, determining the target partition based on the logical planning strategy according to the identification information includes: Determine the message topic corresponding to the change data capture message according to the identification information and the preset logic; Determine the partition field according to the identification information; Based on the partition field, calculate using a preset partitioning algorithm to determine the target partition from the message partitions under the message topic.

[0009] A method for solving out-of-order messages in real-time data collection provided by the present invention, the message topic corresponds to at least one database, the database includes at least one data table, and the number of message partitions under the message topic is determined by the data throughput of the producer and consumer corresponding to the database.

[0010] A method for solving out-of-order messages in real-time data collection provided by the present invention, further includes: Perform verification processing on the change data capture message in the message partition to obtain out-of-order detection information; If it is determined that there is an out-of-order operation according to the out-of-order detection information, determine an out-of-order message group, and the out-of-order message group includes at least two change data capture messages; Modify the out-of-order message group based on a preset time sequence correction strategy to correct the out-of-order operation.

[0011] A method for solving out-of-order messages in real-time data collection provided by the present invention, after distributing the change data capture message to the target partition, further includes: If it is determined that the change data capture message is an update operation message and involves a primary key change, generate a corresponding delete operation message and insert operation message according to the update operation message; Wherein, the delete operation message and the insert operation message are completed in the same transaction.

[0012] A method for solving message out-of-order in real-time data collection provided by the present invention further includes: Obtain the survival time information of the deleted status data in the data table, where the deleted status data is formed after the change data capture message for the delete operation message is consumed, and the survival time information represents the duration in the deleted status; Determine that the survival time information reaches a preset first time threshold, and delete the corresponding deleted status data, or delete the deleted status data whose survival time information is greater than a preset second time threshold every preset period.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the method for solving message out-of-order in real-time data collection as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for solving message out-of-order in real-time data collection as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for solving message out-of-order in real-time data collection as described in any one of the above.

[0016] A method, device, and storage medium for solving message out-of-order in real-time data collection provided by the present invention at least have the beneficial effects: By distributing the change data capture message to the corresponding target partition according to the identification information of the change data capture message, the target partition arranges the change data capture messages based on time sequence and consumes them. Since the change data capture messages corresponding to the same data table are located in the same message partition, the time sequence of operations on the same data table can be guaranteed. In this way, in a concurrent execution scenario, the change data capture messages in multiple message partitions are consumed and executed in time sequence, which can not only avoid the out-of-order problem but also not affect the efficiency of concurrent execution. At the same time, the local sorting in the message partition replaces the global sorting. By the logical planning strategy, the change data capture messages that need to be processed in time sequence are restricted within the same message partition. Only the local order within the message partition needs to be maintained, and there is no need for global management or central global serial number generation to ensure the time sequence, which is beneficial to reducing the calculation overhead. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the attached drawings required in the description of the embodiments or the prior art. Obviously, the attached drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other attached drawings can also be obtained based on these attached drawings.

[0018] Figure 1 It is one of the schematic flowcharts of a method for solving message out-of-order in real-time data acquisition provided by the present invention.

[0019] Figure 2 It is the second of the schematic flowcharts of a method for solving message out-of-order in real-time data acquisition provided by the present invention.

[0020] Figure 3 It is the third of the schematic flowcharts of a method for solving message out-of-order in real-time data acquisition provided by the present invention.

[0021] Figure 4 It is the fourth of the schematic flowcharts of a method for solving message out-of-order in real-time data acquisition provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the corresponding relationship between the message topic and message partition and the database and data tables in one of the embodiments of a method for solving message out-of-order in real-time data acquisition provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed implementation manners

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the attached drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0025] The following combines Figure 1 Describe a method for solving message out-of-order in real-time data acquisition of the present invention, including: S100: Obtain the change data capture message generated by the producer; S200: Determine the identification information according to the change data capture message; S300: Determine the target partition based on the logical planning strategy according to the identification information; S400: Distribute the change data capture message to the target partition; Among them, the change data capture messages in the target partition are arranged in chronological order and consumed in chronological order, and the change data capture messages corresponding to the same data table are located in the same partition.

[0026] By distributing the change data capture messages to the corresponding target partitions according to the identification information of the change data capture messages, and the target partitions arrange and consume the change data capture messages based on chronological order. Since the change data capture messages corresponding to the same data table are located in the same message partition, the chronological order of operations on the same data table can be guaranteed. In this way, in the scenario of concurrent execution, the change data capture messages in multiple message partitions are consumed in chronological order, which can not only avoid out-of-order problems, but also not affect the efficiency of concurrent execution. At the same time, the local sorting in the message partition replaces the global sorting. By means of a logical planning strategy, the change data capture messages that need to be processed in chronological order are restricted within the same message partition. Only the local order within the message partition needs to be maintained, and there is no need for global management or a central global sequence number generation to ensure chronological order, which is beneficial to reducing the computing overhead.

[0027] The change data capture messages include identifying information such as the producer ID, business ID, location information, etc. One or more of them can be selected as the identification information so that the CDC information corresponding to the same data table can be distributed to the same message partition. In some embodiments of the present invention, the logical planning strategy can be to determine the target partition according to the location where the producer corresponding to the change data capture message is located, such as a city. In some embodiments, it can also be to perform algorithm calculations such as hashing on the identification information, and partition according to the calculation results. Based on the same calculation results of the identification information, the CDC information corresponding to the same data table will be distributed to the same message partition.

[0028] Reference Figure 2 , in some embodiments of a method for solving out-of-order messages in real-time data collection according to the present invention, after the S100, it further includes: S110: Use a Bloom filter to perform preliminary screening processing on the change data capture messages to form preliminary screening messages; S120: Based on the cache information, perform secondary screening processing on the preliminary screening messages to remove duplicate data and form the screened change data capture messages; Among them, the cache information includes the identification information of the historical acquisition of the change data capture messages.

[0029] Due to the influence of delays, retransmissions, etc. in the actual network transmission process, there are cases where duplicate messages are received. If the same change data capture messages are repeatedly executed, it will lead to problems such as repeated processing, resource waste, and reduced reliability.

[0030] In this regard, the change data capture messages are initially screened using a Bloom filter, which can quickly filter out most duplicate messages with low performance consumption and low time complexity, reducing the amount of data to be processed subsequently. Since the Bloom filter cannot guarantee the complete removal of all duplicate messages, based on cache information such as HBase, historical identification information is stored for exact matching. For the change data capture messages with stored identification information, that is, the already obtained change data capture messages, they are removed to completely remove duplicate change data capture messages and ensure the uniqueness of each change data capture message.

[0031] In this way, the pre-screening of the Bloom filter avoids the performance overhead of comparing all data. Based on the cache information, only a small number of suspected duplicate messages after preliminary screening need to be queried. By combining the Bloom filter with the secondary screening based on cache information, the consumption of storage and computing resources is minimized, achieving the effect of removing duplicate change data capture messages, which is beneficial to avoiding duplicate processing and improving processing efficiency.

[0032] The Bloom filter is used to quickly determine whether an element may exist in a set. The principle is to create a bit array of length m, with all bits initially set to 0; use k independent hash functions to map the element to k different positions in the bit array and set these positions to 1; check whether the k bits corresponding to the element are all 1; if all are 1, it is considered that the element may exist (there may be false positives); if any position is 0, the element must not exist (there will be no false negatives). Accordingly, the change data capture messages are initially screened through the Bloom filter to determine the change data capture messages that may exist, which can quickly screen the messages and reduce the amount of data to be processed in the subsequent secondary screening, facilitating the improvement of processing efficiency. For the change data capture messages that may exist, secondary screening processing is implemented by comparing the identification information based on the cache information to accurately remove the duplicate change data capture messages obtained.

[0033] Reference Figure 5 In some embodiments of a method for solving message out-of-order in real-time data acquisition according to the present invention, the S300 includes: Determine the message topic corresponding to the change data capture message according to the identification information and a preset logic; Determine the partition field according to the identification information; Based on the partition field, calculate using a preset partitioning algorithm to determine a target partition from the message partitions under the message topic.

[0034] According to the identification information of the change data capture message, first divide the message topics based on a preset logic to determine the message topic corresponding to the change data capture message, and then determine the target partition from the message partitions under the message topic according to the partition field in the identification information. In this way, based on the identification information, such as user ID, business ID, location, etc., and the preset logic, such as dividing by business type, data table, producer location, etc., the change data capture message is automatically classified into the corresponding target message topic, enabling data isolation in different business scenarios, facilitating management independence, and then calculating based on the partition field in the identification information, such as the primary key value or user ID, etc., according to a preset partitioning algorithm, to ensure that the change data capture messages of the same data table always enter the same message partition.

[0035] In some embodiments of the present invention, the preset logic may be to divide according to location, such as the city where the producer is located. In some embodiments, the identification information may only include the partition field, in which case the identification information is the same as the partition field. In some embodiments, in addition to the partition field, the identification information may also include other information. It can be understood that the logical planning strategy includes the preset logic and the preset partitioning algorithm.

[0036] In some embodiments of the present invention, the message topic and message partition management of the change data capture message can be implemented through a distributed message system, such as a kafka cluster, etc. Figure 5 As shown, it is further realized that the change data capture messages corresponding to the same data table are located in the same partition and are sorted and consumed in chronological order. In some embodiments of the present invention, the producer directly sends the change data capture message to the broker of the kafka cluster through end-to-end communication (such as establishing a socket connection), without passing through the routing layer, and the message transmission does not depend on an intermediate proxy or routing service, so as to ensure that the broker of the kafka cluster receives the corresponding change data capture messages in the order generated by the producer, that is, to ensure the chronology of obtaining the change data capture messages. Since in the message partition of the kafka cluster, the change data capture messages are strictly arranged in chronological order and are strictly provided to the consumer in chronological order, it is beneficial to ensure the chronology of the consumer obtaining the change data capture messages.

[0037] In some embodiments of the present invention, the preset partitioning algorithm satisfies that the probability of being assigned to each message partition is basically the same. In some embodiments, the preset partitioning algorithm may be a hash algorithm. The hash algorithm has the same calculation result for the same partition field. By grouping through the hash calculation method, the change data capture messages are distributed into the message partitions, which is beneficial to make the data distribution under the message topic more balanced, that is, the load of each message partition is more balanced, avoiding data hotspots, and is beneficial to improving the concurrency performance.

[0038] In some embodiments of the present invention, the preset partitioning algorithm is the hash algorithm, and the following requirements should be met: (1) If there is a change in the combined primary key value during the operation of the service, the primary key value field that will change cannot be used as the partitioning field to avoid participating in the partitioning hash field calculation. Otherwise, the change data capture messages before and after the change of the primary key value may be sent to different message partitions, resulting in the inability to guarantee the sequentiality. Do not use the primary key value field that will change as the partitioning field to ensure the stability of the change data capture message partition.

[0039] (2) If the service has only a single primary key value and there is a change in the primary key value, the corresponding data table can only be placed in a single message partition and no hash calculation is performed.

[0040] (3) Data tables with a small business volume (less than 10w / d) can be placed in a single separate message partition or share a message partition with other data tables with the same small business volume to improve resource utilization and management efficiency.

[0041] The primary key is the field in the data table that uniquely identifies a record, and the primary key value is the specific value of this field. The identification information contains the primary key.

[0042] In some embodiments of a method for solving message out-of-order in real-time data collection of the present invention, the message topic corresponds to at least one database, the database includes at least one data table, and the number of message partitions under the message topic is determined by the data throughput of the producer and consumer corresponding to the database.

[0043] The message topic corresponds to at least one database, and there can be multiple data tables in the database. It should be noted that the number of message partitions under the message topic is at least one, and multiple message partitions can be set according to the data throughput of the message topic. The number of data tables corresponding to the message partition is at least one. Generally, it is planned that one data table corresponds to one message partition. For data tables with a small transaction processing volume, multiple data tables can correspond to one message partition, but one data table can only correspond to one message partition to ensure that the change data capture messages corresponding to the data table are executed strictly in chronological order.

[0044] In some embodiments of the present invention, the number of message partitions can be determined based on the following expression: the number of message partitions = Tt / min(Tp, Tc) Wherein, Tt is the load capacity of the server corresponding to the message topic; Tp is the producer data throughput; Tc is the consumer data throughput; min() is the function of taking the minimum value.

[0045] Reference Figure 3 , in some embodiments of a method for solving message out-of-order in real-time data collection of the present invention, it further includes: S500: Perform a verification process on the change data capture messages in the message partition to obtain out-of-order detection information; S510: If it is determined that there is an out-of-order operation based on the out-of-order detection information, determine an out-of-order message group, where the out-of-order message group includes at least two of the change data capture messages; S520: Modify the out-of-order message group based on a preset timing correction strategy to correct the out-of-order operation.

[0046] During the real-time data collection and conversion process, due to the influence of abnormalities in the data collection end, i.e., the producer end, or changes in business-related conditions, etc., the change data capture messages generated from the source are in an out-of-order state, resulting in the change data capture messages received in sequence also being out of order. In response to this situation, through a verification process, it is detected whether there is an out-of-order operation, that is, whether the timing of the change data capture messages is out of order. Furthermore, when there is an out-of-order operation, it is corrected based on a preset timing correction strategy, so that the modified change data capture messages are accurately executed according to the logical timing, ensuring the accuracy of data processing based on the change data capture messages.

[0047] In some embodiments of the present invention, modifying the out-of-order message group may be to adjust the order of the change data capture messages or delete the change data capture messages to achieve the purpose of correcting the out-of-order operation.

[0048] It should be noted that when the collection end is operating normally and the business-related conditions remain unchanged, that is, there is no out-of-order problem when the change data capture messages are generated, by distributing the change data capture messages to the corresponding target partitions and executing them in sequence, the out-of-order problem in the concurrent execution scenario can be avoided. The problem of out-of-order change data capture messages caused by abnormalities in the collection end or business-related conditions is not caused by the concurrent scenario, but exists when the source change data capture messages are generated, that is, the causes of the two out-of-order problems are different. Therefore, the out-of-order messages caused by abnormalities in the collection end or changes in business-related conditions are further problems to be solved.

[0049] In some embodiments of a method for solving out-of-order messages in real-time data collection of the present invention, S500 includes: Based on the change data capture messages and timestamps, obtain the out-of-order detection information.

[0050] In some embodiments of the present invention, the verification process for the change data capture messages in the message partition may be to compare according to the timestamps of the change data capture messages, and verify whether the sorting sequence is consistent with the increasing order of timestamps. When the two are inconsistent, it means that there are out-of-order change data capture messages. Furthermore, based on the change data capture messages corresponding to the abnormal increasing order of timestamps, an out-of-order message group is determined. In some embodiments of the present invention, since the change data capture messages in the same message partition may correspond to different data tables, even if the timestamps of the change data capture messages corresponding to different data tables are abnormal, it may not lead to out-of-order operations. Therefore, by combining the timestamp with a preset rule for verification, first identify whether there are change data capture messages with abnormal timestamps according to the timestamp. When there are change data capture messages with abnormal timestamps, based on the preset rule, such as screening the change data capture messages corresponding to the same data table based on the identification information, and then determine whether there is an out-of-order operation according to the timestamp.

[0051] In some embodiments of the present invention, the S500 includes: Based on the change data capture message, obtain the out-of-order detection information according to the timestamp and the message operation sequence.

[0052] When verifying the change data capture message, on the basis of the timestamp, the out-of-order situation can also be determined in combination with the message operation sequence, and then the out-of-order detection information is obtained, so that the out-of-order detection information includes the out-of-order situation, which is convenient for subsequent determination of the out-of-order message group.

[0053] In some embodiments of the present invention, the out-of-order situation and the corresponding preset timing correction strategy can be as shown in the following table:

[0054] It can be understood that the UPDATE, INSERT, and DELETE operations in the table correspond to one change data capture message.

[0055] In some embodiments of a method for solving out-of-order messages in real-time data acquisition of the present invention, after the S400, it further includes: If it is determined that the change data capture message is an update operation message and involves a primary key change, then according to the update operation message, generate a corresponding delete operation message and an insert operation message; Among them, the delete operation message and the insert operation message are completed in the same transaction.

[0056] The primary key is the field in a data table that uniquely identifies a record. The identification information contains the primary key. Therefore, the primary key value is one of the most important fields of the data. When the business needs to change the primary key, especially in the scenario of a distributed messaging system, it is necessary to ensure the consistency of the primary key change. For the UPDATE operation corresponding to the update operation message, there will be an intermediate state when updating and modifying the primary key, that is, the state where the primary key values of some data are modified while the primary key values of some data are not modified. Furthermore, the intermediate state may cause the risk of processing errors.

[0057] Therefore, when it is determined that the change data capture message is an update operation message involving primary key change, the update operation message is converted into a corresponding delete operation message and insert operation message, that is, the UPDATE operation is converted into a DELETE operation and an INSERT operation, and both are completed in the same transaction to ensure that there is no intermediate state where some data is modified while some data is not modified during the primary key change, which is beneficial to ensuring data consistency, avoiding the problem of message disorder caused by primary key change, and improving the reliability of the primary key change process.

[0058] Reference Figure 4 , in some embodiments of a method for solving message disorder in real-time data acquisition according to the present invention, it further includes: S600: Obtain the survival time information of the deleted state data in the data table. The deleted state data is formed after the change data capture message that is a delete operation message is consumed. The survival time information represents the duration in the deleted state; S610: Determine that the survival time information reaches a preset first time threshold, and delete the corresponding deleted state data, or delete the deleted state data whose survival time information is greater than a preset second time threshold every preset period.

[0059] For purposes such as data traceability, after the change data capture message that is a delete operation message is consumed, that is, after the DELETE operation is executed, the corresponding data in the data table will not be physically deleted, but will be marked as the deleted state to form deleted state data. The deleted state data can provide traceability information in case of accidents, failures, etc. However, retaining a large amount of deleted state data for a long time will occupy too much storage space resources. Therefore, the deleted state data is correspondingly set with survival time information to represent the duration after the deleted state data is subjected to the DELETE operation. When the survival time information reaches a preset first time threshold, the corresponding deleted state data is physically deleted to release storage space resources; or, every preset period, the deleted state data whose survival time information is greater than a preset second time threshold is uniformly physically deleted to facilitate centralized cleaning, improve the efficiency of physical deletion, and avoid the normal operation of physical deletion processing from occupying operating resources.

[0060] The following describes a message disorder resolution device for real-time data acquisition provided by the present invention. The message disorder resolution device for real-time data acquisition described below can be correspondingly referred to the message disorder resolution method for real-time data acquisition described above.

[0061] The present invention also provides a message disorder resolution device for real-time data acquisition, including: A message acquisition module, configured to acquire change data capture messages generated by a producer; A deduplication module, configured to perform a preliminary screening process on the change data capture messages using a Bloom filter to form preliminarily screened messages, and perform a secondary screening process on the preliminarily screened messages based on cache information to remove duplicate data; An identification determination module, configured to determine identification information according to the change data capture messages; A partitioning module, configured to determine a target partition according to the identification information based on a logical planning strategy; A distribution module, configured to distribute the change data capture messages to the target partition; A disorder correction module, configured to perform a verification process on the change data capture messages in the message partition to obtain disorder detection information. If it is determined that there is a disorder operation according to the disorder detection information, a disorder message group is determined. The disorder message group includes at least two change data capture messages, and based on a preset time sequence correction strategy, the disorder message group is modified to correct the disorder operation; A data cleaning module, configured to obtain the survival time information of the deleted state data in a data table. The deleted state data is formed after the change data capture messages for deletion operation messages are consumed. The survival time information represents the duration in the deleted state. If it is determined that the survival time information reaches a preset first time threshold, the corresponding deleted state data is deleted, or the deleted state data with the survival time information greater than a preset second time threshold is deleted every preset period.

[0062] Figure 6 An example of the physical structure diagram of an electronic device is as Figure 6 shown. The electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 complete mutual communication through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the above-mentioned message disorder resolution method for real-time data acquisition.

[0063] In addition, when the logical instructions in the above-mentioned memory 830 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0064] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for solving message out-of-order in real-time data collection provided by the above-mentioned various methods.

[0065] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute a method for solving message out-of-order in real-time data collection provided by the above-mentioned various methods.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0067] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0068] In the description of the present invention, it should be understood that the terms "first" and "second" are only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0069] In the description of this specification, descriptions with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for solving out-of-order messages in real-time data acquisition, characterized in that, Including: Obtaining change data capture messages generated by a producer; Determining identification information according to the change data capture messages; Determining a target partition according to the identification information based on a logical planning strategy; Distributing the change data capture messages to the target partition; Wherein, the change data capture messages in the target partition are arranged in time sequence and consumed in time sequence, and the change data capture messages corresponding to the same data table are located in the same partition.

2. The method for solving out-of-order messages in real-time data acquisition according to claim 1, characterized in that After obtaining the change data capture messages generated by the producer, it further includes: Performing a preliminary screening process on the change data capture messages using a Bloom filter to form preliminary screening messages; Performing a secondary screening process on the preliminary screening messages based on cache information to remove duplicate data and form the screened change data capture messages; Wherein, the cache information includes the identification information of the historical obtained change data capture messages.

3. A method for solving out-of-order messages in real-time data acquisition according to claim 1, characterized in that, The determining the target partition according to the identification information based on a logical planning strategy includes: Determining a message topic corresponding to the change data capture message according to the identification information and a preset logic; Determining a partition field according to the identification information; Determining a target partition from the message partitions under the message topic based on a preset partitioning algorithm according to the partition field.

4. A method for solving out-of-order messages in real-time data acquisition according to claim 3, characterized in that, The message topic corresponds to at least one database, the database includes at least one data table, and the number of message partitions under the message topic is determined by the data throughput of the corresponding producer and consumer of the database.

5. The method for solving the out-of-order problem of messages in real-time data collection according to claim 3, wherein It further includes: Performing a verification process on the change data capture messages in the message partition to obtain out-of-order detection information; If it is determined that there is an out-of-order operation according to the out-of-order detection information, determining an out-of-order message group, and the out-of-order message group includes at least two change data capture messages; Modifying the out-of-order message group based on a preset time sequence correction strategy to correct the out-of-order operation.

6. The method for solving out-of-order messages in real-time data collection according to claim 1, characterized in that, After distributing the change data capture messages to the target partition, it further includes: If it is determined that the change data capture message is an update operation message and involves a primary key change, generating a corresponding delete operation message and insert operation message according to the update operation message; Wherein, the delete operation message and the insert operation message are completed in the same transaction.

7. A method for solving the out-of-order problem of messages in real-time data acquisition according to claim 1, characterized in that, It further includes: Obtaining the survival time information of the deletion status data in the data table, where the deletion status data is formed after the change data capture message that is a delete operation message is consumed, and the survival time information represents the duration in the deletion status; If it is determined that the survival time information reaches a preset first time threshold, deleting the corresponding deletion status data, or deleting the deletion status data whose survival time information is greater than a preset second time threshold every preset period.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for solving message out-of-order in real-time data acquisition according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a method for solving message out-of-order in real-time data acquisition according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements a method for solving out-of-order messages in real-time data acquisition as described in any one of claims 1 to 7.

Images