Data difference real-time detection method and device based on streaming job migration scene

Abstract

The invention relates to a data difference real-time detection method and device based on a streaming job migration scene. The method comprises the following steps: after a streaming job program is migrated from an original cluster to a target cluster, running streaming jobs in parallel in the original cluster and the target cluster; during the parallel running period, obtaining a first processing result and a second processing result after the original cluster and the target cluster respectively process data in a target theme corresponding to the streaming job, and issuing the first processing result to a downstream system; and performing matching verification on the first processing result and the second processing result, stopping operation of the streaming job in the original cluster under the condition that multiple verification results in a preset duration pass verification, and consuming real-time data in the target theme from the event platform in the target cluster. And issuing a third processing result obtained by processing the real-time data to a downstream system. By adopting the method, the streaming operation can be stably migrated, and the accuracy of the streaming data can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of stream data processing, in particular to a data difference real-time detection method and device based on a stream job migration scenario. BACKGROUND

[0002] With the development of big data technology, stream computing engines have become the core infrastructure for processing real-time data. In the process of business evolution, stream jobs need to be migrated due to reasons such as function iteration, performance optimization, or framework upgrade. For companies with less data and simple business scenarios, the "hard switching" method of stopping the original job and starting a new job is often used, and the data comparison scheme adopts single offline batch comparison or simple double-stream real-time comparison. However, the financial industry and large Internet companies have complex internal scenarios and high business continuity requirements, so they often need safer and smoother job migration solutions and real-time comparison solutions that can handle data out-of-order and delayed arrival to meet the scenario requirements of lossless, accurate real-time comparison, final consistency, and traceability of business data. How to smoothly migrate stream jobs and ensure the accuracy of stream data is a problem that needs to be solved. SUMMARY

[0003] Therefore, it is necessary to provide a data difference real-time detection method and device based on a stream job migration scenario, computer equipment, computer readable storage medium, and computer program product to smoothly migrate stream jobs and ensure the accuracy of stream data.

[0004] In a first aspect, the present application provides a data difference real-time detection method based on a stream job migration scenario, comprising:

[0005] After the program of the stream job is migrated from the original cluster to the target cluster, the stream job is run in parallel in the original cluster and the target cluster;

[0006] During parallel running, the first processing result and the second processing result obtained by the original cluster and the target cluster respectively processing the data in the target topic corresponding to the stream job are obtained, and the first processing result is sent to the downstream system;

[0007] The first processing result and the second processing result are matched and verified, and if a plurality of verification results within a preset time length are all verified, the running of the stream job in the original cluster is stopped, real-time data in the target topic is consumed from the event platform in the target cluster, and the third processing result obtained by processing the real-time data is sent to the downstream system.

[0008] In one of the embodiments, obtaining the first processing result and the second processing result obtained by the original cluster and the target cluster respectively processing the data in the target topic corresponding to the stream job comprises:

[0009] consuming the target topic corresponding to the stream job from the event platform by a second user in the target cluster;

[0010] obtaining a first processing result of processing data in the target topic by the original cluster, and obtaining a second processing result of processing data in the target topic by the target cluster.

[0011] In one of the embodiments, the data in the target topic comprises a plurality of key-value pairs; and the first processing result and the second processing result of processing the data in the target topic corresponding to the stream job by the original cluster and the target cluster respectively comprises:

[0012] In the original cluster, the keys of the plurality of key-value pairs are spliced, and the values of the plurality of key-value pairs are spliced to obtain the first processing result;

[0013] In the target cluster, the keys of the plurality of key-value pairs are spliced, and the values of the plurality of key-value pairs are spliced to obtain the second processing result.

[0014] In one of the embodiments, the matching and checking of the first processing result and the second processing result comprises:

[0015] In the case that the key in the second processing result matches the key in the first processing result, the value in the second processing result is matched with the value in the first processing result field by field;

[0016] In the case that all the fields are matched, it is determined that the checking result is checked through.

[0017] In one of the embodiments, the real-time data in the target topic is consumed from the event platform in the target cluster, comprising:

[0018] In the target cluster, the real-time data in the target topic is consumed from the event platform by a first user.

[0019] In one of the embodiments, the method further comprises:

[0020] The first processing result and the second processing result are written into a preset database respectively;

[0021] The batch checking script is executed every interval of a preset time period, and the first processing result and the second processing result stored in the preset database are batch checked.

[0022] In a second aspect, the application further provides a data difference real-time detection device based on a stream job migration scene, comprising:

[0023] migrate the program of the stream job from the original cluster to the target cluster, and run the stream job in parallel in the original cluster and the target cluster;

[0024] obtain a first processing result and a second processing result of processing data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively during the parallel running, and deliver the first processing result to a downstream system;

[0025] perform matching verification on the first processing result and the second processing result, stop running of the stream job in the original cluster in a case that multiple verification results within a preset time length are all verified to be passed, consume real-time data in the target topic from an event platform in the target cluster, deliver a third processing result of processing the real-time data to the downstream system.

[0026] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0027] migrate the program of the stream job from the original cluster to the target cluster, and run the stream job in parallel in the original cluster and the target cluster;

[0028] obtain a first processing result and a second processing result of processing data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively during the parallel running, and deliver the first processing result to a downstream system;

[0029] perform matching verification on the first processing result and the second processing result, stop running of the stream job in the original cluster in a case that multiple verification results within a preset time length are all verified to be passed, consume real-time data in the target topic from an event platform in the target cluster, deliver a third processing result of processing the real-time data to the downstream system.

[0030] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0031] migrate the program of the stream job from the original cluster to the target cluster, and run the stream job in parallel in the original cluster and the target cluster;

[0032] obtain a first processing result and a second processing result of processing data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively during the parallel running, and deliver the first processing result to a downstream system;

[0033] The first processing result and the second processing result are matched and verified, in a case where multiple verification results within a preset time length are all verified, the running of the stream job in the original cluster is stopped, real-time data in the target topic is consumed from the event platform in the target cluster, and a third processing result obtained by processing the real-time data is sent to a downstream system.

[0034] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0035] After the program of the stream job is migrated from the original cluster to the target cluster, the stream job is run in parallel in the original cluster and the target cluster;

[0036] During the parallel running, a first processing result and a second processing result obtained by processing data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively are acquired, and the first processing result is sent to a downstream system;

[0037] The first processing result and the second processing result are matched and verified, in a case where multiple verification results within a preset time length are all verified, the running of the stream job in the original cluster is stopped, real-time data in the target topic is consumed from the event platform in the target cluster, and a third processing result obtained by processing the real-time data is sent to a downstream system.

[0038] The above data difference real-time detection method, device, computer equipment, computer readable storage medium and computer program product based on stream job migration scenario, by migrating the program of the stream job from the original cluster to the target cluster, the original cluster and the new cluster job consume data based on the same target topic, realizing data input homology; the original cluster and the target cluster run the stream job in parallel, during the parallel running, a first processing result and a second processing result obtained by processing data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively are acquired, and the first processing result is sent to a downstream system, so that the original cluster job continues to run during the migration process and interfaces with the downstream system, and the new cluster job processes data in parallel without interfacing with the downstream system; during the parallel running, the target cluster matches and verifies the first processing result and the second processing result, in a case where multiple verification results within a preset time length are all verified, it is indicated that the target cluster can output consistent correct results with the original cluster, the stream job migration is accurate; then entering the cluster switching stage, the running of the stream job in the original cluster is stopped, real-time data in the target topic is consumed from the event platform in the target cluster, and a third processing result obtained by processing the real-time data is sent to a downstream system, ensuring the continuity of business logic, smooth migration of stream job, and accuracy of stream data. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained from these drawings without creative effort.

[0040] Figure 1 An application environment diagram of a data difference real-time detection method based on a streaming job migration scenario in an embodiment;

[0041] Figure 2 A flowchart of a data difference real-time detection method based on a streaming job migration scenario in an embodiment;

[0042] Figure 3 A general flowchart of a data difference real-time detection method based on a streaming job migration scenario in an embodiment;

[0043] Figure 4 A schematic diagram of data comparison in an embodiment;

[0044] Figure 5 A flowchart of field-by-field matching in an embodiment;

[0045] Figure 6 A structural block diagram of a data difference real-time detection apparatus based on a streaming job migration scenario in an embodiment;

[0046] Figure 7 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0048] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the solutions, or any combination of multiple solutions.

[0049] The data difference real-time detection method based on a streaming job migration scenario provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The embodiments take the terminal as an example to illustrate the method, and it can be understood that the method can also be applied to the server, and can also be applied to a system including the terminal and the server, and realized through the interaction of the terminal and the server. The terminal 102 runs the streaming job in the original cluster and the target cluster in parallel after the program of the streaming job is migrated from the original cluster to the target cluster; during the parallel running, the first processing result and the second processing result obtained by processing data in a target topic corresponding to the streaming job in the original cluster and the target cluster respectively are obtained, and the first processing result is issued to a downstream system; the first processing result and the second processing result are matched and verified, and in the case that a plurality of verification results within a preset time length are all verified, the running of the streaming job in the original cluster is stopped, and the real-time data in the target topic is consumed from an event platform in the target cluster, and the third processing result obtained by processing the real-time data is issued to the downstream system. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude aircraft, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0050] In an exemplary embodiment, as shown in Figure 2 A data difference real-time detection method based on a streaming job migration scenario is provided. The method is applied to the terminal 102 in Figure 1 for example, and includes the following steps 202 to 206. Wherein:

[0051] Step 202, after the program of the streaming job is migrated from the original cluster to the target cluster, the streaming job is run in the original cluster and the target cluster in parallel.

[0052] The stream job refers to a task program running on a stream computing engine (such as Flink, Spark Streaming) and used for real-time processing of continuously generated, unbounded data streams. Exemplarily, the stream job can be a real-time risk control task, a query task or the like commonly used in the financial field.

[0053] The original cluster refers to a stream computing cluster where the stream job is located before migration. The original cluster can be composed of multiple server nodes and provide computing resources (CPU, memory), storage resources and cluster management capabilities. During the migration process, the original cluster is the core of business continuity assurance and needs to be continuously and stably operated before the target cluster verification is completed.

[0054] The target cluster refers to the final carrying cluster of the stream job after migration. Due to the aging of the hardware of the original cluster, performance bottleneck and other problems, the stream job needs to be migrated to the target cluster. The purpose of the stream job migration is to completely replace the business functions of the original cluster by the target cluster and ensure that each stream job can be stably operated.

[0055] The stream job migration mainly migrates the program of the stream job from the original cluster to the target cluster. After the program of the stream job is migrated, the stream job is run in parallel in the original cluster and the target cluster to verify whether the target cluster can be stably operated.

[0056] In step 204, during the parallel running, a first processing result and a second processing result obtained by the original cluster and the target cluster respectively processing data in a target topic corresponding to the stream job are obtained, and the first processing result is sent to a downstream system.

[0057] The target topic is a specific logical channel in an event platform Kafka used for storing raw data to be processed of the stream job. The original cluster and the target cluster both consume the target topic from the event platform Kafka to ensure consistency of data sources.

[0058] The original cluster and the target cluster process the obtained data in the target topic respectively. The original cluster obtains the first processing result, and the target cluster obtains the second processing result. The processing is a processing method for supporting the business operation requirements of the downstream system. The processing methods in the original cluster and the target cluster are consistent. Due to the environment difference between the original cluster and the target cluster, the first processing result and the second processing result after processing can be different.

[0059] During the parallel running, the first processing result generated by the original cluster is sent to the downstream system to ensure business continuity.

[0060] At step 206, the first processing result and the second processing result are matched and verified. If multiple verification results within a preset time period are all verified, the running of the stream job in the original cluster is stopped, real-time data in the target topic from the event platform is consumed in the target cluster, and a third processing result obtained by processing the real-time data is delivered to the downstream system.

[0061] The matching verification refers to an operation of associating and checking consistency between the first processing result output by the original cluster and the second processing result output by the target cluster by using specific rules and tools.

[0062] The preset time period is a time period for matching verification. In the time period, multiple sets of the first processing result and the second processing result can be matched and verified, and each set obtains a verification result. The verification passing condition can be at least one of no field difference, no data loss, no timeout unmatched, and an abnormality rate lower than a preset threshold.

[0063] If multiple verification results within the preset time period are all verified, it indicates that the target cluster job has the replacement ability of logical correctness and stable running, and then enters the cluster switching stage. The verification result within the preset time period is used to ensure the continuous and stable running of the target cluster and avoid misjudgment caused by normal short-term data.

[0064] In the cluster switching stage, the running of the stream job in the original cluster is stopped, and the stream job is run in the target cluster. Specifically, the latest real-time data of the target topic from the event platform is consumed, the real-time data is processed to obtain a third processing result, which is "official production data" for replacing the first processing result to support the running of the downstream system. The target cluster delivers the third processing result to the downstream system, so that the business in the switching stage does not stop running, and the downstream system is not aware of the switching.

[0065] In the data difference real-time detection method based on the stream job migration scenario, after the program of the stream job is migrated from the original cluster to the target cluster, the original cluster and the new cluster consume data based on the same target topic, realizing data input homology; the original cluster and the target cluster run the stream job in parallel, during the parallel running, the first processing result and the second processing result of the data in the target topic corresponding to the stream job processed by the original cluster and the target cluster respectively are acquired, and the first processing result is issued to the downstream system, so that the original cluster job continues to run during the migration process and connects the downstream system, and the new cluster job processes data in parallel without connecting the downstream system; during the parallel running, the target cluster matches and checks the first processing result and the second processing result after processing, and in the case that multiple checking results within a preset time length are all checked, it is indicated that the target cluster can output correct results consistent with the original cluster, the stream job migration is accurate; and then the cluster switching stage is entered, the running of the stream job in the original cluster is stopped, and real-time data in the target topic is consumed from the event platform in the target cluster, and the third processing result obtained by processing the real-time data is issued to the downstream system, thereby ensuring the continuity of the business logic, smooth migration of the stream job, and accuracy of the stream data.

[0066] In an exemplary embodiment, acquiring the first processing result and the second processing result of the data in the target topic corresponding to the stream job processed by the original cluster and the target cluster respectively comprises:

[0067] The target topic corresponding to the stream job is consumed from the event platform by a first user in the original cluster, and the target topic corresponding to the stream job is consumed from the event platform by a second user in the target cluster;

[0068] The first processing result of the data in the target topic processed by the original cluster is acquired, and the second processing result of the data in the target topic processed by the target cluster is acquired.

[0069] The first user refers to a computing entity in the original cluster that has the right to consume data from the event platform, and the essence is a consumption account / service identity corresponding to the original cluster job. The core role is to subscribe to the target topic through a legal identity, to ensure that the original cluster can safely and stably acquire original data, and to be a permission carrier of a data consumption link. The second user refers to a consumption account / service identity of the target cluster job in the event platform, which has independent permission to subscribe to the target topic. The permissions of the second user and the first user are isolated, the original cluster and the target cluster consume the target topic by using independent first users and second users respectively, to ensure that the consumption operation of the target cluster does not affect the data acquisition of the original cluster, and to ensure the data source consistency of the two.

[0070] The original cluster processes data in the target topic to obtain a first processing result, and the target cluster processes data in the target topic to obtain a second processing result.

[0071] As shown in Figure 3 Fig. 1 shows a general flowchart of a data difference real-time detection method based on a streaming job migration scenario in some embodiments. The original cluster consumes a target topic SourceTopic corresponding to a streaming job from an event platform through a first user, and obtains a first processing result, i.e., an original Sink Topic, after processing ETL. The target cluster consumes the target topic SourceTopic corresponding to the streaming job from the event platform through a second user, and obtains a second processing result, i.e., a new Sink Topic, after processing ETL.

[0072] In this embodiment, the first user and the second user are independent identities in the event platform and have independent consumer groups and consumption progress. The subscription operation of the target cluster does not affect the consumption of the target topic by the original cluster, and neither causes data missing consumption or repeated consumption of the original cluster nor interferes with the original business link due to job debugging or abnormal stop of the target cluster. This isolation guarantees the continuity of the original cluster business from the data source, which is the core prerequisite for migration continuity. The original cluster and the target cluster both subscribe to the same target topic of the event platform, which guarantees data source consistency.

[0073] In one exemplary embodiment, the data in the target topic includes a plurality of key-value pairs; obtaining a first processing result and a second processing result after the original cluster and the target cluster respectively process the data in the target topic corresponding to the streaming job includes: in the original cluster, splicing keys of the plurality of key-value pairs and splicing values of the plurality of key-value pairs to obtain the first processing result; in the target cluster, splicing keys of the plurality of key-value pairs and splicing values of the plurality of key-value pairs to obtain the second processing result.

[0074] The data in the target topic is in the form of key-value pairs, which can include a plurality of key-value pairs. In the original cluster, the keys in the plurality of key-value pairs can be spliced into an integral key, and the values in the plurality of key-value pairs can be spliced into an integral value. The spliced key and the spliced value are taken as the first processing result.

[0075] Similarly, in the target cluster, the keys in the plurality of key-value pairs can be spliced into an integral key, and the values in the plurality of key-value pairs can be spliced into an integral value. The spliced key and the spliced value are taken as the second processing result.

[0076] In this embodiment, the original and target clusters adopt a unified data processing mode of key-value pair splicing to process the data in the target topic, adapting to the real-time processing requirements of streaming operations.

[0077] In an exemplary embodiment, the matching and verification of the first processing result and the second processing result includes: if the key in the second processing result matches the key in the first processing result, matching the value in the second processing result with the value in the first processing result field by field; if all fields match, determining the verification result as successful.

[0078] In the target cluster, the key in the second processing result is matched with the key in the first processing result. For keys that match, the corresponding value in the second processing result is matched field by field with the corresponding value in the first processing result to obtain the matching result for each field.

[0079] If all fields match, the validation is considered successful, meaning the first and second processing results are consistent.

[0080] refer to Figure 3 The verification job is run in the target cluster to compare the first and second processing results. If the verification passes, the second processing result, i.e. Sink Topic, is stored in Hive.

[0081] like Figure 4 The diagram illustrates data comparison in some embodiments. By receiving the original SinkTopic data from the original cluster job and the new SinkTopic data from the new cluster job, and adding job identifiers (JobFlag, 1 for new, 0 for original) to each, dual-stream merging is performed. A unique data identifier (MessageId, etc.) is found based on the job output data and configured as the key expression for the KeyBy operator, ensuring that data with the same key is routed to the same comparison task instance. The verification operator performs real-time field-by-field data matching based on the key. If a match fails, the abnormal result is stored in Hive, and an alarm message is generated.

[0082] like Figure 5 The diagram shows a flowchart of field-by-field matching in some embodiments.

[0083] (1) When data flows in, determine whether there is data in the local ValueState. If not, execute (2), otherwise execute (3). ValueState is a local state storage container used to cache first processing data and second processing data, with a business unique key as an index, to ensure that data with the same key can be accurately associated. (2) Store the incoming data in ValueState, and start a timer. The timer duration is configured by the reconciliation operator. If the waiting time exceeds the timer duration, it is determined to be a timeout, and (9) is executed. If the value with the same key is waited for within the timer, and the identification is the same, it is determined to be data duplication, and (3) is executed. If the value with the same key is waited for within the timer, and the identification is different, (4)-(8) are executed. (3) Read data from ValueState, and read reconciliation fields from the operator configuration. If the incoming data and the ValueState value job identification are the same, it is determined to be an exception, and the field information is output, otherwise the timer is deleted. (4) Compare the values of each reconciliation field one by one (both are null values, or the field does not exist and is considered the same). If the comparison has differences, it means that the comparison fails, and the information accumulation (StringBuild.append) of the old job value different from the new job value of this field is increased, and the next field comparison is continued. (5) After all field comparisons are completed, determine whether to input the comparison result according to the comparison identification. (6) If the identification is false, it means an exception, and the field information is output. (7) Clear the ValueState data. (8) If the identification is true, directly clear the ValueState data. (9) The timer is triggered, and the ValueState data job output is determined, and the ValueState data is cleared after determination. Through the Keyby operator, the same Key data is routed to one comparison task, and the key-controlled comparison solves the unordered data flow problem. The comparison operator compares the two streams of data in real time after Union, uses a timer to solve the problems of delay and loss, and performs forced comparison after timeout, and marks the data as "timeout", so as to release the state in time to avoid memory leakage, and ensure the stability of the comparison task.

[0084] The result of real-time comparison is output through the target cluster Sink Topic. For abnormal comparison results, an alarm is sent to the operation and maintenance platform, and the operation and maintenance personnel process the exception in time. The abnormal result is also written into the Hive database, which is convenient for data tracing and operation and maintenance in the future.

[0085] In this embodiment, the verification is divided into two levels of key matching and value matching. The key matching serves as a precondition to quickly filter out the comparison of non-same business data and avoid invalid value comparison from consuming cluster resources. Meanwhile, the field-by-field matching of the value ensures that the granularity of the verification reaches the field level, directly locks the difference field, and accurately identifies the tiny difference of a single field. The real-time requirement of the streaming job can be adapted to ensure that the verification result can be fed back in time to provide real-time monitoring support for the parallel stage of migration.

[0086] In one example embodiment, consuming real-time data in a target topic from an event platform in a target cluster includes: in the target cluster, consuming real-time data in the target topic from the event platform by a first user.

[0087] In the cluster switching stage, the original cluster stops running the streaming job, and the target cluster consumes real-time data in the target topic from the event platform by the first user. The first user is a dedicated consumption account of the event platform in the original cluster. In the cluster switching stage, the target cluster reuses the first user to inherit the consumption progress of the original cluster, avoid data missing or repeated consumption, ensure seamless connection of the data processing link, and output results meeting the requirements of the downstream without adjusting the job processing logic, realize non-perception switching of the downstream system, and guarantee business continuity.

[0088] In one example embodiment, the method further includes: writing the first processing result and the second processing result into a preset database respectively; and executing a batch verification script every interval of a preset time period to perform batch verification on the first processing result and the second processing result stored in the preset database.

[0089] The preset database can be a Hive database. The first processing result and the second processing result processed in the original cluster and the target cluster are written into the preset database. After a preset time period, a certain amount of first processing results and second processing results are stored in the preset database. The target cluster can perform batch verification on the first processing result and the second processing result in the preset database every interval of a preset time period (e.g., every day).

[0090] The batch verification script is an automated program for batch verification, which can exist in the form of a SQL script, a Python script, etc., and has built-in logic such as data association, field comparison, and exception statistics. The core function of the batch verification script is to replace manual verification to achieve efficient batch comparison of the first and second processing results and improve verification efficiency and accuracy.

[0091] Real-time verification focuses on "comparison by piece, immediate feedback", but may ignore small probability differences due to data disorder and transient fluctuations; while batch verification can find problems missed in real-time verification through "centralized correlation and statistical analysis". For example, cumulative error (small differences in individual data, amplified differences after batch), differences in specific time periods, and field implicit differences, etc., to improve the integrity of verification.

[0092] In some embodiments, by performing the batch verification script at a fixed time every day, the total amount of new and old Sink Topic data is compared to assist and back up data checking, verify long-term data consistency, and avoid the pressure of high-cost aggregation calculation throughout the process.

[0093] In this embodiment, by persisting the first and second processing results to the preset database, historical data can be traced back through the database, and the total amount of daily batch calibration data is issued at the batch level to capture large deviations accumulated over a long period of time due to small logical differences. The combination of the two comparison methods achieves high accuracy and low false positives, ensuring timeliness and final consistency, which is conducive to finding problems missed in real-time verification and improving the reliability of migration switching.

[0094] To explain the data difference real-time detection method and effect based on the streaming job migration scene in this scheme in detail, a most detailed embodiment is described as follows:

[0095] The data difference real-time detection method based on the streaming job migration scene includes two stages of parallel and switching. During the parallel stage, the original cluster job consumes the topic of the event platform Kafka from the original consumption group, processes logically, and then issues to the downstream system consumption through the original Sink Topic of the original cluster Kafka. At the same time, the Sink Topic data is stored in the real-time comparison job Source Topic for real-time data comparison and Hive library for batch data verification. The new cluster runs the parallel job during the parallel stage, consumes the event platform Kafka topic using the new consumption group, processes logically, and then the new Sink Topic data of the new cluster Kafka is not connected to the downstream system, but is stored in the real-time comparison job Source Topic of the new cluster for real-time data comparison and Hive library for batch data verification. The checking job simultaneously consumes the Sink Topic of the original cluster and the Sink Topic of the new cluster, compares whether all the field data is consistent through the business unique ID as the Key as the pairing, and falls the Hive library and notifies the operation and maintenance personnel for real-time troubleshooting. When the data comparison and verification result of the original cluster job and the new cluster parallel job is normal for a period of time, the parallel stage ends and enters the switching stage. Before switching, the original job of the original cluster is stopped, and after verifying that all the operators have completed processing through the log, the switching job of the new cluster consumes the event platform Kafka message using the original consumption group, processes logically, and then issues to the downstream system through the original Sink Topic of the new cluster Kafka and stores in the Hive library for problem troubleshooting. If the new job has a problem, the old job can be rolled back after the operator processing is completed to ensure business continuity. After the switching job of the new cluster runs normally and stably, the downstream system is notified to switch to running.

[0096] In some embodiments, the streaming job, comparison job development and operator configuration are realized through dragging and connecting on the development platform canvas. The entire migration and comparison can also be realized through script code, and the implementation method is flexible and diverse.

[0097] The data comparison method adopts operator-level real-time data comparison, with low delay (milliseconds), and can accurately find the specific field difference of each result data, greatly facilitating problem positioning. The original Sink Topic data of the original cluster original job and the new Sink Topic data of the new cluster parallel job are received, and after adding job identification (JobFlag, 1 represents new and 0 represents original), double-flow merging is performed. According to the data of the job output result, a unique data identifier (MessageId, etc.) is found, configured as the Key expression of the KeyBy operator, and the same Key data is routed to the same comparison task instance.

[0098] The result of real-time comparison is output through a new cluster Sink Topic, and an alarm is sent to the operation and maintenance platform for abnormal comparison results, and the operation and maintenance personnel timely handle the abnormality. The abnormal result is also written into the Hive database, which is convenient for data tracing and operation and maintenance in the future.

[0099] The Sink Topic data of all jobs is written into the Hive database for storage, providing a rich data basis for post-tracing and root cause analysis. By executing a batch verification script at a fixed time every day, the total amount of new and old Sink Topic data is compared to assist and bottom out data checking, verify long-term data consistency, and avoid the pressure of high-cost aggregation calculation throughout the process.

[0100] The above-mentioned data difference real-time detection method based on the stream job migration scene is not smooth compared to the traditional migration scheme, and there may be business interruption risk, the data comparison scheme has poor real-time performance, and cannot quickly locate specific problems, and has poor processing capability for out-of-order and delayed data. Large financial institutions and the banking industry have many real-time continuous business scenarios, and there is no smooth stream job migration scheme and real-time accurate data comparison method, which is undoubtedly inconvenient for operation and maintenance personnel. The method of the embodiment of the application, by using the Keyby and state management in the real-time comparison operator, processing out-of-order and delayed data problems, comparing real-time data piece by piece, and at the same time assisting batch comparison to verify the final consistency of the data, all original data and comparison results during migration are traceable.

[0101] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0102] Based on the same inventive concept, the embodiments of the present application also provide a device for implementing the data difference real-time detection method based on the stream job migration scenario as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more device embodiments for data difference real-time detection based on the stream job migration scenario provided below can be referred to the limitations of the data difference real-time detection method based on the stream job migration scenario described above, and will not be repeated here.

[0103] In one exemplary embodiment, as shown in Figure 6 A device 600 for data difference real-time detection based on the stream job migration scenario is provided, comprising a migration module 620, a processing module 640 and a verification module 660, wherein:

[0104] The migration module 620 is configured to run the stream job in the original cluster and the target cluster in parallel after the program of the stream job is migrated from the original cluster to the target cluster;

[0105] The processing module 640 is configured to obtain a first processing result and a second processing result after the data in the target topic corresponding to the stream job is processed by the original cluster and the target cluster respectively during the parallel running, and to issue the first processing result to a downstream system;

[0106] The verification module 660 is configured to perform matching verification on the first processing result and the second processing result, and to stop the running of the stream job in the original cluster and issue a third processing result obtained by processing real-time data in the target topic from the event platform in the target cluster to the downstream system when the multiple verification results within a preset time length are all verified to be correct.

[0107] The data difference real-time detection device based on the stream job migration scene realizes data input homology by the original cluster and the new cluster consuming data based on the same target topic after the program of the stream job is migrated from the original cluster to the target cluster. The original cluster and the target cluster run the stream job in parallel, and during the parallel running, the first processing result and the second processing result of the data in the target topic corresponding to the stream job processed by the original cluster and the target cluster are obtained, and the first processing result is issued to the downstream system. In this way, the original cluster job continues to run during the migration process and connects the downstream system, and the new cluster job processes data in parallel without connecting the downstream system. During the parallel running, the target cluster matches and checks the first processing result and the second processing result after processing, and in the case that multiple check results within a preset time length are all checked, it is indicated that the target cluster can output correct results consistent with the original cluster, and the stream job migration is accurate. Then, the cluster switching stage is entered, the running of the stream job in the original cluster is stopped, and real-time data in the target topic is consumed from the event platform in the target cluster, and the third processing result obtained by processing the real-time data is issued to the downstream system, thereby ensuring the continuity of the business logic, the smooth migration of the stream job, and the accuracy of the stream data.

[0108] In one of the embodiments, the processing module 640 is further configured to: consume the target topic corresponding to the stream job from the event platform by a first user in the original cluster, and consume the target topic corresponding to the stream job from the event platform by a second user in the target cluster; obtain the first processing result of the data in the target topic processed by the original cluster, and obtain the second processing result of the data in the target topic processed in the target cluster.

[0109] In one of the embodiments, the data in the target topic includes a plurality of key-value pairs; the processing module 640 is further configured to: in the original cluster, splice the keys of the plurality of key-value pairs respectively, splice the values of the plurality of key-value pairs respectively, and obtain the first processing result; and in the target cluster, splice the keys of the plurality of key-value pairs respectively, splice the values of the plurality of key-value pairs respectively, and obtain the second processing result.

[0110] In one of the embodiments, the first processing result and the second processing result are matched and verified, and the verification module 660 is further configured to: in the case that the key in the second processing result matches the key in the first processing result, the value in the second processing result is matched with the value in the first processing result field by field; and in the case that all fields are matched, the verification result is determined to be verified.

[0111] In one of the embodiments, the real-time data in the target topic is consumed from the event platform in the target cluster, and the verification module 660 is further configured to: in the target cluster, the real-time data in the target topic is consumed from the event platform by the first user.

[0112] In one of the embodiments, the verification module 660 is further configured to: write the first processing result and the second processing result into a preset database respectively; and execute a batch verification script every interval of a preset time period, and perform batch verification on the first processing result and the second processing result stored in the preset database.

[0113] The above various modules in the data difference real-time detection device based on the stream job migration scene can be all or partially realized by software, hardware and combinations thereof. The above various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above various modules.

[0114] In one of the exemplary embodiments, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 7As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the external terminal in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to realize a data difference real-time detection method based on a stream job migration scene. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0115] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0116] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the method embodiments described above.

[0117] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the processor executes the computer program to implement the steps in each of the method embodiments described above.

[0118] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by the processor to implement the steps in each of the method embodiments described above.

[0119] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0120] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0121] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0122] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for real-time detection of data difference based on a streaming job migration scenario, characterized in that, The method comprises: After a program of a streaming job is migrated from an original cluster to a target cluster, running the streaming job in parallel in the original cluster and the target cluster; During the parallel running, obtaining a first processing result and a second processing result of data in a target topic corresponding to the streaming job processed by the original cluster and the target cluster respectively, and delivering the first processing result to a downstream system; Matching and checking the first processing result and the second processing result, and in the case that multiple checking results within a preset time length are all checked, stopping running of the streaming job in the original cluster, and consuming real-time data in the target topic from an event platform in the target cluster, and delivering a third processing result of processing the real-time data to the downstream system.

2. The method of claim 1, wherein, The obtaining of the first processing result and the second processing result of data in the target topic processed by the original cluster and the target cluster respectively comprises: In the original cluster, a first user consumes the target topic corresponding to the streaming job from an event platform, and in the target cluster, a second user consumes the target topic corresponding to the streaming job from the event platform; Obtaining the first processing result of data in the target topic processed by the original cluster, and obtaining the second processing result of data in the target topic processed by the target cluster.

3. The method of claim 1, wherein, The data in the target topic comprises multiple key-value pairs; the obtaining of the first processing result and the second processing result of data in the target topic processed by the original cluster and the target cluster respectively comprises: In the original cluster, the keys of the multiple key-value pairs are spliced, and the values of the multiple key-value pairs are spliced to obtain the first processing result; In the target cluster, the keys of the multiple key-value pairs are spliced, and the values of the multiple key-value pairs are spliced to obtain the second processing result.

4. The method of claim 1, wherein, The matching and checking of the first processing result and the second processing result comprises: In the case that the keys in the second processing result match the keys in the first processing result, the values in the second processing result are matched with the values in the first processing result field by field; In the case that all fields are matched, it is determined that the checking result is checked.

5. The method of claim 1, wherein, The consumption of real-time data in the target topic from the event platform in the target cluster comprises: In the target cluster, a first user consumes real-time data in the target topic from an event platform.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: Writing the first processing result and the second processing result into a preset database respectively; Executing a batch checking script every preset time period to batch check the first processing result and the second processing result stored in the preset database.

7. A data difference real-time detection device based on a streaming job migration scene, characterized in that, The device comprises: A migration module is configured to run the stream job in the original cluster and the target cluster in parallel after the program of the stream job is migrated from the original cluster to the target cluster; A processing module is configured to obtain a first processing result and a second processing result of data in a target topic corresponding to the stream job in the original cluster and the target cluster respectively during the parallel running, and deliver the first processing result to a downstream system; A verification module is configured to perform matching verification on the first processing result and the second processing result, and stop running of the stream job in the original cluster and consume real-time data in the target topic from an event platform in the target cluster in a case that a plurality of verification results within a preset time length are all verified to be passed, and deliver a third processing result of processing the real-time data to the downstream system. 8.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.

Images