Data flow processing method and device between different task nodes and electronic equipment

Abstract

The application discloses a data stream processing method and device between different task nodes and electronic equipment, and relates to the technical field of big data. The method comprises the following steps: performing a modulo operation on a hash value corresponding to an extreme value in a first data stream sent by an upstream task node, and dividing the first data stream into L data blocks, wherein the cut point positions corresponding to the L data blocks are determined by the result of the modulo operation; when a modification operation occurs in the first data stream, recording the modification time and modification content of the first data stream, wherein the modification content comprises the change of the data blocks in the first data stream; and controlling the second data stream update of a downstream task node according to the modification time and modification content of the first data stream, wherein the second data stream is the first data stream before the modification that is saved by the downstream task node. The application solves the technical problem of high transmission network cost in the prior art when data streams are transmitted between different task nodes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a data flow processing method and device between different task nodes and electronic equipment. BACKGROUND

[0002] With the continuous expansion of the bank core business system, and the data volume of each business system is also growing, there is a complex execution dependency relationship between business systems. For the demand of bank business application system batch task, most small and medium-sized banks have too high cost of manual supervision because of the variety of business, and the manual operation process of front and rear task dependency is complex, which leads to the independent operation of the analysis, processing, sending, and supervision process between systems. The running connection cost and task deadline cannot be controlled, and it cannot be considered as a perfect batch system.

[0003] In the financial field, employees still need to run a lot of batch systems after work every day, such as batch card opening, batch transfer, batch settlement, batch opening, batch query, batch analysis and summary system, etc. Among them, the batch task nodes of the batch card opening system include application collection, identity verification, risk assessment, approval decision, development preparation, and opening operation, etc. There are also complex dependency relationships between thousands of batch systems. The execution of the first task of the downstream system relies on the completion of the upstream system as data input, that is, the start time of the task of the downstream system.

[0004] When transmitting data from upstream task nodes to downstream task nodes in a single business application system through full transmission, there will be a problem of high transmission network cost. For example, when determining the changed part of the data, if the system transmits a snapshot file, the network transmission cost will be very high when the change frequency is high.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] The present application provides a data flow processing method and device between different task nodes and electronic equipment to at least solve the technical problem of high transmission network cost when transmitting data flow between different task nodes in the prior art.

[0007] According to an aspect of the present application, a data stream processing method between different task nodes is provided, comprising: dividing a first data stream sent by an upstream task node into L data blocks by performing a modulo operation on a hash value corresponding to an extreme value in the first data stream, wherein L is an integer greater than 1, and the cut point positions corresponding to the L data blocks are determined by the result of the modulo operation; recording a modification time and modification content of the first data stream when a modification operation occurs in the first data stream, wherein the modification content includes the change of the data blocks in the first data stream; and controlling a second data stream update of a downstream task node according to the modification time and the modification content of the first data stream, wherein the second data stream is the first data stream before the modification occurs saved by the downstream task node.

[0008] In an optional embodiment, the data stream processing method between different task nodes further comprises: setting a sliding window at a starting position of the first data stream; repeatedly performing M target operations, wherein M is an integer greater than or equal to 1, the target operation is used to slide the sliding window in the first data stream while performing a modulo operation on a hash value corresponding to an extreme value of data in the sliding window, and determining a cut point position of the data according to the result of the modulo operation; determining N cut point positions according to the operation results of the M target operations, wherein N is an integer greater than or equal to 1; and dividing the data stream into L data blocks according to the N cut point positions.

[0009] In an optional embodiment, the target operation comprises the following steps: obtaining a hash value corresponding to an extreme value of data in the sliding window in the first data stream; performing a modulo operation on the hash value and detecting whether the result of the modulo operation is a preset threshold; in the case where the result of the modulo operation is the preset threshold, taking the position of the extreme value of the data in the sliding window as a cut point position; and in the case where the result of the modulo operation is not the preset threshold, sliding the sliding window to the next data.

[0010] In an optional embodiment, the modification operation comprises at least one of the following operations: a first operation for deleting at least one data block in the L data blocks corresponding to the first data stream; a second operation for inserting at least one data block in the L data blocks corresponding to the first data stream; and a third operation for modifying the content of at least one data block in the L data blocks corresponding to the first data stream.

[0011] In an optional embodiment, the data stream processing method between different task nodes further comprises: in the case that the modification operation is the first operation or the second operation, taking each data block deleted in the first data stream or each data block newly inserted as a target data block; determining identification information of the target data block, wherein the identification information of the target data block at least comprises index identification of the target data block, character index identification and modification identification, wherein the index identification is used to record the actual position of the target data block when the upstream task node sends the target data block to the downstream task node, the character index identification is used to determine the first character and the last character of the target data block, and the modification identification is used to determine the modification operation type corresponding to the target data block; taking the identification information of the target data block and the target data block as the modification content of the first data stream, and recording the modification time and the modification content of the first data stream in the differential file.

[0012] In an optional embodiment, the data stream processing method between different task nodes further comprises: in the case that the modification operation is the third operation, calculating a rolling check value and an MD5 check value for each data block in the first data stream before the modification operation occurs; taking the first data stream after the third operation occurs as a comparison data stream, and calculating a rolling check value and an MD5 check value for each data block in the comparison data stream; determining the modification content of the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream; and recording the modification time and the modification content of the first data stream in the differential file.

[0013] In an optional embodiment, the data stream processing method between different task nodes further comprises: determining at least one updated data block in the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, wherein the updated data block is a data block with different rolling check values before and after the third operation occurs in the first data stream or a data block with different MD5 check values before and after the third operation occurs in the first data stream; and taking the updated data block and the identification information of the updated data block as the modification content of the first data stream.

[0014] In an alternative embodiment, the data stream processing method between different task nodes further comprises: placing the at least one changed data block in the first data stream into a memory area of the downstream task node in order after sorting the at least one changed data block according to the index identification of the at least one changed data block in the first data stream; extracting a data block different from the index identification of the data block in the memory area from the second data stream as a third data block; merging the third data block with the data block in the memory area into a third data stream according to the index identification of the third data block; and updating the second data stream to the third data stream.

[0015] In an alternative embodiment, the data stream processing method between different task nodes further comprises: when a fault occurs in the data stream transmission between the downstream task node and the upstream task node, storing the rolling check value and the MD5 check value of each data block of the upstream task node in a check file respectively; when the data stream transmission between the downstream task node and the upstream task node is restored to normal, controlling the upstream task node to send the check file to the downstream task node; when the downstream task node detects that the rolling check value of the kth data block stored by itself is different from the rolling check value of the kth data block in the check file or the MD5 check value of the kth data block stored by itself is different from the MD5 check value of the kth data block in the check file, determining that the kth data block is an abnormal data block in transmission, where k is a positive integer; and controlling the upstream task node to retransmit the kth data block and the data blocks after the kth data block to the downstream task node.

[0016] According to another aspect of the embodiments of the present application, a data stream processing device between different task nodes is also provided, wherein the data stream processing device between different task nodes comprises: a first processing unit configured to divide a first data stream sent by an upstream task node into L data blocks by performing a modulo operation on a hash value corresponding to an extreme value in the first data stream, where L is an integer greater than 1, and the cut point positions corresponding to the L data blocks are determined by the result of the modulo operation; a second processing unit configured to record a modification time and a modification content of the first data stream when a modification operation occurs in the first data stream, wherein the modification content includes the change of the data block in the first data stream; and a third processing unit configured to control the update of a second data stream of a downstream task node according to the modification time and the modification content of the first data stream, wherein the second data stream is the first data stream before the modification saved by the downstream task node.

[0017] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, wherein the computer readable storage medium stores a computer program, and when the computer program runs, the computer readable storage medium controls the device where the computer readable storage medium is located to perform the data stream processing method between different task nodes of any one of the above.

[0018] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes one or more processors and a memory, the memory being configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data flow processing method between different task nodes of any one of the above.

[0019] In the present application, the first data stream is divided into L data blocks by taking modulo operation on the hash value corresponding to the extreme value in the first data stream sent by the upstream task node, and when a modification operation occurs in the first data stream, the modification time and the modification content of the first data stream are recorded, wherein the modification content includes the change of the data block in the first data stream. Then the second data stream of the downstream task node is updated according to the modification time and the modification content of the first data stream, wherein the second data stream is the first data stream before the modification. Wherein L is an integer greater than 1, and the cutting point position corresponding to the L data blocks is determined by the result of the modulo operation, thereby solving the technical problem of high transmission network cost when transmitting the data stream between different task nodes in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and its description, which serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of an optional data flow processing system according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of an optional data flow processing method between different task nodes according to an embodiment of the present application;

[0023] Figure 3 is a flowchart of another optional data flow processing method between different task nodes according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an optional data flow division according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an optional new data stream block after deleting character data according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of an optional new data stream block after inserting character data according to an embodiment of the present application;

[0027] Figure 7is a schematic diagram of an optional data flow modification process according to an embodiment of the application;

[0028] Figure 8 is a structure diagram of a distributed batch scheduling architecture method based on differential transmission and rollback according to an embodiment of the application;

[0029] Figure 9 is a schematic diagram of an optional data flow processing device between different task nodes according to an embodiment of the application;

[0030] Figure 10 is a schematic diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the personnel in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected by the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards in the relevant region, necessary security measures are taken, and public order and good customs are not violated, and appropriate operation portals are provided for users to choose authorization or refusal. For example, interfaces are provided between the system and related users or institutions, and before obtaining the relevant information, the interface needs to send a request to the aforementioned user or institution, and after receiving the consent information feedback from the aforementioned user or institution, the relevant information is obtained.

[0034] In addition, in the present application, the customer information is collected, the customer information is analyzed, and the corresponding operation portal is provided for the user to select to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0035] In a single business application system, when the data of the upstream node is transmitted to the downstream node in batches through full transmission, firstly, the data consistency is affected due to some fault reconnection during transmission; secondly, the problem of network cost of transmission exists, when determining the changed part of the data, if the system transmits a snapshot file, when the change frequency is high, the network transmission cost will be very high; thirdly, there is a problem of system complexity, when the traditional incremental data stream transmission needs to build a large system framework, the system complexity is increased, which may need to invest more resources and cost for the bank; fourthly, when the traditional incremental transmission model is selected, it will cause a lot of calculation delay when checking the data, which is not friendly to the bank's day cut-off date; fifthly, the changed data in the traditional incremental transmission may contain sensitive information, so the safety and reliability of the amount and balance data need to be ensured during the transmission process to ensure that the data is not leaked and tampered; seventhly, when the transmitted data is not only the row data of the database, the log transmission sql statement mode is no longer applicable.

[0036] In order to improve the accuracy and rapidity of incremental transmission, according to the embodiments of the present application, an embodiment of a data stream processing method between different task nodes is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0037] In an alternative embodiment, a data stream processing system can serve as an execution subject of a data stream processing method among different task nodes in the embodiments of the present application, wherein, as shown in Figure 1 , the data stream processing system comprises a business application system 1 and a differential control system 2.

[0038] The business application system 1 is deployed in a production environment and is used to process daily batch business data of a financial institution, Figure 1 The scheduling relationship between two task nodes in the business application system.

[0039] In addition, as shown in Figure 1 , the present application builds a differential control system 2 on the basis of the business application system 1. When the business application system 1 runs daily batch business data, the data stream of the upstream and downstream task nodes is monitored through the cooperation of the differential file and the listening module. Whenever the data stream timestamp changes, data block, verification value comparison, and grabbing of the changed data packet are performed. The upstream node differentially transmits the changed data and calls the downstream node to accept the differential changed old data stream.

[0040] Optionally, Figure 2 is a flowchart of an alternative data stream processing method among different task nodes according to an embodiment of the present application, as shown in Figure 2 , the method comprises the following steps:

[0041] In step S201, the first data stream uploaded by the upstream task node is divided into L data blocks by performing a modulo operation on the hash value corresponding to the extreme value in the first data stream.

[0042] In step S201, L is an integer greater than 1, and the cut point position corresponding to the L data blocks is determined by the result of the modulo operation.

[0043] Optionally, Figure 3 is a flowchart of another alternative data stream processing method among different task nodes according to an embodiment of the present application, as shown in Figure 3 , in Figure 3In step 1 of the method, when the business personnel find that there is a new transaction business that has not been checked and needs to be supplemented on the same day, including audit data, verification data, user information, amount information, etc., the business personnel can first cut off the running state of the current task node (corresponding to the upstream task node) through the data flow processing system. If the downstream task node depends on the data of the upstream task node, the downstream task node will also be suspended by the data flow processing system. Secondly, the business personnel writes the new business data into the file of the current task node, and when the file is modified and the submit button is clicked, the data flow processing system will automatically modify the timestamp of the current file to the latest state. Finally, when the listener listens to the change of the timestamp of the data flow, the subsequent process of data block, verification value comparison, and grabbing changed data packets is prepared.

[0044] Optionally, in step 2 of the method, Figure 3 In step 2 of the method, the data flow processing system can perform data flow block on the data flow (corresponding to the first data flow) sent by the upstream task node, wherein after the first data flow is blocked, an index identifier is attached to each data block, and a character index identifier is attached to the cut point position before and after each data block.

[0045] In step S202, when a modification operation occurs in the first data flow, the modification time and the modification content of the first data flow are recorded.

[0046] In step S202, the modification content includes the change of the data block in the first data flow.

[0047] Optionally, the modification time of the first data flow can be represented by a timestamp, and the modification operation includes at least one of the following operations: a first operation for deleting at least one data block in the L data blocks corresponding to the first data flow; a second operation for inserting at least one data block in the L data blocks corresponding to the first data flow; and a third operation for modifying the content of at least one data block in the L data blocks corresponding to the first data flow.

[0048] In step S203, the second data flow of the downstream task node is updated according to the modification time and the modification content of the first data flow.

[0049] In step S203, the second data flow is the first data flow saved by the downstream task node before the modification.

[0050] Based on the contents of the above steps S201 to S203, in the present application, the first data stream is divided into L data blocks by taking modulo operation on the hash value corresponding to the extreme value in the first data stream sent by the upstream task node. When a modification operation occurs in the first data stream, the modification time and the modification content of the first data stream are recorded, wherein the modification content includes the change of the data block in the first data stream. Then, the second data stream of the downstream task node is updated according to the modification time and the modification content of the first data stream, wherein the second data stream is the first data stream before the modification saved by the downstream task node. Wherein, L is an integer greater than 1, and the cut point positions corresponding to the L data blocks are determined by the result of the modulo operation, thereby solving the technical problem of high transmission network cost in the prior art when transmitting data streams between different task nodes.

[0051] In an optional embodiment, in order to divide the data stream into L data blocks, the data stream processing system can set a sliding window at the starting position of the first data stream, and then repeatedly perform M times of target operations, wherein M is an integer greater than or equal to 1, the target operation is used to slide the sliding window in the first data stream, and perform modulo operation on the hash value corresponding to the extreme value of the data in the sliding window, and determine the cut point position of the data according to the result of the modulo operation. Finally, the data stream processing system determines N cut point positions according to the operation results of M times of target operations, and divides the data stream into L data blocks according to the N cut point positions, wherein N is an integer greater than or equal to 1.

[0052] Optionally, the target operation includes the following steps:

[0053] Obtaining the hash value corresponding to the extreme value of the data in the sliding window in the first data stream; performing modulo operation on the hash value, and detecting whether the result of the modulo operation is a preset threshold value; in the case that the result of the modulo operation is the preset threshold value, the position of the extreme value of the data in the sliding window is taken as a cut point position; in the case that the result of the modulo operation is not the preset threshold value, the sliding window is slid to the next data.

[0054] In an optional embodiment, Figure 4 is an optional data stream division schematic diagram according to an embodiment of the present application, as Figure 4 shown, first set a cut point 1 at the starting position of the sliding window, and then take the extreme value f max in the data stream in the sliding window (assuming the length of the sliding window is 3) to the hash value (i.e. hash(f max ), and take modulo operation on the hash value with a specified D (assuming D = 4), when the value after the modulo operation is equal to a specified value d (assuming d is 2), the position of the cut point is divided to f maxAt the location, block 1 is formed; when the f is not found... max And simultaneously satisfy hash(f) max When mod D = d, the fixed-length window will continue to move to the right until a tangent point is reached.

[0055] Using the above method, the entire data stream can be divided into many blocks of unequal length, for example... Figure 4 The data is divided into blocks 1 and 2, where block 1 is determined by cutpoints 1 and 2, and block 2 is determined by cutpoints 2 and 3. The extreme value at cutpoint 2 corresponds to a hash value of 10, and the extreme value at cutpoint 3 corresponds to a hash value of 14.

[0056] It should be noted that the modulo method is used in this application to determine the cut point position in order to increase the resistance to byte drift after the data stream is modified. The addition of extreme values ​​is to increase the gap between different data blocks so as to avoid the problem of excessively frequent data block verification.

[0057] For example, Figure 5 This is a schematic diagram of an optional data stream segmentation after deleting character data according to an embodiment of this application, such as... Figure 5 As shown, when the deletion is Figure 4 In addition to hash(f) max When the character at position 10 is deleted, it has absolutely no effect on the cutoff points 1, 2, and 3 in the original data stream. However, when the character deleted is hash(f... max When the character at position 10 is removed, segment 2 will be automatically removed and segments 1 and 2 will be merged. However, the relative positions of segment 1 and 3 are not affected, indicating that this method is effective in resisting byte drift when dealing with deleted data.

[0058] Optionally, Figure 6 This is a schematic diagram of an optional data stream segmentation after inserting character data according to an embodiment of this application, such as... Figure 6 As shown, if the inserted character is anywhere within the interval of block 1 of the old data stream, and if both the extreme value and modulo conditions are not met simultaneously, the inserted character will not affect cutoff points 1, 2, and 3; if the inserted character meets both the extreme value and modulo conditions simultaneously, a new cutoff point 1 will be added. * Re-divide block 1 into Figure 6 The blocks (1) and (2) in the middle, but the tangent point 1 at this time * The relative positions of cut points 1, 2, and 3 were not affected, indicating that the sliding window model is effective in resisting byte drift when inserting data in data blocks.

[0059] In an alternative embodiment, the modifying operation comprises at least one of the following operations: a first operation for deleting at least one data block in the L data blocks corresponding to the first data stream; a second operation for inserting at least one data block in the L data blocks corresponding to the first data stream; and a third operation for modifying the content of at least one data block in the L data blocks corresponding to the first data stream.

[0060] Optionally, in the case where the modifying operation is the first operation or the second operation, each data block deleted in the first data stream or each newly inserted data block is taken as a target data block; the identification information of the target data block is determined, wherein the identification information of the target data block at least comprises an index identification, a character index identification and a modifying identification, wherein the index identification is used to record the actual position of the target data block when the upstream task node sends the target data block to the downstream task node, the character index identification is used to determine the first character and the last character of the target data block, and the modifying identification is used to determine the type of the modifying operation corresponding to the target data block; the identification information of the target data block and the target data block are taken as the modifying content of the first data stream, and the modifying time and the modifying content of the first data stream are recorded in the differential file.

[0061] Optionally, when the first data stream in the upstream task node changes, the data stream processing system first creates a differential file in the upstream task node, and each differential file is provided with a time stamp identification for data stream rollback of the downstream task node.

[0062] Secondly, as shown in FIG. 2, the data stream processing system represents the data stream on which the first operation and / or the second operation is performed according to data blocks, and stores these data blocks (corresponding to the deleted data blocks and the inserted data blocks in FIG. 1) in the differential file. Figure 3 Figure 3 Optionally, the data stream processing system can also store the data blocks (corresponding to the modified data blocks in FIG. 1) in the differential file.

[0063] Optionally, the data stream processing system can also store the data blocks (corresponding to the modified data blocks in FIG. 1) in the differential file. Figure 7 is a schematic diagram of an alternative data stream modifying process according to an embodiment of the present application, as shown in FIG. 3, the data stream f(t1) is divided into four data blocks, namely data block 0, data block 1, data block 2 and data block 3, and secondly, Figure 7 the arrow in FIG. 3 indicates that the data block of the old data stream in the upstream reaches the data block position in the downstream, and the identification information of each data block at least comprises: the index identification (for example, the number below each data block in FIG. 3), the character index identification attached to the cutting point before and after each data block, and the time stamp (for example, the time stamp below each data block in FIG. 3) of the modifying data stream record. Figure 7 Figure 6 Figure 6 ​​​t1 and t2 in the first data stream), a character index, a timestamp, and a modification identifier. The index identifier is used to record the placement position of the block to be deleted and the block to be inserted by the upstream node when the block is uploaded to the downstream node. The character index is used to determine the start and end of the characters in the data block, facilitating subsequent calculation of the check value. The timestamp is used to record the rollback time of the current data stream, facilitating subsequent rollback operation. The modification identifier is used to determine whether the current block is in the whole block deletion, the whole block insertion, or the modification of individual characters (including deletion, insertion, and modification) in the block, facilitating use in data stream reorganization at the downstream node.

[0064] In an optional embodiment, in the case where the modification operation is the third operation, the data stream processing system calculates the rolling check value and the MD5 check value for each data block in the first data stream before the third operation, wherein the rolling check value is used to represent the mathematical relationship between the i th character and the j th character in the data block, and the i th character and the j th character are different characters in the data block. In addition, the data stream processing system also takes the first data stream after the third operation as a comparison data stream, and calculates the rolling check value and the MD5 check value for each data block in the comparison data stream. Subsequently, the data stream processing system determines the modification content of the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, and finally records the modification time and the modification content of the first data stream in the difference file.

[0065] Optionally, the data stream processing system can determine at least one updated data block in the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, wherein the updated data block is a data block with different corresponding rolling check values before and after the third operation in the first data stream or a data block with different corresponding MD5 check values before and after the third operation in the first data stream. Finally, the data stream processing system takes the updated data block and the identification information of the updated data block as the modification content of the first data stream.

[0066] Optionally, as shown in step 3 in the first data stream processing system, Figure 3 As shown in step 3 in the first data stream processing system, the data stream processing system calculates the rolling check value and the MD5 check value for each data block in the data stream with the latest timestamp and the data stream with the previous timestamp, and compares the two check values. If one of the two check values is not equal, it indicates that there is an updated character in the data block, and the update operation includes the deletion, insertion, and modification of the character in the data block. Then, the data stream processing system stores the data block in the difference file with the latest timestamp, and uploads the file to the downstream node. The rolling check value can be defined as follows:

[0067]

[0068]

[0069] inspect value =inspect1+2 16 inspect2

[0070] Wherein, inspect value is a rolling inspection value, indicating the algebraic sum of the rolling modulo operation on the i-th character to the j-th character of the data block, 2 16 is designed to discretize the value of the character in the hash memory unit. The calculation speed of the rolling inspection value mode is fast, but the accuracy is not high, and it can be used for preliminary judgment. If the value does not pass, it is directly considered that the two data blocks must be different. The MD5 inspection value can be realized by a special plug-in tool. The MD5 inspection mode is relatively complex, but the accuracy is high enough, so only when the rolling inspection value passes, the MD5 inspection value is determined.

[0071] In an optional embodiment, as shown in step 4 in Figure 3 , the data stream processing system can send all the deletion chunks, insertion chunks and update chunks in the difference file to the downstream node in the format of file data stream, store them in the cache, and wait for data block recombination.

[0072] In an optional embodiment, the data stream processing system can sort the at least one data block changed in the first data stream according to the index identifier of the at least one data block changed in the first data stream, and then put the at least one data block changed in the first data stream into the memory area of the downstream task node. Then, the data stream processing system extracts data blocks different from the index identifier of the data blocks in the memory area from the second data stream as third data blocks, and then merges the third data blocks with the data blocks in the memory area according to the index identifier of the third data blocks to form a third data stream. Finally, the data stream processing system updates the second data stream to the third data stream.

[0073] Optionally, as shown in step 5 in Figure 3 , during the difference transmission process, the data stream processing system first opens a new temporary memory area in the downstream node; secondly, the insertion chunks and deletion chunks in the difference file are put into the temporary memory area according to the index order. Then, the update chunks in the difference file are put into the temporary memory area according to the index order. Finally, the data stream processing system fills the missing index in the temporary memory area with the corresponding data chunk in the downstream node to form a new data stream, and then replaces the old data stream in the downstream node with the new data stream. After that, every time difference transmission is needed, only the difference file needs to be uploaded, and the full data stream file does not need to be uploaded.

[0074] Optionally, as shown in step 5 in Figure 3As shown in step 6, during differential rollback, if the upstream task node detects an error in the updated file, it first needs to send the specified timestamp identifiers t2, t3, ..., t n The data is then sent to downstream nodes; secondly, it is identified by the timestamp t sent upstream. n Obtain all the previous timestamps t2, t3, ..., t n-1 The difference files f(t2), f(t3), ..., f(t) n-1 The process involves initializing a full copy of the file data stream f(t1); finally, performing a data stream differential operation at a specified timestamp allows rollback to a previous data stream at any timestamp, such as... Figure 7 f(t2), f(t3), ..., f(t) n )

[0075] In one optional embodiment, when a data stream transmission failure occurs between a downstream task node and an upstream task node, the data stream processing system stores the rolling checksum and MD5 checksum of each data block of the upstream task node in a checksum file. When the data stream transmission between the downstream and upstream task nodes resumes normal operation, the system controls the upstream task node to send the checksum file to the downstream task node. If the downstream task node detects that the rolling checksum of its stored k-th data block differs from the rolling checksum of the k-th data block in the checksum file, or that the MD5 checksum of its stored k-th data block differs from the MD5 checksum of the k-th data block in the checksum file, it determines that the k-th data block is a data block with transmission anomalies, where k is a positive integer. The system then controls the upstream task node to retransmit the k-th data block and all subsequent data blocks to the downstream task node.

[0076] Optionally, such as Figure 3 Step 7: The data stream processing system also provides an incremental resume function after data stream failure and reconnection. When a data stream transmission failure occurs, the rolling check value and MD5 check value of each data block of the upstream node are first stored in a check file. Each set of check values ​​has two values, indicating the corresponding data block index. Then, after the downstream receives the check value file, it compares each set of check values ​​with the check values ​​of each data block downstream. Only data blocks that satisfy both check values ​​are marked as unmodified. Finally, if one of the two check values ​​is different, it means that there is a discrepancy between the data block and the uploaded file data block, which is the failure point. Incremental data can be re-uploaded from the failure point.

[0077] In one alternative embodiment, such as Figure 3In step 8 of the above, each time a batch job is performed, the data flow processing system performs a day cut in the early morning of the day, indicating that the work of the previous day has been completed, and all differential files, timestamp identifiers, and node caches are automatically cleaned up.

[0078] From the above, it can be seen that the application provides a distributed batch scheduling architecture method based on differential transmission and rollback. By using differential files to record the task node data stream changes at each timestamp, any rollback at any time can be flexibly controlled, and differential incremental transmission is performed, and differential continuous transmission is completed after fault reconnection, avoiding data loss of batch transactions in business application systems and ensuring transmission security. The data difference test is placed in the node that modifies the data, and the completion process of the differential file is waited for, which can greatly reduce the network bandwidth occupation and improve the transmission security of sensitive data such as amount, balance, and user information.

[0079] In an optional embodiment, Figure 8 A structure diagram of the distributed batch scheduling architecture method based on differential transmission and rollback is shown in FIG. 1. Figure 8 As shown in FIG. 1, the various functional models of the data flow processing system can include an initialization module, a network module, a differential control module, and an exception handling module.

[0080] The initialization module is configured to analyze, verify, and aggregate the batch transaction business of the day. The initialization module subdivides the processes of various businesses into the levels of task nodes, such as the batch task nodes of the batch card opening system, including application collection, identity verification, risk assessment, approval decision, development preparation, and opening operation. First, the initialization module initializes the dependency relationships of the task nodes in the business application system module, the properties and job contents of the batch tasks, and the like. Second, the initialization module initializes the network module by using a server cluster to schedule the task nodes of the business application system. Finally, the initialization module initializes the differential file cache.

[0081] Second, after the network module and the business application system module establish a matching relationship of mutual scheduling, when the network module sends a task node request, the differential control module selects a corresponding server for each recorded task node to perform network connection. When the connection is successful, a data packet is returned to perform batch processing of the task node.

[0082] Optionally, all task node information and corresponding differential files in each node are included in the differential control module. Firstly, when modified data flow appears in the business application system, the differential files are used for inter-node dependency transmission to complete differential transmission. Secondly, when unexpected changes occur in the transaction business, differential files can be issued according to the task dependency sequence to perform differential rollback. Finally, when the task node is abnormally disconnected and reconnected, the differential file can be loaded for the current task node to perform fault positioning and data continuation.

[0083] Finally, the exception handling module is used to cut off the currently running task node when an exception occurs in a transaction or customer information in the business application system, and the exception information is uploaded to the differential control module. When the fault handling is completed and the data flow is normally operated, the normal running identifier is returned, indicating that the exception is handled.

[0084] In summary, the technical scheme of the embodiment of the application can at least achieve the following technical effects:

[0085] 1. When the task node of the business application system changes information each time, only the difference data part of the latest timestamp data flow and the last timestamp data flow needs to be verified, and the network bandwidth occupation and transmission delay during transmission are reduced.

[0086] 2. The data flow incremental transmission and the database incremental log transmission are different. The database incremental log transmission is in units of data behaviors, and only the difference data rows in the modified table need to be compared. Each data row is a block. The data flow increment does not have a data format restriction and can be a picture, a data table, a file, and can also cope with the incremental transmission when the upstream node suddenly fails to reconnect. However, the database log incremental transmission must rely on the transaction commit statement as the log cut point identifier for rollback recovery when fault recovery occurs. This requires a large number of cut point identifiers from the upstream node during recovery, and the application range is not wide enough.

[0087] 3. Compared with the sliding window used in the traditional incremental transmission algorithm for data blocking, whether it is an incremental hash search cut point in a variable window or an extreme value hash search cut point, the anti-byte drift capability is weak. That is, when a field is deleted or inserted in the data flow, the subsequent data block cut point position of the modified field will be affected during data block division, causing an increase in time complexity.

[0088] 4. The breakpoint position can be accurately captured during the line disconnection and reconnection process after the fault occurs, ensuring data consistency.

[0089] 5. The complexity of the traditional incremental system is reduced, and the cost of manual maintenance is reduced. This can enable banks to manage more business application systems with smaller costs.

[0090] 6. Differential change is used to ensure the security of sensitive data such as amount, balance, and customer information during incremental transmission, and to prevent data leakage and tampering.

[0091] According to an embodiment of this application, an embodiment of a data stream processing apparatus between different task nodes is also provided. Figure 9 This is a schematic diagram of an optional data stream processing apparatus between different task nodes according to an embodiment of this application, such as... Figure 9 As shown, the data stream processing device between different task nodes includes: a first processing unit 901, a second processing unit 902, and a third processing unit 903.

[0092] The first processing unit 901 is used to divide the first data stream into L data blocks by performing a modulo operation on the hash values ​​corresponding to the extreme values ​​in the first data stream sent by the upstream task node, where L is an integer greater than 1, and the cutting point positions corresponding to the L data blocks are determined by the result of the modulo operation; the second processing unit 902 is used to record the modification time and modification content of the first data stream when a modification operation occurs in the first data stream, where the modification content includes the changes in the data blocks in the first data stream; the third processing unit 903 is used to control the update of the second data stream of the downstream task node according to the modification time and modification content of the first data stream, where the second data stream is the first data stream before the modification, which is saved by the downstream task node.

[0093] Optionally, the first processing unit 901 includes: a setting subunit, an execution subunit, a determination subunit, and a processing subunit. The setting subunit is used to set a sliding window at the beginning of the first data stream; the execution subunit is used to repeatedly execute a target operation M times, where M is an integer greater than or equal to 1. The target operation involves sliding the sliding window in the first data stream while performing a modulo operation based on the hash value corresponding to the extreme value of the data within the sliding window, and determining the data cut-off point position based on the result of the modulo operation; the determination subunit is used to determine N cut-off point positions based on the results of the M target operations, where N is an integer greater than or equal to 1; and the processing subunit is used to divide the data stream into L data blocks based on the N cut-off point positions.

[0094] Optionally, the executing subunit comprises an obtaining module, a detecting module, a first processing module and a second processing module. The obtaining module is configured to obtain a hash value corresponding to an extreme value of data in the first data stream within the sliding window; the detecting module is configured to perform a modulo operation on the hash value and detect whether a result of the modulo operation is a preset threshold; the first processing module is configured to, in a case where the result of the modulo operation is the preset threshold, take a position of the extreme value of the data in the sliding window as a cut point position; and the second processing module is configured to, in a case where the result of the modulo operation is not the preset threshold, slide the sliding window to the next data.

[0095] Optionally, the modifying operation comprises at least one of the following operations: a first operation for deleting at least one data block in the L data blocks corresponding to the first data stream; a second operation for inserting at least one data block in the L data blocks corresponding to the first data stream; and a third operation for modifying content of at least one data block in the L data blocks corresponding to the first data stream.

[0096] Optionally, the second processing unit comprises a first processing subunit, a first determining subunit and a first recording subunit. The first processing subunit is configured to, in a case where the modifying operation is the first operation or the second operation, take each data block deleted from the first data stream or each newly inserted data block as a target data block; the first determining subunit is configured to determine identification information of the target data block, wherein the identification information of the target data block at least comprises an index identification, a character index identification and a modification identification of the target data block, wherein the index identification is used to record an actual position of the target data block when an upstream task node sends the target data block to a downstream task node, the character index identification is used to determine a first character and a last character of the target data block, and the modification identification is used to determine a modification operation type corresponding to the target data block; and the first recording subunit is configured to take the identification information of the target data block and the target data block as the modification content of the first data stream, and record a modification time and the modification content of the first data stream in the differential file.

[0097] Optionally, the second processing unit comprises a calculation subunit, a second processing subunit, a second determination subunit and a second recording subunit. The calculation subunit is configured to calculate a rolling check value and an MD5 check value for each data block in the first data stream before the modification operation in a case where the modification operation is the third operation, wherein the rolling check value represents a mathematical relationship between an i th character and a j th character in the data block, and the i th character and the j th character are different characters in the data block. The second processing subunit is configured to take the first data stream after the third operation as a comparison data stream, and calculate a rolling check value and an MD5 check value for each data block in the comparison data stream. The second determination subunit is configured to determine the modification content of the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream. The second recording subunit is configured to record the modification time and the modification content of the first data stream in the differential file.

[0098] Optionally, the second determination subunit comprises a first determination module and a second determination module. The first determination module is configured to determine at least one updated data block in the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, wherein the updated data block is a data block with different corresponding rolling check values before and after the third operation in the first data stream or a data block with different corresponding MD5 check values before and after the third operation in the first data stream. The second determination module is configured to take the updated data block and identification information of the updated data block as the modification content of the first data stream.

[0099] Optionally, the third processing unit comprises a third processing subunit, a fourth processing subunit, a fifth processing subunit and an updating subunit. The third processing subunit is configured to sort at least one data block changed in the first data stream according to an index identification of the at least one data block, and place the sorted at least one data block in a memory area of a downstream task node. The fourth processing subunit is configured to extract a data block different from the data block in the memory area in the second data stream as a third data block according to an index identification of the data block in the memory area. The fifth processing subunit is configured to merge the third data block and the data block in the memory area into the third data stream according to the index identification of the third data block. The updating subunit is configured to update the second data stream to the third data stream.

[0100] Optionally, the data stream processing apparatus between different task nodes further comprises a storage unit, a control unit, a determination unit and a retransmission unit. The storage unit is configured to store the rolling check value and the MD5 check value of each data block of the upstream task node in a check file when a fault occurs in data stream transmission between the downstream task node and the upstream task node. The control unit is configured to control the upstream task node to send the check file to the downstream task node when the data stream transmission between the downstream task node and the upstream task node is restored to normal. The determination unit is configured to determine the kth data block as an abnormally transmitted data block when the rolling check value of the kth data block stored by the downstream task node is different from the rolling check value of the kth data block in the check file or the MD5 check value of the kth data block stored by the downstream task node is different from the MD5 check value of the kth data block in the check file, where k is a positive integer. The retransmission unit is configured to control the upstream task node to retransmit the kth data block and the data blocks after the kth data block to the downstream task node.

[0101] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, which includes a stored computer program. When the computer program is run, the computer readable storage medium controls the device where the computer readable storage medium is located to perform the data stream processing method between different task nodes of any one of the above embodiments.

[0102] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the data stream processing method between different task nodes of any one of the above embodiments by executing the executable instructions.

[0103] Figure 10 is a schematic diagram of an electronic device according to an embodiment of the present application. As shown in Figure 10 the embodiments of the present application provide an electronic device, which includes a processor, a memory and a program stored in the memory and executable on the processor. When the processor executes the program, the data stream processing method between different task nodes of any one of the above embodiments is implemented.

[0104] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0105] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0106] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0107] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0108] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0109] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various program code storage media.

[0110] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method for data flow processing between different task nodes, characterized in that, The method comprises the following steps: The first data stream is divided into L data blocks by performing a modulo operation on a hash value corresponding to an extreme value in the first data stream uploaded by an upstream task node, wherein L is an integer greater than 1, and a cut point position corresponding to the L data blocks is determined by a result of the modulo operation; When a modification operation occurs in the first data stream, a modification time and modification content of the first data stream are recorded, wherein the modification content comprises a change of a data block in the first data stream; A second data stream of a downstream task node is updated according to the modification time and modification content of the first data stream, wherein the second data stream is the first data stream before modification saved by the downstream task node.

2. The method of claim 1, wherein, The first data stream is divided into L data blocks by performing a modulo operation on a hash value corresponding to an extreme value in the first data stream uploaded by an upstream task node, comprising: A sliding window is set at a starting position of the first data stream; A target operation is repeatedly performed M times, wherein M is an integer greater than or equal to 1, the target operation is used to slide the sliding window in the first data stream, perform a modulo operation on a hash value corresponding to an extreme value of data in the sliding window, and determine a cut point position of the data according to a result of the modulo operation; N cut point positions are determined according to operation results of the M target operations, wherein N is an integer greater than or equal to 1; The data stream is divided into the L data blocks according to the N cut point positions.

3. The method of claim 2, wherein, The target operation comprises the following steps: A hash value corresponding to an extreme value of data in the sliding window in the first data stream is obtained; A modulo operation is performed on the hash value, and it is detected whether a result of the modulo operation is a preset threshold value; In a case where the result of the modulo operation is the preset threshold value, a position of the extreme value of the data in the sliding window is taken as a cut point position; In a case where the result of the modulo operation is not the preset threshold value, the sliding window is slid to a next data.

4. The method of claim 1, wherein, The modification operation comprises at least one of the following operations: A first operation is used to delete at least one data block in the L data blocks corresponding to the first data stream; A second operation is used to insert at least one data block in the L data blocks corresponding to the first data stream; A third operation is used to modify content of at least one data block in the L data blocks corresponding to the first data stream.

5. The method of claim 4, wherein, When a modification operation occurs in the first data stream, a modification time and modification content of the first data stream are recorded, comprising: In a case where the modification operation is the first operation or the second operation, each data block deleted in the first data stream or each data block newly inserted is taken as a target data block; determining identification information of the target data block, wherein the identification information of the target data block at least includes an index identification, a character index identification and a modification identification of the target data block, wherein the index identification is used to record an actual position of the target data block when the target data block is sent by the upstream task node to the downstream task node, the character index identification is used to determine a first character and a last character of the target data block, and the modification identification is used to determine a modification operation type corresponding to the target data block; record the modification time of the first data stream and the modification content in a differential file.

6. The method of claim 4, wherein, When the modification operation occurs in the first data stream, recording the modification time and the modification content of the first data stream, comprising: in the case that the modification operation is the third operation, calculating a rolling check value and an MD5 check value for each data block in the first data stream before the modification operation occurs, wherein the rolling check value is used to represent a mathematical relationship between an i th character and a j th character in the data block, and the i th character and the j th character are different characters in the data block; taking the first data stream after the third operation as a comparison data stream, and calculating a rolling check value and an MD5 check value for each data block in the comparison data stream; determining the modification content of the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream; record the modification time of the first data stream and the modification content in a differential file.

7. The method of claim 6, wherein, determining the modification content of the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, comprising: determining at least one updated data block in the first data stream according to the rolling check value and the MD5 check value of each data block in the first data stream and the rolling check value and the MD5 check value of each data block in the comparison data stream, wherein the updated data block is a data block with different rolling check values before and after the third operation in the first data stream or a data block with different MD5 check values before and after the third operation in the first data stream; taking the updated data block and the identification information of the updated data block as the modification content of the first data stream.

8. The method of claim 1, wherein, controlling the second data stream update of the downstream task node according to the modification time and the modification content of the first data stream, comprising: sorting the at least one data block changed in the first data stream according to the index identification of the at least one data block, and placing the at least one data block in a memory area of the downstream task node; extracting a third data block different from the data block in the memory area from the second data stream as a data block different from the data block in the memory area; merge the third data block and data blocks in the memory region into a third data stream according to an index of the third data block; update the second data stream as the third data stream.

9. The method of claim 1, wherein, The data stream processing method between the different task nodes further comprises: when a data stream transmission between the downstream task node and the upstream task node fails, store a rolling check value and an MD5 check value of each data block of the upstream task node in a check file respectively; when the data stream transmission between the downstream task node and the upstream task node is restored, control the upstream task node to send the check file to the downstream task node; when the downstream task node detects that a rolling check value of a kth data block stored by itself is different from a rolling check value of the kth data block in the check file or an MD5 check value of the kth data block stored by itself is different from an MD5 check value of the kth data block in the check file, determine that the kth data block is an abnormal data block, wherein k is a positive integer; control the upstream task node to retransmit the kth data block and data blocks after the kth data block to the downstream task node.

10. A data flow processing apparatus between different task nodes, characterized by, comprise: a first processing unit configured to split a first data stream sent by an upstream task node into L data blocks by performing a modulo operation on a hash value corresponding to an extreme value in the first data stream, wherein L is an integer greater than 1, and a cut point position corresponding to the L data blocks is determined by a result of the modulo operation; a second processing unit configured to record a modification time and modification content of the first data stream when a modification operation occurs in the first data stream, wherein the modification content comprises a change of a data block in the first data stream; a third processing unit configured to control a second data stream of a downstream task node to be updated according to the modification time and the modification content of the first data stream, wherein the second data stream is the first data stream saved by the downstream task node before the modification.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein when the computer program runs, the computer readable storage medium controls a device in which the computer readable storage medium is located to execute the data stream processing method between the different task nodes according to any one of claims 1 to 9.

12. An electronic device, comprising: comprise one or more processors and a memory configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data stream processing method between the different task nodes according to any one of claims 1 to 9.

Images