A method, device and system for controlling a data processing link

Abstract

The present invention discloses a control method, device and system for a data processing link, and relates to the field of computer technology. A specific implementation of the method includes: for each of a plurality of current task nodes in a data processing link as a current task node, executing: detecting the data quality of the data generated by the current task node; based on the detection result and the link blocking strategy pre-configured for the current task node, controlling the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the stored multiple link blocking strategies. This implementation can meet the different requirements of different data analysis and data calculation links for data quality.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a control method, device and system for a data processing link. Background Art

[0002] In the field of big data, it is generally necessary to analyze and calculate data based on a data processing link composed of multiple associated task nodes. The results of the analysis and calculation can provide references for corporate decision-making, product strategies, and artificial intelligence technologies. Since the quality of the data received by each of the multiple associated task nodes will directly affect the results of the analysis and calculation of the link, how to block the link based on data quality has become a problem that needs to be solved in the current big data processing process. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a control method, device and system for a data processing link, which can block the data processing link based on data quality and the requirements of each task node in the data processing link for data quality, and can meet the different requirements of different data analysis and data calculation links for data quality.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for controlling a data processing link is provided, comprising:

[0005] For each of the multiple current task nodes in the data processing link as a current task node, execute:

[0006] Detecting the data quality of the data generated by the current task node;

[0007] Based on the detection result and the link blocking strategy pre-configured for the current task node, the data processing link where the current task node is located is controlled, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored.

[0008] Preferably, the link blocking strategy pre-configured for the current task node includes:

[0009] When the detection result indicates that there is a quality problem with the data quality, the next task node of the current task node is directly blocked.

[0010] Preferably, controlling the data processing link where the current task node is located includes:

[0011] The task of the current task node is set to fail to execute, so as to block the next task node of the current task node.

[0012] Preferably, the link blocking strategy pre-configured for the current task node includes:

[0013] Send the detection result to the client, and receive the link execution instruction sent by the client;

[0014] When the link execution instruction indicates to block the data processing link, blocking the next task node of the current task node;

[0015] When the link execution instruction instructs to continue executing the data processing link, the next task node of the current task node is controlled to continue executing.

[0016] Preferably, when the detection result indicates that there is a quality problem with the data quality, the data generated by the current task node is locked;

[0017] Determine whether the next task node of the current task node has permission to access the locked data. If yes, control the next task node to process the locked data; otherwise, block the execution of the next task node.

[0018] Preferably, the data generated by the current task node belongs to multiple data files;

[0019] The data generated by the current task node is locked, including:

[0020] Locking the data files having quality problems among the multiple data files;

[0021] The controlling the next task node to process the locked data includes: controlling the next task node to process the locked data file with quality problems.

[0022] Preferably, the control method of the data processing link further comprises: setting a whitelist for the generated data, wherein the whitelist maintains at least one next task node corresponding to the current task node, and the next task node in the whitelist has the authority to process the locked data;

[0023] Determining whether the next task node has permission to access the locked data includes:

[0024] It is determined whether the next task node belongs to the whitelist of the locked data. If so, it is determined that the next task node has the authority to access the locked data. Otherwise, it is determined that the next task node does not have the authority to access the locked data.

[0025] Preferably, the control method of the data processing link further comprises:

[0026] When the data to be processed by the current task node does not include a data file with quality problems, the next task node is controlled to execute.

[0027] In a second aspect, an embodiment of the present invention provides a control device for a data processing link, comprising: a quality detection unit and a blocking control unit, wherein:

[0028] The quality detection unit is used for, for each of the multiple task nodes in the data processing link as a current task node, to perform: detecting the data quality of the data generated by the current task node;

[0029] The blocking control unit is used to execute, for each of the multiple task nodes in the data processing link as the current task node, the following: based on the detection result and the link blocking strategy pre-configured for the current task node, control the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored.

[0030] In a third aspect, an embodiment of the present invention provides a control system for a data processing link, including: a plurality of task nodes in the data processing link and a control device for the data processing link.

[0031] An embodiment of the above invention has the following advantages or beneficial effects: since the link blocking strategy pre-configured for the task node is one of the multiple link blocking strategies stored, different task nodes can be controlled based on the results of the detected data quality and different link blocking strategies, so as to block the data processing link based on the data quality and the requirements of each task node for data quality, thereby meeting the different requirements for data quality of different data analysis and data calculation.

[0032] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0034] Figure 1 is a schematic diagram of multiple data processing links composed of various task nodes according to an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the main flow of a method for controlling a data processing link according to an embodiment of the present invention;

[0036] Figure 3is a schematic diagram of a link blocking method implemented by multiple data processing links based on different link blocking strategies according to an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of a main process of determining the authority of a task node to access locked data according to an embodiment of the present invention;

[0038] Figure 5 is a schematic diagram of the main devices or systems of the control system of the data processing link according to an embodiment of the present invention;

[0039] Figure 6 is a schematic diagram of main units of a control device for a data processing link according to an embodiment of the present invention;

[0040] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] In the big data processing solution provided by the embodiment of the present invention, multiple data processing links can be formed through different current task nodes to process big data through the multiple data processing links, wherein the call of a task node in the multiple data processing links to the result generated by the previous task node of the task node can be completed by calling the system or directly implemented by the task node. Figure 1 As shown in the figure, the task nodes are task0, task01, task02, task1, task11, task2, task21, task22, task××, task××1, etc. The data processing link composed of these task nodes is as follows Figure 1 The example may include: task0→task01→…, task0→task02→…, task1→task11→…, task1→task2→task21→…, task1→task2→task22→…, task1→task2→task××→task××1→…, etc. Figure 1The control of each data processing link shown can be achieved through the data processing link control method provided by the embodiment of the present invention.

[0044] Figure 2 is a control method for a data processing link according to an embodiment of the present invention, such as Figure 1 As shown, the control method of the data processing link may include the following:

[0045] For each of the multiple current task nodes in the data processing link as the current task node, the following steps S201 to S202 are performed:

[0046] Step S201: Detecting the data quality of the data generated by the current task node;

[0047] Among them, the current task node can be a data cleaning node, a data processing node, a business analysis node, etc. When the current task node is the first task node, the data source on which the data generated by the current task node depends can be the original data obtained from other systems such as order management systems, e-commerce platforms, payment systems, etc., or data that has been preliminarily processed by other systems; when the current task node is a task node other than the first task node, the data source on which the data generated by the current task node depends can be part of the data or all of the data generated by the previous task node corresponding to the current task node. For example, for Figure 1 The data generated by the task nodes task0 and task1 shown in the figure depend on the data source which is the original data obtained from other systems or the data which has been preliminarily processed by other systems. Figure 1 The data source on which the data generated by the task nodes task01 and task02 shown depend is all or part of the data generated by the task node task0; the data source on which the data generated by the task nodes task11 and task2 depend is all or part of the data generated by the task node task1; the data source on which the data generated by the task nodes task21, task22 and task×× depend is all or part of the data generated by the task node task2; the data source on which the data generated by the task node task××1 depends is all or part of the data generated by the task node task××, and so on.

[0048] Step S202: Based on the detection result and the link blocking strategy pre-configured for the current task node, control the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored.

[0049] The detection result may indicate that the data quality of the data source is unqualified or that part of the data of the data source is erroneous or that the qualified rate of the data quality of the data source is lower than a preset qualified threshold, etc.

[0050] The stored multiple link blocking strategies may include the following link blocking strategies.

[0051] The first link blocking strategy is an active blocking mechanism, that is, when the detection result indicates that there is a quality problem with the data quality, the next task node of the current task node is directly blocked. Among them, the quality problem of data quality means that the data quality is unqualified or some data of the data source is wrong or the qualified rate of the data quality of the data source is lower than the preset qualified threshold, etc.

[0052] The second link blocking strategy is a passive blocking mechanism, which sends the detection result to the client and receives the link execution instruction sent by the client; when the link execution instruction instructs to block the data processing link, the next task node of the current task node is blocked; when the link execution instruction instructs to continue executing the data processing link, the next task node of the current task node is controlled to continue execution.

[0053] The third link blocking strategy is the locking file mechanism / data permission granting mechanism, that is, when the detection result indicates that there is a quality problem with the data quality, the data generated by the current task node is locked; it is determined whether the next task node of the current task node has the permission to access the locked data, if so, the next task node is controlled to process the locked data; otherwise, the execution of the next task node is blocked. The locking file mechanism / data permission granting mechanism, first, can solve the problem of relying on a certain scheduling system, because once the file with quality problems is locked, whether it is the next task node in the scheduling system or the data processing process of other task nodes that directly rely on the data file will be blocked. Second, the mechanism can also provide a whitelist, which can configure the task nodes or data processing process execution users that do not have high requirements for the quality of the data file into the whitelist, so that these task nodes and data processing processes will not be blocked and can continue. On the contrary, other data processing processes or current task nodes that have high requirements for the data quality will be blocked because they cannot read the data file.

[0054] The next task node of the current task node is a task node that has execution logic with the current task node on the data processing link and uses the data generated by the current task node as the data source. It is worth noting that there can be multiple next task nodes of the current task node. Figure 1For the data processing link shown, the current task node is task0, and the two next task nodes corresponding to task0 are task01 and task02; the current task node is task1, and the two next task nodes corresponding to task1 are task11 and task2; the current task node is task2, and the two next task nodes corresponding to task2 are task21, task22 and task××, etc.

[0055] against Figure 2 In the embodiment shown, since the link blocking strategy pre-configured for the task node is one of the multiple link blocking strategies stored, different task nodes can be controlled based on the results of the detected data quality and different link blocking strategies, so as to block the data processing link based on the data quality and the requirements of each task node for data quality, thereby meeting the different requirements for data quality of different data analyses and data calculations.

[0056] like Figure 3 As shown, they are Figure 1 Each task node in each data processing link shown is configured with a link blocking strategy, for example, the first link blocking strategy mentioned above is configured for task node task0; the second link blocking strategy is configured for task node task1; and the third link blocking strategy is configured for task node task2 (wherein, the task node task22 after task node task2 has the authority to obtain the problematic data or belongs to the whitelist in the third link blocking strategy, and task21 and task×× do not have the authority to obtain the problematic data or are not on the whitelist of the third link blocking strategy).

[0057] Among them, the first link blocking strategy and the second link blocking strategy are generally implemented with the help of calling the system, and the third link blocking strategy can be implemented directly through the communication between the current task nodes, or by calling the system.

[0058] For the current task node Figure 3In the case of the task node task0 shown, the link blocking strategy configured for the task node task0 is the first link blocking strategy mentioned above. The control method of the data processing links task0→task01→… and task0→task02→… where task0 is located takes the current task node as task0 as an example to illustrate the specific control method of the data processing link: detect the data quality of the data generated by the current task node task0; when the detection result indicates that there is a quality problem with the data quality, directly block the next task nodes task01 and task02 of task0. More specifically, the specific implementation method of controlling the data processing link where the current task node is located may include: setting the task of the current task node (such as task0) to execution failure, and blocking the next task node (task01 and task02) of the current task node. Specifically, the current task node (such as task0) is called by calling the system to block the call. That is, after task0 processes the data, it generates a data file task0_file. The data quality is checked on task0_file and the test result is returned. If the data quality does not meet the requirements, task0 will be set to failure. The downstream task nodes task01 and task02 of task0 will not continue to calculate the data because the dependency is not established, thus blocking the data processing link.

[0059] For the current task node Figure 3 The task node task1 is shown in the figure. The link blocking strategy configured for the task node task1 is the second link blocking strategy mentioned above. The control method of the data processing link task1→task11→… and task1→task2→… where task1 is located is described by taking the current task node task1 as an example to explain the specific control method of the data processing link: detect the data quality of the data generated by the current task node task1; send the detection result to the client, and receive the link execution instruction sent by the client; when the link execution instruction indicates to block the data processing link, block the next task node task11 of the current task node task1; when the link execution instruction indicates to continue to execute the data processing link, control the next task node task11 of the current task node to continue to execute. That is: after task1 processes the data, a data file task1_file is generated, and the data quality detection task1_file is performed and the detection result is returned. After the data processing logic of task1 is completed, the client used by the user obtains the result of the data quality detection through the gateway interface, and the client used by the user decides whether to block the downstream task node task11 of task1 according to the result of this detection.

[0060] For the current task node Figure 3In the case of the task node task2 shown in the figure, the link blocking strategy configured for the task node task2 is the third link blocking strategy mentioned above. The control method of the data processing links task2→task21→…, task2→task22→… and task2→task××→task××1→… where Task2 is located is described by taking the current task node task2 as an example to explain the control method of the data processing link specifically: Detect the current task node (for example Figure 3 When the detection result indicates that there is a quality problem with the data quality, the data generated by the current task node (such as task2) is locked. Figure 3 The data generated by task2 shown is task2_file1 and task2_file2); determine the next task node of the current task node (for example Figure 3The next task nodes of task2 shown in the figure are task21, task22 and taskXX respectively. If they have the permission to access the locked data, the next task node is controlled to process the locked data; otherwise, the next task node is blocked from execution. For example, if task node task22 has the permission to access the locked data, task22 is controlled to process the locked data; if task nodes task21 and taskXX are not, task21 and taskXX are blocked from execution. That is, after task2 finishes processing the data, it generates files task2_file1 and task2_file2. At this time, data quality detection is performed on task2_file1 and task2_file2, and the detection results are returned. If task2_file1 does not meet the quality requirements, the link blocking mechanism will lock task2_file1 by controlling the file permissions. All task nodes except those in the set unblocked task node whitelist have no permission to read the data file task2_file1 with quality problems. At this time, task21 in the downstream task node of task2 has a high quality dependency on task2_file1. Since task2_file1 has its permissions locked and task21 has no execution permission, task21 will fail to execute, and none of the downstream data processing tasks of task21 will be executed, thereby blocking the data processing link where task21 is located; task22 in the downstream task node of task2 has no permission to read task2_file1. The quality requirement of ask2_file1 is not high. There is a task node task22 in the whitelist. Since task22 still has the read permission of task2_file1, task22 can continue to execute and the data processing link where task22 is located can continue. In addition, there is a data processing process task×× in the non-scheduling system, which directly depends on task2_file1. Since task2_file1 has locked the permission, task×× will fail to execute, thereby blocking the data processing link where task×× is located.

[0061] Specifically, the data generated by the current task node belongs to multiple data files; accordingly, locking the data generated by the current task node may include: locking the data files with quality problems among the multiple data files; controlling the next task node to process the locked data may include: controlling the next task node to process the locked data files with quality problems.

[0062] In the embodiment of the present invention, Figure 4 As shown, the third link blocking strategy may further include the following steps:

[0063] Step S401: setting a whitelist for the generated data, wherein the whitelist maintains at least one next task node corresponding to the current task node, and the next task node in the whitelist has the authority to process the locked data;

[0064] Step S402: Determine whether the next task node belongs to the whitelist of locked data, if yes, execute step S403; otherwise, execute step S404;

[0065] Step S403: Determine whether the next task node has the authority to access the locked data, and end the current process;

[0066] Step S404: Determine whether the next task node has no authority to access the locked data.

[0067] In an embodiment of the present invention, the control method of the data processing link may further include: when the data to be processed by the current task node does not include a data file with quality problems, controlling the next task node to execute.

[0068] like Figure 5 As shown, an embodiment of the present invention provides a control device 500 for a data processing link. The control device 500 for a data processing link may include: a quality detection unit 501 and a blocking control unit 502, wherein:

[0069] The quality detection unit 501 is used for detecting the data quality of the data generated by the current task node for each of the task nodes in the data processing link as the current task node;

[0070] The blocking control unit 502 is used to execute, for each of the multiple task nodes in the data processing link as the current task node, the following: based on the detection result and the link blocking strategy pre-configured for the current task node, control the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored.

[0071] In an embodiment of the present invention, the link blocking strategy pre-configured by the blocking control unit 502 for the current task node may include: when the detection result indicates that there is a quality problem with the data quality, directly blocking the next task node of the current task node.

[0072] In the embodiment of the present invention, the blocking control unit 502 is used to set the task of the current task node as execution failure to block the next task node of the current task node.

[0073] In an embodiment of the present invention, the link blocking strategy pre-configured by the blocking control unit 502 for the current task node may include: sending the detection result to the client, and receiving the link execution instruction sent by the client; when the link execution instruction indicates to block the data processing link, blocking the next task node of the current task node; when the link execution instruction indicates to continue to execute the data processing link, controlling the next task node of the current task node to continue execution.

[0074] In an embodiment of the present invention, the link blocking strategy pre-configured by the blocking control unit 502 for the current task node may include: when the detection result indicates that there is a quality problem with the data quality, locking the data generated by the current task node; determining whether the next task node of the current task node has the authority to access the locked data, and if so, controlling the next task node to process the locked data; otherwise, blocking the execution of the next task node.

[0075] In an embodiment of the present invention, the data generated by the current task node belongs to multiple data files; the blocking control unit 502 is used to lock the data files with quality problems among the multiple data files; and control the next task node to process the locked data files with quality problems.

[0076] In an embodiment of the present invention, the blocking control unit 502 is further used to set a whitelist for the generated data, wherein the whitelist maintains at least one next task node corresponding to the current task node, and the next task node in the whitelist has the authority to process the locked data; determine whether the next task node belongs to the whitelist of the locked data, and if so, determine that the next task node has the authority to access the locked data, otherwise, determine that the next task node does not have the authority to access the locked data.

[0077] In the embodiment of the present invention, the blocking control unit 502 is further used to control the execution of the next task node when the data to be processed by the current task node does not include a data file with quality problems.

[0078] like Figure 6 As shown, an embodiment of the present invention provides a control system 600 for a data processing link, and the control system 600 for the data processing link may include: a plurality of task nodes 601 in the data processing link and a control device 500 for the data processing link provided in the above embodiment.

[0079] The control device 500 of the data processing link can configure a corresponding link blocking strategy for each task node, wherein the configured link blocking strategy is one of the stored multiple link blocking strategies.

[0080] In the embodiment of the present invention, Figure 6 As shown, the control system 600 of the data processing link may further include: a scheduling system 602, and the scheduling system 602 may schedule each task node according to the execution logic between each task node.

[0081] Figure 7 An exemplary system architecture 700 is shown to which a method for controlling a data processing link or an apparatus for controlling a data processing link according to an embodiment of the present invention may be applied.

[0082] like Figure 7 As shown, the system architecture 700 may include terminal devices 701, 702, 703, a network 704, a server 705 for controlling data processing links, servers 706, 707, 708 as task nodes, and a server 709 for scheduling task nodes. The network 704 is used to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705 for controlling data processing links, between the servers 706, 707, 708 of the task nodes, and between the servers 706, 707, 708 of the task nodes and the server 709 for scheduling task nodes. The network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0083] The user can use the terminal devices 701, 702, 703 to interact with the server 705 controlling the data processing link through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples). The terminal devices 701, 702, 703 can receive the results of the data quality detection sent by the server 705 controlling the data processing link, and receive the link execution instructions sent by the user through the terminal devices 701, 702, 703, so that the server 705 controlling the data processing link controls the servers 706, 707, 708 of the task nodes to block or continue to execute according to the link execution instructions.

[0084] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0085] The server 705 that controls the data processing link can be a server that provides various services, such as detecting the data quality of data generated by the servers 706, 707, and 708 of the task nodes, and controlling the servers 706, 707, and 708 of the task nodes based on the detection results and the link blocking strategy.

[0086] In addition, the server 705 that controls the data processing link can also call the server 709 that schedules the task node according to the link blocking strategy, so that the server 709 that schedules the task node can sequentially schedule the servers 706, 707, and 708 of each task node according to the execution logic between the servers 706, 707, and 708 of each task node.

[0087] It should be noted that the control method of the data processing link provided in the embodiment of the present invention is generally executed by the server 705 that controls the data processing link. Accordingly, the control device of the data processing link is generally arranged in the server 705 that controls the data processing link.

[0088] It should be understood that Figure 7 The number of terminal devices, networks, servers for controlling data processing links, servers for task nodes, and servers for scheduling task nodes is only illustrative. Depending on the implementation needs, there can be any number of terminal devices, networks, servers for controlling data processing links, servers for task nodes, and servers for scheduling task nodes.

[0089] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0090] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0091] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0092] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0093] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The units involved in the embodiments of the present invention may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a quality detection unit and a blocking control unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the quality detection unit may also be described as a "unit for detecting the data quality of data generated by the current task node".

[0096] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: for each of the multiple current task nodes in the data processing link as the current task node, executing: detecting the data quality of the data generated by the current task node; based on the detection result and the link blocking strategy pre-configured for the current task node, controlling the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored.

[0097] According to the technical solution of an embodiment of the present invention, since the link blocking strategy pre-configured for the task node is one of the multiple link blocking strategies stored, that is, different task nodes can be controlled based on the results of the detected data quality and different link blocking strategies, so as to achieve blocking of the data processing link based on the data quality and the requirements of each task node for data quality, thereby meeting the different requirements for data quality of different data analyses and data calculations.

[0098] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for controlling a data processing link, characterized in that: include: For each of the multiple current task nodes in the data processing link as a current task node, execute: Detecting the data quality of the data generated by the current task node; Based on the detection result and the link blocking strategy pre-configured for the current task node, control the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the stored multiple link blocking strategies; When the detection result indicates that there is a quality problem with the data quality, locking the data generated by the current task node; Determine whether the next task node of the current task node has the authority to access the locked data, and if so, control the next task node to process the locked data; otherwise, block the execution of the next task node; Further comprising: setting a whitelist for the generated data, wherein the whitelist maintains at least one next task node corresponding to the current task node, and the next task node in the whitelist has the authority to process the locked data; Determining whether the next task node has permission to access the locked data includes: It is determined whether the next task node belongs to the whitelist of the locked data. If so, it is determined that the next task node has the authority to access the locked data. Otherwise, it is determined that the next task node does not have the authority to access the locked data.

2. The control method of the data processing link according to claim 1, characterized in that: The link blocking strategy pre-configured for the current task node includes: When the detection result indicates that there is a quality problem with the data quality, the next task node of the current task node is directly blocked.

3. The control method of the data processing link according to claim 2, characterized in that: Controlling the data processing link where the current task node is located, including: The task of the current task node is set to fail to execute, so as to block the next task node of the current task node.

4. The control method of the data processing link according to claim 1, characterized in that: The link blocking strategy pre-configured for the current task node includes: Send the detection result to the client, and receive the link execution instruction sent by the client; When the link execution instruction indicates to block the data processing link, blocking the next task node of the current task node; When the link execution instruction instructs to continue executing the data processing link, the next task node of the current task node is controlled to continue executing.

5. The control method of the data processing link according to claim 1, characterized in that: The data generated by the current task node belongs to multiple data files; The data generated by the current task node is locked, including: Locking the data files having quality problems among the multiple data files; The controlling the next task node to process the locked data includes: controlling the next task node to process the locked data file with quality problems.

6. The control method of the data processing link according to claim 5, characterized in that: Further including: When the data to be processed by the current task node does not include a data file with quality problems, the next task node is controlled to execute.

7. A control device for a data processing link, characterized in that: include: A quality detection unit and a blocking control unit, wherein: The quality detection unit is used for, for each of the multiple task nodes in the data processing link as a current task node, to perform: detecting the data quality of the data generated by the current task node; The blocking control unit is used to execute, for each of the multiple task nodes in the data processing link as a current task node: based on the detection result and the link blocking strategy pre-configured for the current task node, controlling the data processing link where the current task node is located, wherein the link blocking strategy pre-configured for the current task node is one of the multiple link blocking strategies stored; The link blocking strategy pre-configured by the blocking control unit 502 for the current task node may include: when the detection result indicates that there is a quality problem with the data quality, locking the data generated by the current task node; determining whether the next task node of the current task node has the authority to access the locked data, and if so, controlling the next task node to process the locked data; otherwise, blocking the execution of the next task node; The blocking control unit 502 is further used to set a whitelist for the generated data, wherein the whitelist maintains at least one next task node corresponding to the current task node, and the next task node in the whitelist has the authority to process the locked data; determine whether the next task node belongs to the whitelist of the locked data, and if so, determine that the next task node has the authority to access the locked data, otherwise, determine that the next task node does not have the authority to access the locked data.

8. A control system for a data processing link, characterized in that: include: Multiple task nodes in a data processing link and a control device for the data processing link as claimed in claim 7.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images