Data processing method, device, computer equipment and storage medium

Abstract

The present application relates to the field of data analysis, and provides a data processing method, apparatus, device and computer storage medium, the method comprising: determining the blood relationship between multiple target data based on preset blood relationship determination rules; generating a blood relationship graph according to the blood relationship between the multiple target data, wherein the blood relationship graph is a directed graph; identifying the blood relationship graph and determining the ring structure in the blood relationship graph; performing data processing tasks on the multiple target data according to the ring structure in the blood relationship graph. It is capable of identifying and processing the ring structure in the blood relationship graph of the target data, preventing the waste of computing resources and improving the stability of the computing system. The present application also relates to artificial intelligence, and the data processing method of the present application can be applied to cloud servers of cloud computing services of big data and artificial intelligence platforms.

Description

Technical Field

[0001] The present application relates to the field of data analysis, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] Data warehouses generate a large amount of data during business processing. Data lineage analysis can quickly determine the source of the data and the target data generated based on it. However, when a data warehouse stores a large amount of data, a ring structure may easily appear in the data lineage. For example, a data lineage relationship similar to "node 1=>node 2=>node 3=>node 1" may appear. Users cannot detect the dead loop in the data lineage relationship in time, causing the computer to consume a large amount of computing resources in the process of executing the ring structure, and even cause a crash. Summary of the Invention

[0003] The main purpose of this application is to provide a data processing method, device, equipment and computer storage medium, which aims to identify and process the ring structure in the blood relationship diagram of the target data, prevent the waste of computing resources, and improve the stability of the computing system.

[0004] In a first aspect, the present application provides a data processing method, the data processing method comprising the following steps:

[0005] Determine the blood relationship between multiple target data based on preset blood relationship determination rules;

[0006] Generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph;

[0007] Identifying the blood relationship graph to determine a ring structure in the blood relationship graph;

[0008] According to the ring structure in the blood relationship graph, data processing tasks are performed on the multiple target data.

[0009] In a second aspect, the present application further provides a data processing device, comprising:

[0010] A blood relationship determination module is used to determine the blood relationship between multiple target data based on preset blood relationship determination rules;

[0011] A blood relationship graph generating module, configured to generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph;

[0012] A blood relationship diagram recognition module, configured to recognize the blood relationship diagram and determine a ring structure in the blood relationship diagram;

[0013] The data processing module is used to perform data processing tasks on the multiple target data according to the ring structure in the blood relationship diagram.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the data processing method as described above is implemented.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the data processing method as described above is implemented.

[0016] The present application provides a data processing method, apparatus, device, and computer storage medium. The present application determines the blood relationship between multiple target data based on a preset blood relationship determination rule; generates a blood relationship graph based on the blood relationship between the multiple target data, wherein the blood relationship graph is a directed graph; identifies the blood relationship graph and determines the ring structure in the blood relationship graph; and performs data processing tasks on the multiple target data based on the ring structure in the blood relationship graph. The present application is capable of determining the blood relationship graph between the target data, identifying and processing the ring structure in the blood relationship graph of the target data, preventing waste of computing resources, and improving the stability of the computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a data processing method provided in one embodiment of the present application;

[0019] Figure 2 A diagram showing a usage scenario of a data processing method provided in one embodiment of the present application;

[0020] Figure 3 A schematic block diagram of a data processing device provided in one embodiment of the present application;

[0021] Figure 4 This is a schematic block diagram of the structure of a computer device involved in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0024] Embodiments of the present application provide a data processing method, apparatus, computer device, and computer-readable storage medium.

[0025] Please refer to Figure 1 , Figure 1 A flow chart of a data processing method provided for an embodiment of the present application. The data processing method can be used in a terminal or a server to process target data in the terminal or the server. The terminal can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device; the server can be an independent server, a server cluster, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0026] Please refer to Figure 2 , Figure 2 This is a usage scenario diagram provided by an embodiment of the present application. Figure 2 As shown, the server can generate a blood relationship diagram based on the blood relationship between target data generated by its own processing of target data, perform data processing tasks based on the blood relationship diagram, and send the blood relationship diagram to the terminal, where it is displayed. Figure 2 The server in the example is not limited to an independent server, but may also be a distributed computing system including multiple service nodes, which is not limited here.

[0027] like Figure 1 As shown, the data processing method includes steps S101 to S104.

[0028] Step S101: Determine the blood relationship between multiple target data based on a preset blood relationship determination rule.

[0029] Exemplarily, the server can process the target data generated by the business system according to preset data processing statements, such as data extraction, conversion, cleaning, loading, etc., so that the target data meets the needs of data warehouse storage and query.

[0030] Exemplarily, based on a preset blood relationship determination rule, the blood relationship between the plurality of target data is determined according to the data processing statement.

[0031] Exemplarily, the data processing statement may be written based on the Structured Query Language (SQL), but is certainly not limited thereto and may also be written based on other languages, which is not limited here.

[0032] Taking Structured Query Language (SQL) as an example, we identify pre-defined statements in SQL. These are statements within data processing statements used to indicate data lineage relationships. For example, they may identify operations such as conversion or generation of target data. Specifically, these pre-defined statements may be statements such as INSERT and CREAT in SQL, but this is not a limitation here.

[0033] Step S102: Generate a blood relationship graph according to the blood relationship between the multiple target data, wherein the blood relationship graph is a directed graph.

[0034] For example, the kinship relationship between the plurality of target data is represented by a directed graph, i.e., a kinship graph. The directed graph for identifying the kinship relationship of data is usually a directed acyclic graph (DAG). However, in practical applications, it is usually difficult to ensure that no ring structure appears in the kinship relationship of data.

[0035] Exemplarily, the blood relationship graph includes target data nodes for representing multiple target data, and data flow directions for representing the blood relationship between the multiple target data nodes.

[0036] Exemplarily, the user can view the target data corresponding to the target data node through the target data node in the blood relationship diagram. For example, when the user clicks on any target data node, the target data corresponding to the target data node is displayed to the user. Of course, this is not limited to this.

[0037] For example, the target data may contain sensitive information. In order to prevent the risk of leakage of the sensitive information, the target data may be desensitized when generating the blood relationship diagram.

[0038] In some embodiments, step S102 generates a blood relationship diagram based on the blood relationship between the multiple target data, including: determining the sensitive data in the target data based on a preset sensitive data determination rule; desensitizing the sensitive data based on a preset desensitizing rule; and generating the blood relationship diagram based on the target data after desensitization.

[0039] For example, the sensitive data may be a mobile phone number, ID number, account number, etc. The server may determine whether the target data is sensitive data by identifying the field name or the data format of the target data. For example, if a preset keyword is identified in the field name, the target data corresponding to the field is determined to be sensitive data; or if the target data format is identified as conforming to a preset sensitive data format, the target data is determined to be sensitive data.

[0040] Exemplarily, based on a preset desensitization rule, the sensitive data is desensitized, including: based on the preset desensitization rule, determining the characters to be replaced in the sensitive data; and replacing the characters to be replaced with preset non-sensitive characters.

[0041] Exemplarily, the desensitization rule can determine the replacement characters according to the sensitive data type. For example, if the sensitive data is a mobile phone number, the last four digits of the sensitive data are determined as the characters to be replaced according to the preset desensitization rule. Of course, it is not limited to this.

[0042] Exemplarily, the non-sensitive character may be, for example, an "*", but is certainly not limited thereto and is not limited here.

[0043] Exemplarily, the target data includes different types of data, for example, the data types in the structured query language include text, numbers, and dates, etc. In order to facilitate users to view the blood relationship between target data of a specific data type through a blood relationship diagram, a blood relationship diagram can be generated for the target data corresponding to the target data type selected by the user.

[0044] In some embodiments, step S102 generates a blood relationship diagram based on the blood relationship between the multiple target data, including: according to the target data type selected by the user, determining the target data corresponding to the target data type in the multiple target data as candidate target data; and generating a blood relationship diagram based on the blood relationship between the candidate target data.

[0045] For example, the user can select text data as the target data type, determine the target data whose data type is text data as the candidate target data, and generate a lineage relationship diagram corresponding to the text data so that the user can clearly and intuitively understand the data lineage between the target data of a specific data type.

[0046] Exemplarily, the blood relationship diagram also includes a target service node corresponding to each target data node, that is, a service node that processes the target data of the target data node, so that the user can determine the service node corresponding to the target data in the distributed system. The user can quickly find the service node in the distributed system based on the target data node, thereby improving the maintenance efficiency of the distributed system.

[0047] Step S103: Identify the blood relationship graph and determine the ring structure in the blood relationship graph.

[0048] For example, when a data warehouse stores a large amount of data, the blood relationship between the target data may form a ring structure, causing the computing system to fall into an infinite loop when executing the ring structure, consuming a large amount of computing resources. In order to prevent this from happening, it is necessary to identify the ring structure in the blood relationship graph.

[0049] In some implementations, step S103 identifies the blood relationship graph and determines the ring structure in the blood relationship graph, including: traversing the blood relationship graph based on a preset depth-first algorithm to determine the ring structure in the blood relationship graph.

[0050] Exemplarily, based on a preset depth-first search algorithm (DFS), the target data nodes are traversed according to the direction of data flow in the blood relationship graph. If there are target data nodes that are repeatedly visited in the blood relationship graph, it indicates that there is a ring structure in the blood relationship graph.

[0051] Exemplarily, the access status when traversing the target data node can be recorded by a numerical value, for example, the numerical value 0 indicates that the target data node has not been accessed, the numerical value 1 indicates that the target data node has been accessed once, and the numerical value 2 indicates that the target data node has been accessed more than once. Of course, it is not limited to this.

[0052] Exemplarily, the directed graph structure composed of adjacent repeatedly visited target data nodes in the blood relationship graph is determined as the ring structure. It can be understood that the target data nodes in the ring structure are all adjacent repeatedly visited target data nodes.

[0053] Step S104: performing data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph.

[0054] For example, in order to prevent the execution of the data lineage relationship of the ring structure from consuming the computing resources of the computing system, the identified ring structure needs to be processed.

[0055] In some embodiments, step S104 performs data processing tasks on the multiple target data based on the ring structure in the blood relationship graph, including: searching for data processing tasks related to the target data corresponding to the ring structure in the blood relationship graph based on the blood relationship graph; when it is determined that there is a data processing task related to the target data corresponding to the ring structure in the blood relationship graph, terminating the data processing task.

[0056] Exemplarily, the consumption of computing resources is prevented by terminating (killing) the data processing tasks related to the target data corresponding to the ring structure in the blood relationship graph.

[0057] Exemplarily, before terminating the data processing task, a prompt may be given to the user, for example, task information corresponding to the data processing task to be terminated may be displayed to the user so that the user can confirm the termination of the data processing task.

[0058] In some embodiments, step S104 performs data processing tasks on the multiple target data based on the ring structure in the blood relationship graph, including: determining the current data processing volume of the target service node based on the blood relationship graph, and the target service node is a service node in the distributed system that performs data processing tasks; if the current data processing volume of the target service node is greater than a preset threshold, increasing the number of service nodes to distribute the data processing tasks of the target service node to the increased service nodes.

[0059] Exemplarily, the target service node corresponding to each target data node can also be determined based on the bloodline relationship graph, that is, the service node that processes the target data corresponding to the target data node can be determined, and the current data processing volume of the target service node can be determined through the bloodline relationship graph. If the current data processing volume of the target service node is greater than a preset threshold, the current data processing volume is dispersed, and the target data processed by the target service node is dispersed to other service nodes to prevent data skew and improve the stability of the computing system.

[0060] In some embodiments, determining the current data processing capacity of the target service node based on the bloodline relationship graph includes: determining the target data table connected to the target service node based on the bloodline relationship graph; and determining the current data processing capacity of the target service node based on the data volume of the target data table.

[0061] Exemplarily, based on the correspondence between the target service node and the target data node, a target data table connected to the target service node is determined. Specifically, a target service node may correspond to multiple data tables or only one data table. Based on the data volume of the target data table, for example, based on the sum of the data volumes of the multiple target data tables connected to the target service node, the current data processing capacity of the target service node is determined. When the current data processing capacity exceeds a preset threshold corresponding to the target service node, the target data processed by the target service node is distributed to other service nodes, thereby improving the stability of the computing system.

[0062] The data processing method provided in the above embodiment determines the blood relationship between multiple target data based on a preset blood relationship determination rule; generates a blood relationship graph based on the blood relationship between the multiple target data, wherein the blood relationship graph is a directed graph; identifies the blood relationship graph to determine the ring structure in the blood relationship graph; and performs data processing tasks on the multiple target data based on the ring structure in the blood relationship graph. This method can determine the blood relationship graph between the target data and identify and process the ring structure in the blood relationship graph of the target data, thereby preventing waste of computing resources and improving the stability of the computing system.

[0063] See also Figure 3 , Figure 3 This is a schematic diagram of a data processing device provided in one embodiment of the present application. The data processing device can be configured in a server or a terminal to execute the aforementioned data processing method.

[0064] like Figure 3 As shown, the data processing device includes: a blood relationship determination module 110, a blood relationship graph generation module 120, a blood relationship graph recognition module 130, and a data processing module 140.

[0065] A blood relationship determination module 110 is configured to determine the blood relationship between multiple target data based on a preset blood relationship determination rule;

[0066] A blood relationship graph generating module 120 is configured to generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph;

[0067] A blood relationship diagram recognition module 130 is used to recognize the blood relationship diagram and determine the ring structure in the blood relationship diagram;

[0068] The data processing module 140 is used to perform data processing tasks on the multiple target data according to the ring structure in the blood relationship diagram.

[0069] Exemplarily, the blood relationship graph recognition module 130 further includes: a blood relationship graph recognition submodule.

[0070] The blood relationship graph recognition submodule is used to traverse the blood relationship graph based on a preset depth-first algorithm to determine the ring structure in the blood relationship graph.

[0071] Exemplarily, the blood relationship diagram generation module 120 further includes: a sensitive data determination module, a desensitization processing submodule, and a blood relationship diagram generation submodule.

[0072] The sensitive data determination module is used to determine the sensitive data in the target data based on a preset sensitive data determination rule.

[0073] The desensitization processing submodule is used to desensitize the sensitive data based on preset desensitization rules.

[0074] The blood relationship diagram generation submodule is used to generate the blood relationship diagram based on the target data after desensitization processing.

[0075] Exemplarily, the data processing module 140 further includes: a task search module and a task termination module.

[0076] The task search module is used to search for data processing tasks related to target data corresponding to the ring structure in the blood relationship diagram according to the blood relationship diagram.

[0077] The task termination module is used to terminate the data processing task when it is determined that a data processing task is related to the target data corresponding to the ring structure in the blood relationship diagram.

[0078] Exemplarily, the blood relationship diagram generating module 120 further includes: a candidate target data determining module and a candidate target data relationship diagram generating module.

[0079] The candidate target data determining module is used to determine, according to the target data type selected by the user, target data corresponding to the target data type from the plurality of target data as candidate target data.

[0080] The candidate target data relationship graph generation module is used to generate a blood relationship graph based on the blood relationship between candidate target data.

[0081] Exemplarily, the data processing module 140 further includes: a data processing volume determination module and a data processing task distribution module.

[0082] The data processing capacity determination module is used to determine the current data processing capacity of the target service node based on the blood relationship diagram. The target service node is a service node in the distributed system that performs data processing tasks.

[0083] The data processing task distribution module is used to increase the number of service nodes if the current data processing volume of the target service node is greater than a preset threshold, so as to distribute the data processing tasks of the target service node to the increased service nodes.

[0084] Exemplarily, the data processing volume determination module further includes: a target data table determination module and a data processing volume determination sub-module.

[0085] The target data table determination module is used to determine the target data table connected to the target service node according to the blood relationship diagram.

[0086] The data processing capacity determination submodule is used to determine the current data processing capacity of the target service node according to the data capacity of the target data table.

[0087] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0088] The methods and apparatus of the present application can be used in a wide variety of general or specialized computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0089] For example, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0090] See also Figure 4 , Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0091] like Figure 4As shown, the computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a storage medium and an internal memory.

[0092] The storage medium can store an operating system and a computer program. The computer program includes program instructions, which, when executed, can cause the processor to perform any data processing method.

[0093] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0094] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any data processing method.

[0095] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0096] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0097] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0098] Determine the blood relationship between multiple target data based on preset blood relationship determination rules;

[0099] Generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph;

[0100] Identifying the blood relationship graph to determine a ring structure in the blood relationship graph;

[0101] According to the ring structure in the blood relationship graph, data processing tasks are performed on the multiple target data.

[0102] In one embodiment, when the processor recognizes the blood relationship graph and determines the ring structure in the blood relationship graph, it is configured to implement:

[0103] Based on a preset depth-first algorithm, the blood relationship graph is traversed to determine the ring structure in the blood relationship graph.

[0104] In one embodiment, when generating a blood relationship graph based on the blood relationship between the plurality of target data, the processor is configured to implement:

[0105] Determining sensitive data in the target data based on preset sensitive data determination rules;

[0106] Desensitizing the sensitive data based on preset desensitization rules;

[0107] The blood relationship diagram is generated based on the target data after desensitization processing.

[0108] In one embodiment, when the processor performs the data processing task on the plurality of target data according to the ring structure in the blood relationship graph, the processor is configured to implement:

[0109] According to the blood relationship graph, searching for data processing tasks related to target data corresponding to a ring structure in the blood relationship graph;

[0110] When it is determined that a data processing task is related to the target data corresponding to the ring structure in the blood relationship graph, the data processing task is terminated.

[0111] In one embodiment, when generating a blood relationship graph based on the blood relationship between the plurality of target data, the processor is configured to implement:

[0112] According to the target data type selected by the user, determining target data corresponding to the target data type from the plurality of target data as candidate target data;

[0113] Generate a kinship graph based on the kinship between candidate target data.

[0114] In one embodiment, when the processor performs the data processing task on the plurality of target data according to the ring structure in the blood relationship graph, the processor is configured to implement:

[0115] Determining a current data processing capacity of a target service node based on the blood relationship graph, wherein the target service node is a service node in a distributed system that performs data processing tasks;

[0116] If the current data processing volume of the target service node is greater than a preset threshold, the number of service nodes is increased to distribute the data processing tasks of the target service node to the increased service nodes.

[0117] In one embodiment, when determining the current data processing capacity of the target service node according to the blood relationship graph, the processor is configured to implement:

[0118] Determining a target data table connected to the target service node according to the blood relationship diagram;

[0119] The current data processing capacity of the target service node is determined according to the data capacity of the target data table.

[0120] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of data processing described above can refer to the corresponding process in the aforementioned data processing control method embodiment, and will not be repeated here.

[0121] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the data processing method of the present application.

[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0123] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0124] It should also be understood that the term "and / or" used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0125] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The data processing method includes: Based on a preset blood relationship determination rule, the blood relationship between the plurality of target data is determined according to a data processing statement, wherein the data processing statement includes: a structured query language; Generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph; Identifying the blood relationship graph to determine a ring structure in the blood relationship graph; performing data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph; The performing of data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph includes: Determining a current data processing capacity of a target service node based on the blood relationship graph, wherein the target service node is a service node in a distributed system that performs data processing tasks; If the current data processing volume of the target service node is greater than a preset threshold, increase the number of service nodes to distribute the data processing tasks of the target service node to the increased service nodes; Determining the current data processing capacity of the target service node according to the blood relationship graph includes: Determining a target data table connected to the target service node according to the blood relationship diagram; The current data processing capacity of the target service node is determined according to the data capacity of the target data table.

2. The data processing method according to claim 1, wherein: The identifying the blood relationship graph and determining the ring structure in the blood relationship graph includes: Based on a preset depth-first algorithm, the blood relationship graph is traversed to determine the ring structure in the blood relationship graph.

3. The data processing method according to claim 1, wherein: Generating a blood relationship graph according to the blood relationship among the plurality of target data includes: Determining sensitive data in the target data based on preset sensitive data determination rules; Desensitizing the sensitive data based on preset desensitization rules; The blood relationship diagram is generated based on the target data after desensitization processing.

4. The data processing method according to claim 1, wherein: The performing of data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph includes: According to the blood relationship graph, searching for data processing tasks related to target data corresponding to a ring structure in the blood relationship graph; When it is determined that a data processing task is related to the target data corresponding to the ring structure in the blood relationship graph, the data processing task is terminated.

5. The data processing method according to claim 1, wherein: Generating a blood relationship graph according to the blood relationship among the plurality of target data includes: According to the target data type selected by the user, determining target data corresponding to the target data type from the plurality of target data as candidate target data; Generate a kinship graph based on the kinship between candidate target data.

6. A data lineage analysis device, characterized in that: The data lineage analysis device comprises: A blood relationship determination module is used to determine the blood relationship between multiple target data based on a preset blood relationship determination rule and a data processing statement, wherein the data processing statement includes: structured query language; A blood relationship graph generating module, configured to generate a blood relationship graph according to the blood relationship between the plurality of target data, wherein the blood relationship graph is a directed graph; A blood relationship diagram recognition module, configured to recognize the blood relationship diagram and determine a ring structure in the blood relationship diagram; a data processing module, configured to perform data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph; The performing of data processing tasks on the plurality of target data according to the ring structure in the blood relationship graph includes: Determining a current data processing capacity of a target service node based on the blood relationship graph, wherein the target service node is a service node in a distributed system that performs data processing tasks; If the current data processing volume of the target service node is greater than a preset threshold, increase the number of service nodes to distribute the data processing tasks of the target service node to the increased service nodes; Determining the current data processing capacity of the target service node according to the blood relationship graph includes: Determining a target data table connected to the target service node according to the blood relationship diagram; The current data processing capacity of the target service node is determined according to the data capacity of the target data table.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 5 are implemented.

Images