A method, apparatus and system for generating a blood relationship

Abstract

The disclosure provides a blood relationship generation method, device and system, and relates to the technical field of big data utilization. The specific implementation of the method comprises: receiving one or more file operation logs; wherein the file operation log comprises a task identifier, an operation action, an operation time and an operation path; grouping the file operation logs according to the task identifier; for each group, cleaning the operation path in the file operation log according to the operation type of the operation action, sorting the cleaned file operation log according to the operation time, and extracting the dependency relationship between the files in the cleaned operation path from the sorting result; and generating the blood relationship of the task identifier by using the dependency relationship between the files. The implementation can reduce the labor and time cost, is suitable for various engine analysis scenarios, flexibly perceives the flow of data, supports offline and real-time blood relationship analysis, and the blood relationship is accurate and reliable.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data utilization, and particularly relates to a method, device and system for generating blood relationship. BACKGROUND

[0002] The blood relationship can represent the flow process of data, and by analyzing the blood relationship between data, it is helpful for a user to discover data abnormalities in time and repair, and improve the accuracy and management efficiency of data.

[0003] In the existing blood relationship analysis process, a commonly used blood relationship extraction tool usually depends on a directed graph, metadata, relational statement analysis, etc., displays a graphical interface of data flow, or stores metadata information of data, or disassembles the logic of a relational statement to infer the blood relationship.

[0004] However, with the explosive growth of data and the diversification of scenarios, the human cost of user labeling is increasingly high, and the graphical interface drawing is time-consuming and slow in response, resulting in a poor user experience; the management of metadata requires a large amount of human and time cost, and once there is a local fault or data loss, the entire system management will be chaotic and collapse, and the query is time-consuming, also resulting in poor user satisfaction; the analysis of relational statements not only needs to modify each analysis engine, which is high in development cost and has uneven accuracy of data recognition, and cannot construct real blood relationship for unstructured and complex data such as logs, resulting in low utilization of big data and poor flexibility. SUMMARY

[0005] Therefore, the embodiments of the present disclosure provide a method, device and system for generating blood relationship, which can solve the problems of long time consumption, poor user experience, high human and time cost, chaotic management and collapse of blood relationship due to fault or data loss, time-consuming query, poor user satisfaction, high development cost, uneven accuracy, and complex blood relationship scenarios of unstructured data, resulting in low utilization of big data and poor flexibility.

[0006] To achieve the above object, according to an aspect of the present disclosure, a method for generating blood relationship is provided, comprising:

[0007] The method comprises:

[0008] receiving one or more file operation logs; wherein the file operation logs comprise task identification, operation action, operation time and operation path;

[0009] grouping the file operation logs according to the task identification;

[0010] cleaning the operation path in the file operation log according to the operation type of the operation action for each of the groups, sorting the cleaned file operation log according to the operation time, and extracting the dependency relationship between the files in the cleaned operation path from the sorting result;

[0011] generating the blood relationship of the task identifier by using the dependency relationship between the files.

[0012] According to another aspect of the present disclosure, a blood relationship generation apparatus is provided, comprising:

[0013] a receiving module configured to receive one or more file operation logs; wherein the file operation log comprises a task identifier, an operation action, an operation time and an operation path;

[0014] a grouping module configured to group the file operation log according to the task identifier;

[0015] a cleaning module configured to clean the operation path in the file operation log according to the operation type of the operation action for each of the groups, sort the cleaned file operation log according to the operation time, and extract the dependency relationship between the files in the cleaned operation path from the sorting result;

[0016] a generating module configured to generate the blood relationship of the task identifier by using the dependency relationship between the files.

[0017] According to still another aspect of the present disclosure, a blood relationship generation system is provided, comprising a blood relationship generation apparatus and a plurality of computing engines, wherein:

[0018] the generation apparatus receives one or more file operation logs generated by the plurality of computing engines, and generates the blood relationship of the task identifier according to the file operation log.

[0019] According to still another aspect of the present disclosure, an electronic device is provided, comprising:

[0020] a processor; and

[0021] a memory storing a program,

[0022] wherein the program comprises instructions which, when executed by the processor, cause the processor to perform the blood relationship generation method.

[0023] According to still another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the blood relationship generation method.

[0024] One or more technical solutions provided in the embodiments of the present application can realize the following technical effects. The non-institutionalized text operation log is cleaned according to the operation type, the blood relationship in the cleaned file operation log is generated after being sorted according to the operation time, the user labeling is not required, the generation efficiency is greatly improved, the human and time costs are reduced, the analysis scene of various engines is suitable, the data flow is flexibly perceived, the offline and real-time blood relationship analysis is supported, and the blood relationship is accurate and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0025] In the following description of the example embodiments in conjunction with the accompanying drawings, more details, features and advantages of the present disclosure are disclosed, in the accompanying drawings:

[0026] Figure 1 A flowchart of a blood relationship generation method according to an example embodiment of the present disclosure is shown;

[0027] Figure 2 A flowchart of a file operation log generation method according to an example embodiment of the present disclosure is shown;

[0028] Figure 3 A schematic diagram of a blood relationship generation system according to an example embodiment of the present disclosure is shown;

[0029] Figure 4 A flowchart of a file operation log cleaning method according to an example embodiment of the present disclosure is shown;

[0030] Figure 5 A flowchart of a file dependency relationship extraction method according to an example embodiment of the present disclosure is shown;

[0031] Figure 6 A flowchart of a business identification blood relationship extraction method according to an example embodiment of the present disclosure is shown;

[0032] Figure 7 A schematic block diagram of a blood relationship generation device according to an example embodiment of the present disclosure is shown;

[0033] Figure 8 A structural block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0035] It should be understood that each step recited in the method embodiments of the present disclosure can be performed in different order, and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect.

[0036] The term "comprising" and variations thereof as used herein are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment". The term "another embodiment" means "at least one additional embodiment". Related terms are defined as follows. It should be noted that the concepts mentioned in the present disclosure are merely used for distinguishing different apparatuses, modules or units, and are not intended to limit the functions of the apparatuses, modules or units.

[0037] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0038] The names of the messages or information exchanged between the plurality of apparatuses in the embodiments of the present disclosure are only used for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0039] Big Data Computing Engine: Big Data Computing Engine refers to computer software or hardware systems specifically designed for processing large-scale data, which can quickly process, analyze and mine massive data to obtain valuable information.

[0040] SQL: Structured Query Language, which is a programming language for interacting with and managing relational database systems, supports various database operations such as inserting, updating, deleting and querying data statements.

[0041] Hive: is a data warehouse tool based on Hadoop distributed architecture, which is used for data extraction, transformation, loading (Extract-Transform-Load, ETL) and other processing.

[0042] Spark: is a fast, general, and scalable big data processing framework that provides efficient data processing and analysis capabilities, supporting large-scale data processing, machine learning, graph computing, and streaming processing tasks.

[0043] Presto is an open-source distributed SQL query engine for big data, suitable for interactive analytical queries, supporting numerous data sources, and providing interface development for data source connectors.

[0044] With the advent of the big data era, data volume is exploding. To understand the origins and relationships of data, traditional graphical interfaces, metadata, and relational statement analysis are increasingly unable to handle complex scenarios such as unstructured data, failing to meet the diverse needs for obtaining these relationships. On one hand, graphical interface tools construct directed acyclic graphs (DAGs) in the backend, with each node representing the analysis target and edges representing dependencies. Responding to user configuration commands can display the flow of data, allowing users to infer relationships. However, with the surge in data volume, response times are too long, drawing and querying are time-consuming, resulting in poor performance and a poor user experience. On the other hand… On the one hand, metadata tools record the definition, change time, and hierarchy of data. By querying metadata, the changes in lineage can be inferred. However, with the explosive growth of data volume, a complete metadata management system requires a huge amount of manpower and time. Once it fails or is lost, the system is at risk of collapse, and the lineage cannot be deduced. On the other hand, relational statement analysis tools deduce the data processing process by deconstructing statement syntax, thereby obtaining the data lineage sequentially. However, different processing engines have different rules, and the syntax analysis varies greatly, resulting in inconsistent recognition accuracy. This leads to low accuracy in lineage relationships and makes it impossible to analyze complex scenarios of unstructured data.

[0045] The lineage generation method disclosed herein extracts the lineage of objects based on the analysis and utilization of object read and write records in log data. It is time-efficient and highly effective, which can greatly improve user satisfaction. It is adaptable to the generation and query of lineage in various scenarios such as relational and non-relational relationships. At the same time, since it tracks the dynamic changes in the logs and timely perceives the actual behavior of the data, it can flexibly and accurately capture lineage, avoiding a large amount of manpower, time and development costs. The generated lineage is accurate, reliable and complete.

[0046] The present disclosure is described below with reference to the accompanying drawings.

[0047] Figure 1 A flowchart illustrating a method for generating blood relations according to exemplary embodiments of the present disclosure is shown, such as... Figure 1 As shown, the method for generating blood relations disclosed herein includes the following steps:

[0048] Step S101: Receive one or more file operation logs; wherein the file operation logs include task identifier, operation action, operation time, and operation path.

[0049] In the embodiments of the present disclosure, the blood relationship generation method of the present disclosure is executed by a blood server. The blood server receives file operation logs reported by a unified file interface. The unified file interface is connected to various computing engines. The unified file interface is pre-configured with collection standards adapted to the various computing engines. The unified file interface collects operation information related to file operations in the process of executing a business request by the various computing engines according to the collection standards. File operation logs of the various computing engines are generated and reported.

[0050] Further, the file operation log includes a task identifier, an operation action, an operation time, an operation path, and the like. The task identifier is a request identifier of the business request. For example, the task identifier is "score". The task identifiers of multiple file operation logs of the same business request are the same. The operation action includes obtaining, creating, moving, and the like. For example, the operation action is "create" (i.e., creating) or "rename" (i.e., moving). The operation time is a start time of the operation action. For example, the operation time is 0701. The operation path is a file path of a file corresponding to the operation action. The operation path is composed of a task identifier, a step identifier, and / or a code block object identifier, a file identifier, and the like.

[0051] Further, the unified file interface can also report task information logs of the various computing engines. The task information log includes a task name, an engine identifier, a task state, and computing resource consumption, and the like.

[0052] In the embodiments of the present disclosure, as shown in Figure 2 The file operation log generation method of the present disclosure includes the following steps:

[0053] In the embodiments of the present disclosure, the file operation log generation method of the present disclosure is executed by the various computing engines.

[0054] Step S201, in response to the business request, the business request is decomposed into multiple business tasks.

[0055] In the embodiments of the present disclosure, the business request includes an engine identifier and a request task. The request end sends the business request to the computing engine corresponding to the engine identifier. The various computing engines decompose the request task into multiple small business tasks to reasonably allocate computing resources and improve task processing efficiency. For example, the business request is to score one or more users. The decomposed business tasks include a feature division task, a behavior division task, a feedback division task, and a summation task.

[0056] Step S202, respectively, each of the business tasks is allocated a computing node.

[0057] In this embodiment, the computing engine is a distributed engine, comprising multiple computing nodes for executing various types of business tasks. The computing engine matches corresponding computing nodes based on the number of business tasks decomposed from the business request, assigning matching nodes to each business task. For example, computing nodes are matched separately for feature-based tasks, behavior-based tasks, feedback-based tasks, and summation tasks.

[0058] Step S203: Based on the configuration information of the file interface, collect the execution actions and results of each computing node executing the business task; wherein, the execution actions are obtained by executing statements or code block objects.

[0059] In this embodiment of the disclosure, when each matched computing node executes a business task, the computing engine collects the execution actions and results of each computing node based on the pre-acquired file interface configuration information. Execution actions are operations such as acquisition, creation, or movement derived from the execution statements or code block objects corresponding to each task step when each computing node executes each task step. Execution results include the files acquired, created, or moved by each execution action. For example, the task steps of a feature scoring task include reading information such as the user's ID number, contact information, emergency contact information, address, workplace, and monthly transaction amount from the user information file, calculating the user's feature score according to preset weights, and the execution statements or code block objects that the computing node needs to execute include reading the user information file, creating a feature score file in a temporary path to store the calculated feature score, and moving the feature score file to a fixed path. Correspondingly, the execution actions include reading, creating, and moving, and the execution results include the feature score file in the temporary path and the feature score file in the fixed path.

[0060] Furthermore, such as Figure 3 As shown, the lineage generation system disclosed herein includes a lineage analyzer, a unified file interface, and multiple computing engines. Each computing engine includes a task management component and a task computing component. The task management component decomposes each business request and distributes the resulting business tasks to the computing nodes of the task computing component. While processing the business tasks, the computing nodes acquire files and create temporary paths to store them. After the task management module determines that all computing nodes have completed processing the business request, each computing node moves the files stored in the temporary paths to a fixed path. For example, the task management component could be HiveServer2 in the Hive engine or driver in the Spark engine, and the task computing component could be MR in the Hive engine or executer in the Spark engine.

[0061] Furthermore, the operation of moving files can be performed by the task management component.

[0062] Step S204, taking each of the execution actions as an operation action, determining an operation time of the operation action according to an execution time of the execution action.

[0063] In the embodiments of the present disclosure, the execution time is the start time of the execution action, and correspondingly, the operation time is the start time of the operation action. It should be noted that the operation time is set to facilitate more intuitive, clear and explicit blood relationship, and therefore can be selectively set according to actual needs, and can be globally uniform. For example, the execution time can also be the end time of the execution action, and correspondingly, the operation time is the end time of the operation action.

[0064] Step S205, determining the operation path from the execution action and the execution result, the operation path including the task identifier, the node identifier of the computing node, the step identifier of the step, or the object identifier and the file identifier of the code block object.

[0065] In the embodiments of the present disclosure, the operation path represents a plurality of information, which can be composed of a task identifier, a node identifier, a step identifier, an operation time, and a file identifier; or, can be composed of a task identifier, a node identifier, a code block identifier, an operation time, and a file identifier. For example, the node identifier is a combination of engine identifier and node number, such as hive1, spark2, presto3, …, the step identifier is s1, s2, s3, …, the code block identifier is b1, b2, b3, …, and the file identifier is 000000-0, 000000-1, 001111-0, 001111-2, …, and correspondingly, the operation path can be score / dt=0701 / hive1 / s1 / 001111-2, score / dt=0701 / spark5 / b1 / 000000-1, etc.

[0066] Further, the operation path of the temporary storage file usually adds a temporary identifier, which can be a combination of a special character and an engine identifier, such as a combination of an underscore and an engine identifier “_hive”, and correspondingly, the temporary path can be score / dt=0701215608 / hive1 / s1 / _hive / 000000-0; or, the temporary identifier can be selectively set according to actual needs, such as “_temporary”, and correspondingly, the temporary path can be score / dt=0701 / hive1 / s1 / _temporary.

[0067] Step S206, combining the task identifier, the operation action, the operation time and the operation path into a file operation log, and sending it to the file interface, so that the file interface reports the file operation log.

[0068] In this embodiment, each computing engine combines the task identifier, operation action, operation time, and operation path of each file generated during the execution of business tasks by each computing node into a file operation log. This log is then sent to a unified file interface, which in turn reports it to the lineage server. Upon receiving the file operation log, the lineage server analyzes the lineage relationships between files and tasks. The file operation log is in text format, and one log entry corresponds to one operation action of a computing node.

[0069] Furthermore, file operation logs can also include information such as the IP address and port of each computing node.

[0070] In this embodiment of the disclosure, the task information log is generated by each computing engine during the processing of business tasks and sent to a unified file interface for reporting.

[0071] In this embodiment of the disclosure, through the file operation log generation method of the disclosure, each computing engine collects various information such as actions, time, and file paths according to the pre-configuration of a unified file interface to generate file operation logs, which are then sent to the unified file interface and reported to the lineage server by the file interface. This facilitates the subsequent generation and analysis of lineage relationships, can adapt to various analysis scenarios, improves the flexibility of lineage relationship generation, and enhances the utilization rate of big data.

[0072] Step S102: Group the file operation logs according to the task identifier and the node identifier in the operation path.

[0073] In this embodiment, the lineage server groups file operation logs according to task identifiers, grouping multiple file operation logs belonging to the same business task into one group. Since business requests are typically allocated to various computing engines according to business type, the grouping by task identifiers is essentially also the grouping by computing engine.

[0074] Furthermore, the lineage server divides the file operation logs belonging to the same group into multiple groups according to the node identifiers in the operation path.

[0075] Step S103: For each group, clean the operation path in the file operation log according to the operation type of the operation action, sort the cleaned file operation log according to the operation time, and extract the dependency relationship between file identifiers in the cleaned operation path from the sorting result.

[0076] In the embodiments of the present disclosure, the file operation logs of each group are sorted according to operation time, and are cleaned according to operation types of operation actions, whether each operation action is abnormal is judged, and the file operation logs with abnormalities are deleted, and the dependent relationship between files is obtained by using the cleaned and accurate file operation logs.

[0077] In the embodiments of the present disclosure, as shown in Figure 4 The cleaning method of the file operation log of the present disclosure includes the following steps:

[0078] Step S401, for each group of the groups, the plurality of file operation logs in the group are sorted according to operation time.

[0079] In the embodiments of the present disclosure, each group is composed of a plurality of groups, in order to facilitate the judgment of abnormal write file operation, the file operation logs in each group need to be sorted first. Therefore, for each group of business tasks, or for each group of computing nodes, the file operation logs in the group under the same node identifier are sorted according to operation time for the first time.

[0080] Step S402, the operation path and operation action in each file operation log are obtained.

[0081] Step S403, whether the operation type of the operation action is read file is judged, if yes, go to step S404; if no, go to step S406.

[0082] In the embodiments of the present disclosure, the operation type includes read file and write file, the operation action of read file is acquisition, and the operation action of write file includes creation and movement.

[0083] Step S404, whether the path type of the operation path is business path is judged, if yes, go to step S405; if no, go to step S410.

[0084] In the embodiments of the present disclosure, the path type includes a business path and a code path, the business path stores files of various business tasks, and the code path stores code blocks for performing various standard actions. The code path is usually pre-configured by each computing engine, and there is a specific path label. Since the blood relationship disclosed by the present disclosure is the blood relationship of files and the blood relationship between business tasks, the log of the file needs to be read for analysis, rather than reading code data for analysis, and the code block is not the analysis target of the present disclosure, so the file operation log of reading code data needs to be excluded. Therefore, in the case of reading a file, the operation path is compared with the specific path label to determine whether the operation path is a business path, so as to exclude the file operation log of the code path. For example, the specific path label can be a language label of the engine language used by the computing engine.

[0085] In step S405, the file identifier in the operation path is deleted to obtain the cleaned file operation log.

[0086] In the embodiments of the present disclosure, the blood relationship disclosed by the present disclosure is extracted between files and business tasks according to the actions of reading and writing files when the blood relationship is generated. Therefore, only the business meaning represented by the path to which the reading file belongs needs to be focused, and the specific number of files under the path will not substantially change the blood relationship. For example, the operation path under the feature division task can include user information files of multiple users, and the blood relationship disclosed by the present disclosure does not care how many user information files are included in the operation path under the feature division task. Only the action that the file operation log comes from the user information file read by the feature division node is needed to be determined.

[0087] Therefore, when the file operation log of the reading file of the business path is cleaned, the file identifier of the operation path in the file operation log is deleted. For example, the operation path of the reading file is / home / test.db / test / dt=0701 / 000000-0, and the file identifier 000000-0 in the operation path is directly deleted to obtain the cleaned target path / home / test.db / test / dt=0701 / .

[0088] In step S406, it is determined whether there is a moving action after each creating action. If yes, go to step S407; if no, go to step S408.

[0089] In the embodiments of the present disclosure, as Figure 3In the working principle of the blood relationship analysis system shown, the creation of the temporary path storage file and the moving of the file stored in the temporary path to the fixed path are usually continuous. Only when the creation and the moving are continuous, it is indicated that the business task is running normally. Otherwise, if the file is not moved to the fixed path after the creation, it is indicated that the file is not moved to the unified output path, that is, the corresponding business task is abnormal, which will inevitably cause a large difference in the response result of the business request and a lack of data consistency. Therefore, in the case of the write file operation type, the data consistency after the creation needs to be judged to clean the abnormal business data and standardize the blood relationship. Then, after the blood relationship is generated, the abnormality can be located and repaired in time.

[0090] Further, in the case of the creation operation, whether the moving operation exists after the creation is judged to clean the abnormal file operation log.

[0091] In step S407, the operation path with the temporary identifier is deleted, and the cleaned file operation log is obtained.

[0092] In the embodiments of the present disclosure, the blood relationship of the present disclosure is generated according to the blood relationship between the file and the business task extracted from the read file and the write file operation. Therefore, only the business meaning represented by the path of the final result of the write file needs to be focused on. The file in the intermediate temporary path will not substantially change the blood relationship. Therefore, the present disclosure does not care about the change of the intermediate temporary path, and only the result file of each operation of the computing node needs to be determined.

[0093] Therefore, in the case of the moving operation or the moving operation after the creation operation, the file operation log of the write file of the business path is cleaned, and the operation path with the temporary identifier in the file operation log is deleted. For example, the operation and the operation path of the create operation and the moving operation in the file operation log are as follows:

[0094] create, / home / test.db / test / dt=0701 / _temporary / app-1234 / 000000-0;

[0095] rename, / home / test.db / test / dt=0701 / _temporary / app-1234 / 000000-0, / home / test.db / test / dt-0701 / 000000-0;

[0096] The operation path with the temporary identifier_temporary is deleted, and the target path after cleaning is obtained:

[0097] rename, / home / test.db / test / dt-07-01 / 000000-0.

[0098] Step S408, determine that the file operation log of the creation action is abnormal.

[0099] In the embodiment of the present disclosure, in the case that there is no moving action after the creation action, it is indicated that the write file operation is abnormal, that is, the file operation log corresponding to the write file is abnormal.

[0100] Step S409, delete the abnormal file operation log.

[0101] Step S410, delete the file operation log of the code path.

[0102] In the embodiment of the present disclosure, the file operation log of the code path is irrelevant to the generation of the blood relationship, so it can be directly deleted without affecting the analysis of the blood relationship.

[0103] In the embodiment of the present disclosure, by the cleaning method of the file operation log of the present disclosure, different cleaning methods are performed on the file operation log according to the operation type, the file identifier of the read file is deleted, the sequence of the write file is judged to locate the abnormal log and delete it, and the temporary path is deleted at the same time, so that the cleaned file operation log is obtained, the read-write relationship of the entire file operation log is accurate, simple, clear and clear, the source and flow direction of the file can be more accurately grasped, and the quality and reliability of the blood relationship analysis data are ensured, the generation efficiency and accuracy of the blood relationship are improved, and the blood relationship of the data is better monitored and managed.

[0104] In the embodiment of the present disclosure, as shown in Figure 5 The extraction method of the file dependency relationship of the present disclosure includes the following steps:

[0105] Step S501, for each of the small groups, the cleaned file operation log is sorted according to the operation time.

[0106] In the embodiment of the present disclosure, the purpose of judging the sequence of the write file action in the first sorting is different, because the cleaning of the file operation log is accompanied by the deletion of the abnormal file operation log, the deletion of the read file for the file operation log of the temporary path, etc., so it is necessary to re-sort the cleaned file operation log, so as to achieve the purpose of accurately analyzing the blood relationship through the sorting result of the second sorting.

[0107] Step S502, find the file operation log whose operation type is write file, and go to step S503 or step S504.

[0108] Step S503: According to the file identifier of each write file, and the path meaning of the operation path of one or more read files between the write file and the previous write file, the file dependency is extracted.

[0109] In the embodiments of the present disclosure, after the secondary sorting, within the same group, according to the file identifier of each write file, and the path meaning of the operation path of one or more read files between the write file and the previous write file, the file dependency in the business task is extracted. For example, the file identifier of the write file is 000000-1, the 000000-1 file is a feature split file, and the path meaning of the operation path of the read file between the write file and the previous write file is the user information file of 0701. The obtained file dependency is:

[0110] The feature split file depends on the user information file of 0701.

[0111] Further, if the write file is the first write file, the file dependency can be directly extracted according to the write file and the path meaning of the operation path of one or more read files before the write file.

[0112] Step S504: According to the file identifier of the plurality of write files, the file dependency is extracted.

[0113] In the embodiments of the present disclosure, after the secondary sorting, within the same group, according to the file identifier and the file sequence of the plurality of write files, the file dependency in the business task is extracted. For example, before the write D file, there are read***, write B file, and read***. It is considered that the D file depends on the B file.

[0114] Further, in the business task, the file dependency is extracted according to the path meaning of the operation path of the read file and the file identifier of the write file. For example, the user behavior split file depends on the user repayment behavior file of 0101-0630, the user repayment behavior file depends on the user repayment plan file, and the user repayment plan file is the calculation result of the user loan file and the user repayment period.

[0115] In the embodiments of the present disclosure, by using the file dependency extraction method of the present disclosure, the sorted file operation log after cleaning is sorted, the file dependency in the business task is extracted according to the read-write relationship of the read file and the write file, and the file dependency is the file blood relationship in the business task. Further, the blood relationship of the business request can be extracted according to the file dependency between the plurality of business tasks.

[0116] Step S104: The blood relationship of the task identifier is generated by using the dependency relationship between the file identifiers.

[0117] In the embodiments of the present disclosure, since the business request is decomposed into multiple business tasks for execution, in order to obtain the blood relationship of the business request, the blood relationship corresponding to the task identifier is generated according to the file dependency relationship between the read file and the write file between the business tasks.

[0118] In the embodiments of the present disclosure, as shown in Figure 6 the method for extracting the blood relationship of the business identifier of the present disclosure includes the following steps:

[0119] In step S601, the file dependency relationship of each node identifier corresponding to the task identifier is obtained.

[0120] In step S602, the operation path of the last file dependency relationship of each node identifier is matched with the operation path of the first read file of the first file dependency relationship, and the identifier dependency relationship between each node identifier is connected.

[0121] In the embodiments of the present disclosure, the file dependency relationship of different business tasks under the same task identifier is associated by the blood relationship server, and the blood relationship of the task identifier is constructed therefrom. The blood relationship of the task identifier is actually the blood relationship between the business tasks, that is, the blood relationship between the node identifiers. The blood relationship between the node identifiers can be connected through the first read file and the last write file of the node identifier. Therefore, the operation path is matched, the matched operation path is used as a positioning reference, the identifier dependency relationship between the node identifiers is connected, and then the blood relationship of the task identifier, that is, the blood relationship of the business request, is extracted.

[0122] Further, in the case that the operation path of the last file dependency relationship of one node identifier includes the operation path of the first read file of the first file dependency relationship of another node identifier, it is determined that the operation paths of the two node identifiers are matched.

[0123] In step S603, the blood relationship of the task identifier is extracted from the identifier dependency relationship.

[0124] For example, task one reads E file and writes F file, task two reads G file and writes H file, and task three reads F file and H file and writes K. It is determined that task three depends on task one and task two at the same time, and the tasks one, two and three have blood relationship.

[0125] In the embodiments of the present disclosure, or the blood relationship of the business request can also be obtained through the order of the file dependency relationship. Specifically:

[0126] Each file dependency relationship is sorted three times according to the operation time, and the file dependency relationship is concatenated into the blood relationship of the business identifier according to the sorting result of the three times.

[0127] In the embodiments of the present disclosure, the blood relationship extraction method of the service request can analyze the blood relationship of the service request corresponding to each service task based on the file dependency relationship extracted from each service task, thereby improving the accuracy and stability of the blood relationship.

[0128] In the embodiments of the present disclosure, the blood relationship can be displayed to master the change trend of the data. Moreover, since the display of the blood relationship is a kind of backstepping of the processing process of the actual business data in the actual application scenario, the same blood relationship of different users can be compared, and the abnormal data can be located according to the loss therein. For example, the user score is known, the file operation log of the generation process of the entire user score is analyzed to generate the blood relationship, it can be known from the blood relationship of the user score that the user score is synthesized from the user characteristics, user behavior and the third feedback, and the reason for the abnormal user score is determined.

[0129] In the embodiments of the present disclosure, the blood relationship generation method can improve the generation efficiency of the blood relationship, shorten the display and query time of the blood relationship, improve the user experience, and local faults or data loss will not affect the generation process of the blood relationship, thereby ensuring the stability and accuracy of the blood relationship generation, reducing the labor and time cost, and positioning the abnormal data through the display of the blood relationship, thereby ensuring the stability of the blood management system, without the modification cost of the engine. The blood relationship generated based on the unstructured data is accurate and can adapt to various analysis scenarios, thereby fully improving the utilization rate of big data and improving the flexibility of data use.

[0130] Figure 7 is a schematic diagram of main modules of the blood relationship generation device according to the embodiments of the present disclosure, as shown in Figure 7 The blood relationship generation device 700 of the present disclosure includes:

[0131] The receiving module 701 is configured to receive one or more file operation logs; wherein the file operation log includes a task identifier, an operation action, an operation time and an operation path.

[0132] The grouping module 702 is configured to group the file operation logs according to the task identifier.

[0133] The cleaning module 703 is configured to clean the operation path in the file operation log according to the operation type of the operation action for each group, sort the cleaned file operation log according to the operation time, and extract the dependency relationship between the files in the cleaned operation path from the sorting result.

[0134] The generating module 704 is configured to generate the blood relationship of the task identifiers according to the dependency relationship between the files.

[0135] The electronic device according to the embodiments of the present disclosure can be configured to execute the method according to the embodiments of the present disclosure.

[0136] The computer program product according to the embodiments of the present disclosure can be configured to cause the computer to execute the method according to the embodiments of the present disclosure.

[0137] The computer program product according to the embodiments of the present disclosure can be configured to cause the computer to execute the method according to the embodiments of the present disclosure.

[0138] Reference Figure 8 A block diagram of an electronic device 800, which can be used for the server or client of the present disclosure, will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computing devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device can also represent various forms of mobile devices such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0139] As shown in Figure 8 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded into a random access memory (RAM) 803 from a storage unit 808. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0140] The plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 800, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 807 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 804 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0141] The computing unit 801 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above. For example, in some embodiments, Figures 1-2 ,The methods of Figures 4-6 can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage unit 808. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 can be configured, by way of firmware or software, to execute the methods of Figures 4-6 . Figures 1-2 ,The methods of Figures 4-6 can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage unit 808. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 can be configured, by way of firmware or software, to execute the methods of Figures 4-6 .

[0142] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be supplied to a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the machine, produces a means for implementing the functions / operations specified in the flowchart diagrams and / or the block diagrams. The program code can be implemented in a global computer system, a distributed computer system, and / or various other modularized computer systems. ​​

[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] As used in the present disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or data to a programmable processor.

[0145] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0146] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A method for generating blood relations, characterized in that, The generation method includes: Receive one or more file operation logs; wherein the file operation logs include task identifier, operation action, operation time, and operation path; The file operation logs are grouped according to the task identifier; For each group, the operation paths in the file operation log are cleaned according to the operation type of the operation action, the cleaned file operation logs are sorted according to the operation time, and the dependencies between files in the cleaned operation paths are extracted from the sorting results. By utilizing the dependencies between the files, the lineage of the task identifiers is generated; The task identifier corresponds to multiple node identifiers, and each node identifier represents a group; for each group, the operation path in the file operation log is cleaned according to the operation type of the operation action, including: Sort the multiple file operation logs within each group according to the operation time, and obtain the operation path and operation action in each file operation log; Determine whether the operation type of the operation is reading a file; if the operation type is reading a file, determine the path type of the operation path; wherein, the operation type includes reading a file and writing a file; Delete the file identifier from the operation path of the file read operation path whose path type is business path to obtain the cleaned file operation log.

2. The generation method as described in claim 1, characterized in that, The file writing operation includes creation and movement actions; when the operation type of the operation is file writing, it also includes: Determine whether there is a move action after each creation action in the file operation log; If a move action occurs after the creation action, delete the operation path with temporary identifiers in the file being written, and obtain a cleaned file operation log.

3. The generation method as described in claim 1 or 2, characterized in that, If no move action occurs after the creation action, the file operation log corresponding to the creation action is determined to be abnormal. Delete the abnormal file operation logs; or, If the path type of the operation path is a code path, delete the file operation log of the code path.

4. The generation method as described in claim 1, characterized in that, The step of sorting the cleaned file operation logs according to the operation time and extracting the dependencies between files in the cleaned operation paths from the sorting results includes: For each of the aforementioned groups, the cleaned file operation logs are sorted a second time according to the operation time. The file operation log to be searched is a file operation log for writing files; The file dependencies are extracted based on the file identifiers of each of the written files and the path meaning of the operation paths of one or more read files between the written file and the previous written file; alternatively, the file dependencies are extracted based on the file identifiers of multiple written files.

5. The generation method as described in claim 1, characterized in that, The step of generating the lineage of the task identifier by utilizing the dependencies between the files includes: Find the same file identifiers among the file dependencies; The lineage of the task identifiers is generated based on the flow process of each of the same file identifiers.

6. A device for generating blood relations, characterized in that, The generating apparatus includes: A receiving module is used to receive one or more file operation logs; wherein the file operation logs include task identifier, operation action, operation time, and operation path; The grouping module is used to group the file operation logs according to the task identifier; The cleaning module is used to clean the operation paths in the file operation log for each group according to the operation type of the operation action, sort the cleaned file operation log according to the operation time, and extract the dependency relationships between files in the cleaned operation paths from the sorting results. The task identifier corresponds to multiple node identifiers, and each node identifier represents a group. When cleaning the operation paths in the file operation logs according to the operation type of the operation action for each group, the cleaning module is specifically used to sort the multiple file operation logs within each group by operation time, obtain the operation paths and operation actions in each file operation log; determine whether the operation type of the operation action is "read file"; if the operation type is "read file", determine the path type of the operation path; wherein, the operation type includes "read file" and "write file"; delete the file identifiers in the operation paths of "read file" with the path type being a business path, and obtain the cleaned file operation logs. The generation module is used to generate the lineage of the task identifier by utilizing the dependencies between the files.

7. A system for generating blood relations, characterized in that, include: The blood relation generation device and multiple computing engines as described in claim 6, wherein: The generation device receives one or more file operation logs generated by multiple computing engines, and generates the lineage of the task identifier based on the file operation logs.

8. The generation system as described in claim 7, characterized in that, The computing engine generates the file operation log, including: In response to a business request, the business request is decomposed into multiple business tasks, and computing nodes are allocated to each of the business tasks. Collect the execution actions and results of each computing node in executing the business task; wherein, the execution actions are obtained from execution statements or code block objects; Each of the execution actions is taken as an operation action. Based on the execution time of the execution action, the operation time of the operation action is determined. The operation path is determined from the execution action and the execution result. The operation path includes the task identifier, the node identifier of the computing node, or the object identifier and file identifier of the code block object. The task identifier, the operation action, the operation time, and the operation path are combined to form a file operation log.

9. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method for generating blood relations according to any one of claims 1-5.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method for generating blood relations according to any one of claims 1-5.

Images