Test data lineage management method based on knowledge graph and evolution pedigree

Abstract

The invention provides a test data lineage management method based on a knowledge graph and an evolution lineage. The method comprises the following steps: tracing current test data, and generating a mapping relation set of the test data according to a tracing result; initially generating data related to the test data based on a data source, each piece of intermediate data, the current test data and a mapping relation set, and generating a plurality of lt; the first data, the mapping relation and the second data gt; a triad in a form; based on all the triads, a knowledge graph is generated, first metadata corresponding to the first data and second metadata corresponding to the second data serve as entity nodes of the knowledge graph, and the mapping relation in the triads serves as the relation in the knowledge graph. According to the method, an evolution pedigree of data is constructed in a knowledge graph mode, and lineage analysis is carried out on the data in the combined test; and different versions of the same data are stored as the data chain table, so that data traceability is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method for managing test data lineage based on a knowledge graph and an evolutionary pedigree. Background Art

[0002] Data lineage describes the mechanism and process of data generation and evolution, which is of great significance to data quality assessment, data recovery and data analysis. This technology introduces the concept of Input and Output Degree Sequence with Scale i (IO-iD) model, which not only describes the node degree distribution of lineage workflow, but also takes into account the extraction of workflow direction characteristics; the Previous-Next temporal sequence structure is used to describe the substructure characteristics of nodes and their adjacent nodes in the workflow; the privacy-preserving lineage workflow publishing algorithm based on differential privacy, DpriPP, realizes the publishing of privacy-preserving lineage workflows with weak background knowledge dependence and the effective maintenance of the availability of workflow temporal dependency.

[0003] The current lineage management and analysis technology basically adopts the method based on the mapping relationship database. The mapping relationship of the lineage is stored in the lineage relationship database. When querying, the corresponding data location is found according to the mapping relationship recorded in the relationship database, and the result is obtained and returned in the corresponding database. Figure 1 The disadvantage of the prior art is that the efficiency of operations such as query and modification is relatively low, especially when the amount of data is huge, each time a certain mapping relationship is queried, the entire database needs to be traversed.

[0004] In military joint tests, the source of test data is clear and highly traceable, and test data with uncertain sources rarely appear. However, the throughput of test data is large and the items are numerous, and existing technologies cannot meet its requirements. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a method for test data lineage management based on knowledge graph and evolutionary genealogy, which is used to solve the technical problem that when the data volume is huge, the entire database needs to be traversed each time a certain mapping relationship is queried.

[0006] The present invention proposes a method for managing test data lineage based on knowledge graph and evolutionary lineage, the method comprising:

[0007] Step S1: trace the source of the current test data, and generate a mapping relationship set of the test data according to the traceability result. The test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data generates the current test data through conversion or operation; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships and black box relationships; based on the data source initially generating data related to the test data, each intermediate data, the current test data and the mapping relationship set, generate multiple triples in the form of <first data, mapping relationship, second data>; the second data is generated by the first data through the operation or conversion represented by the mapping relationship in the triple, the distribution relationship means that after conversion or operation, the second data is a subset of the first data, the aggregation relationship means that after conversion or operation, the first data is a true subset of the second data, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship;

[0008] Step S2: Generate a knowledge graph based on all triples, in which the first metadata corresponding to the first data and the second metadata corresponding to the second data are both used as entity nodes of the knowledge graph, and the mapping relationship in the triples is used as the relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; based on the knowledge graph, a link is established between each test data stored in the memory and the current test data.

[0009] Preferably, the repository has an index table, which indexes the metadata and memory storage location corresponding to each test data; the metadata includes the data identifier of the test data, a timestamp, all mapping relationships related to the data corresponding to the metadata, and the location of the metadata in the knowledge graph, and the timestamp is used to record the collection or generation time of the test data.

[0010] Preferably, a linked list is established for each entity node. When the test data is modified, the modification time of the test data is used as a timestamp, and various versions of the test data are stored in the linked list in sequence according to the timestamp; the modification refers to: the data source that generates the test data corresponding to the entity node generates new test data over time, and the test object corresponding to the new test data is the same as the test object corresponding to the test data; when a modification occurs, the new test data is added to the test data corresponding to the entity node and stored in the linked list.

[0011] Preferably, the knowledge graph is traversed regularly, entity nodes in the knowledge graph that have no mapping relationship with any other entity nodes are regarded as isolated entity nodes, and the isolated entity nodes are marked. When the number of markings exceeds a preset number, the isolated entity nodes and the test data of each version corresponding to the isolated entity nodes are deleted.

[0012] Preferably, when locating the problem, for the knowledge graph, the entity nodes that have been analyzed are marked as analyzed nodes; the entity nodes to be analyzed are obtained, a list is generated for the entity nodes to be analyzed, and the child nodes of the entity nodes to be analyzed in the knowledge graph, the parent nodes in the knowledge graph, and each data information in the linked list corresponding to the entity nodes to be analyzed are stored in the list, and each entity node stored in the list is traversed in a recursive manner until a node that overlaps with the analyzed node is found as the overlapping node; the ratio of the data volume of the test data corresponding to the child nodes of the entity nodes to be analyzed in the knowledge graph to the data volume of the test data corresponding to the entity nodes to be analyzed is used as the affinity value between the nodes, and the problem is located based on the affinity value, and the entity nodes to be analyzed are marked as analyzed nodes.

[0013] The present invention provides a device for managing test data lineage based on knowledge graph and evolutionary lineage, the device comprising:

[0014] Triple generation module: configured to trace the current test data, and generate a mapping relationship set of the test data according to the traceability result, wherein the test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data generates the current test data through conversion or operation; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships and black box relationships; based on the data source initially generating data related to the test data, each intermediate data, the current test data and the mapping relationship set, multiple triplets in the form of <first data, mapping relationship, second data> are generated; the second data is generated by the first data through the operation or conversion represented by the mapping relationship in the triple, the distribution relationship means that after conversion or operation, the second data is a subset of the first data, the aggregation relationship means that after conversion or operation, the first data is a true subset of the second data, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship;

[0015] Knowledge graph construction module: Generate a knowledge graph based on all triples, with the first metadata corresponding to the first data and the second metadata corresponding to the second data serving as entity nodes of the knowledge graph. The mapping relationship in the triples serves as a relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; a link is established between each test data stored in the memory and the current test data based on the knowledge graph.

[0016] The present invention provides an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0020] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method as described above.

[0021] The present invention has the following technical effects:

[0022] 1. The present invention uses a knowledge graph to construct the evolutionary pedigree of data and conducts lineage analysis on the data in the joint experiment. Different versions of the same data are saved as a data linked list to facilitate data tracing.

[0023] 2. The present invention solves the problem of high data throughput in joint experiments by traversing and deleting idle data and optimizing storage space. The present invention can effectively solve the problem of fast iterative update of test data in joint experiments and provide traceable data support for the analysis of test results.

[0024] 3. The present invention can effectively analyze the association and affinity between nodes. When problems occur in the test data, the source of the problem can be quickly checked and data irrelevant to the problem can be excluded, thus providing convenience for the joint test guarantee operation.

[0025] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0027] Figure 1 A schematic diagram of a lineage management analysis method in the prior art;

[0028] Figure 2 A schematic diagram of the process of the method for managing the lineage of experimental data based on the knowledge graph and evolutionary lineage proposed in the present invention;

[0029] Figure 3 It is a schematic diagram of the data display system analysis function of the present invention;

[0030] Figure 4 It is a schematic diagram of data conversion classification of the present invention;

[0031] Figure 5 A schematic diagram of part of the information stored in the nodes in the knowledge graph of the present invention;

[0032] Figure 6 This is a schematic diagram of the data link proposed by the present invention;

[0033] Figure 7 A schematic diagram of an embodiment of a method for managing test data lineage based on a knowledge graph and an evolutionary lineage proposed by the present invention;

[0034] Figure 8 Schematic diagram of the device structure for test data lineage management based on knowledge graph and evolutionary pedigree proposed in the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] Figure 2 FIG. 1 is a schematic diagram of a method for managing the lineage of experimental data based on a knowledge graph and an evolutionary lineage. It should be understood that in other embodiments, the order of some steps can be interchanged according to actual needs, or some steps can be omitted or deleted. Figure 2 As shown, the method includes:

[0037] Step S1: trace the source of the current test data, and generate a mapping relationship set of the test data according to the traceability result. The test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data generates the current test data through conversion or operation; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships and black box relationships; based on the data source initially generating data related to the test data, each intermediate data, the current test data and the mapping relationship set, generate multiple triples in the form of <first data, mapping relationship, second data>; the second data is generated by the first data through the operation or conversion represented by the mapping relationship in the triple, the distribution relationship means that after conversion or operation, the second data is a subset of the first data, the aggregation relationship means that after conversion or operation, the first data is a true subset of the second data, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship;

[0038] Step S2: Generate a knowledge graph based on all triples, in which the first metadata corresponding to the first data and the second metadata corresponding to the second data are both used as entity nodes of the knowledge graph, and the mapping relationship in the triples is used as the relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; based on the knowledge graph, a link is established between each test data stored in the memory and the current test data.

[0039] The present invention relates to a lineage analysis technology for test data based on evolutionary genealogy, which is applied to military joint simulation tests. In military joint tests, the source of test data is clear and traceable. By analyzing the inheritance relationship between data, a lineage knowledge graph of data is established to provide support for the full-process tracking and storage of test data, as well as test data quality evaluation, verification and recovery.

[0040] In the present invention, data lineage is data bloodline. Data bloodline is the relationship between data. Data changes often have certain time characteristics. When data flows from upstream to downstream through data processing, a complete data production chain will be formed. Based on the data bloodline relationship map, the data production chain can be traced back to its source.

[0041] Data lineage construction is achieved by abstracting and grouping the transformations in the data processing flow to obtain a mapping set, and then a data lineage knowledge graph is constructed through the data lineage algorithm.

[0042] 1) Conversion category

[0043] The conversion between data can be roughly divided into three modes: distribution, aggregation and black box. Figure 4 shown.

[0044] The analysis of data lineage mainly involves analyzing the data processing process. The following is a specific definition of the conversion T type:

[0045] For each transformation, the set of input data items is denoted as

[0046] I=Input(D1,……,D n )

[0047] The output data item set is recorded as:

[0048] I=Output(O1,……,O n )

[0049] The conversion process is recorded as:

[0050] T(I)=O

[0051] (a) Distribution

[0052] If each input data item produces 0 or more output data items, such a transformation is called a distribution. After the data item {1} is transformed, the output is the data item {4, 5, 6, 7}, the data item {3} is transformed and the output is {6, 7, 8, 9}, and the data item {2} has no output data items. The distribution type of transformation is recorded as:

[0053] T{I i =O uv

[0054] in

[0055] I i ={D i}

[0056]

[0057] Therefore, the data mapping relationship in the distribution transformation is

[0058] D i →D o →|D o ∈T({D i})=O uv

[0059] (b) Polymerization

[0060] If one or more input data items produce only one output data item, this type of transformation is called aggregation. The output item after the data item {1, 2, 3} is transformed is {7}, the output item after the data item {4} is transformed is {8}, and the output item after the data item {5, 6} is transformed is {9}. Aggregate type transformations are recorded as:

[0061] T{I pq =O o

[0062] in

[0063]

[0064] However, for black box type transformations, the specific mapping relationship cannot be known, so each input item and output item is treated as a data association. Therefore, the data mapping relationship in the aggregate transformation is expressed as:

[0065] {D i →D o |D i ∈I pq}

[0066] (c) Black Box

[0067] If the transformation process is neither distribution nor aggregation, and the internal data processing is unknown, then this type of transformation is called a black box. As shown in Figure (c), the output items after the data items {1, 2, 3, 4, 5, 6} are transformed into {7, 8, 9, 10, 11, 12). The black box type of transformation is recorded as:

[0068] T{I pq =O uv

[0069] in

[0070]

[0071] However, for black box type transformations, the specific mapping relationship cannot be known, so each input item and output item is associated as data. Therefore, the data mapping relationship in the aggregate transformation is:

[0072] {D i →D o →|D o ∈I uv , D o ∈O uv}

[0073] The set of transformations in the data processing flow is denoted as

[0074] TS(TranformationSet)={T1,...,Tn}

[0075] The data processing flow is expressed as:

[0076] TS(I)=O

[0077] Where I is the initial set of input data items, and O is the final set of output data items. Each transformation is expressed as:

[0078] T w (I w )=O w

[0079] And the mapping set obtained by conversion and parsing is recorded as

[0080] M(Map)={M1,…,M w , …, M n}

[0081] However, during the conversion, temporary input and output data items may be generated, which are recorded as D', and temporary input and output data item sets are recorded as I', O', so the relevant data mapping may be generated as follows:

[0082] {D′ i →D o , D i →D′ o , D′ i →D′ o , D i →D o}

[0083] That is, for temporary input items and temporary output items that may be generated during the conversion, the temporary input items and temporary output items are marked separately, and their mapping relationships are marked as temporary relationships.

[0084] In the data lineage construction result, the data containing these temporary data items need to be mapped together. The data lineage construction algorithm is as follows.

[0085]

[0086] Through knowledge graph technology, the mapping relationship in data lineage is transformed from discrete database storage to graph structure storage.

[0087] The step S2: based on all triples, a knowledge graph is generated, the first metadata corresponding to the first data and the second metadata corresponding to the second data are both used as entity nodes of the knowledge graph, the mapping relationship in the triples is used as the relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges, wherein:

[0088] The repository has an index table, which indexes the metadata corresponding to each test data and the memory storage location. The metadata includes the data identifier of the test data, a timestamp, all mapping relationships related to the data corresponding to the metadata, and the location of the metadata in the knowledge graph. The timestamp is used to record the collection or generation time of the test data. The mapping relationship records the time when the mapping relationship is generated while recording the relationship type. For the test data stored in the memory and not recorded in the index table, it is recorded in the index table, and the corresponding nodes and edges are established in the knowledge graph according to the method of step S1.

[0089] Furthermore, the mapping relationship associated with the data corresponding to the metadata has a sub-mapping relationship, and the sub-mapping relationship includes an inheritance relationship, an inherited relationship, and the like.

[0090] In the present invention, if the test data already exists in the graph, the position of the metadata corresponding to the test data in the knowledge graph is updated without repeated storage.

[0091] By storing the evolutionary genealogy of test data in the knowledge graph, the analysis and management of data lineage relationships can be effectively realized, providing a basis for querying and modifying data lineages. Figure 5 As shown in the figure, in the knowledge graph of evolutionary genealogy, as the amount of data increases, the storage pressure of the knowledge graph also increases. In order to improve the storage efficiency of the bloodline graph, it is necessary to optimize it and delete the repeated content in the graph. The above processing method can compress the storage space of the evolutionary genealogy.

[0092] Furthermore, a linked list is established for each entity node. When the test data is modified, the modification time of the test data is used as a timestamp, and various versions of the test data are stored in the linked list in sequence according to the timestamp. The modification refers to: the data source that generates the test data corresponding to the entity node generates new test data over time, and when the test object corresponding to the new test data is the same as the test object corresponding to the test data, the new test data is added to the test data corresponding to the entity node.

[0093] The modifications in this application do not allow data deletion operations. A modification relationship is constituted and a successor node is created for the linked list only when and only when the new test data has the same data source and the same corresponding test object as the existing experimental data. Otherwise, it is deemed to generate a new entity node. The entity node name must be different from the old node, and the two are in an inheritance relationship.

[0094] The data identifier of the entity node in the knowledge graph consists of two parts. The first is the unique identifier of the data in the entire joint test system. The bloodline graph can use this identifier to interact with the outside world. The second is the modification time of the data. In order to prevent confusion in the lineage after the data item is modified, when a certain data in the evolutionary lineage is modified, a new data node will be created according to the new modification time. The original data node will still be retained and pointed to the new node with a one-way edge. The versions of the same node modified at different times will form a linked list. When the child node is updated, it will traverse the linked list to find the latest version of the parent node and modify the mapping of the inheritance relationship. The linked list is in the form of Figure 6 shown.

[0095] Furthermore, the knowledge graph is traversed regularly, entity nodes in the knowledge graph that have no mapping relationship with any other entity nodes are regarded as isolated entity nodes, and the isolated entity nodes are marked. When the number of markings exceeds a preset number, the isolated entity nodes and the test data of each version corresponding to the isolated entity nodes are deleted.

[0096] In the bloodline graph, when a data node is deleted, if its parent node loses all its child nodes, it will become a node that cannot be traversed in the lineage relationship analysis and cannot affect the analysis results. When the data concurrency is large, such nodes will take up a lot of space, especially the timestamp marking method mentioned above, which will create multiple different versions of backups for the same data. In the evolutionary pedigree, in order to optimize storage efficiency, the evolutionary pedigree will be traversed regularly to query the references of all nodes and mark all nodes that are not referenced and do not have a reference relationship. If the number of marks on a node accumulates to a certain number, it is considered to have been abandoned and removed from the linked list composed of different versions of the data. The standard for abandoning data nodes is dynamically determined according to the system storage space and computing load, optimizing the storage space and traversal speed as much as possible while ensuring operating efficiency.

[0097] Furthermore, the number of parent nodes and the number of child nodes of each node in the knowledge graph are counted, and the importance of the node and the future frequency of use are predicted based on the number of parent nodes and the number of child nodes.

[0098] In the present invention, traditional database queries can only simply query the integrated relationship of data lineage, but through the analysis function of the knowledge graph, when analyzing a specified data node, it is not only possible to query its parent node and child node, but also to analyze the sibling node and the sibling node of the parent node. The current importance of the data is evaluated by the number of node references, and the future usage frequency of the data is predicted based on the references of its associated nodes, so as to avoid the deletion of data that is about to be used when cleaning data.

[0099] Furthermore, a priority is set for the data stored in the memory. The smaller the difference between the timestamp of the entity node corresponding to the test data and the current time, the higher the priority; the more child nodes the entity node corresponding to the test data has, the higher the priority.

[0100] In the present invention, when analyzing the evolutionary pedigree, it is necessary to take into account that in the joint test, the test data has a strong real-time nature, and the timestamp of the data must be considered when analyzing the data. If the data has been recently updated, or its past version is referenced by other nodes, its priority is increased in the analysis, otherwise the priority is reduced. According to the analysis results, the importance of the nodes is analyzed according to the lineage relationship, which can provide a basis for the optimization of the evolutionary pedigree. By analyzing a single node in the evolutionary pedigree, the importance and relevance of the data in the joint test system can be intuitively reflected, providing a reference for the analysis after the test.

[0101] Furthermore, when locating problems for entity nodes of the knowledge graph, nodes that have been analyzed are marked as analyzed nodes; entity nodes to be analyzed are obtained, a list is generated for the entity nodes to be analyzed, and the child nodes, parent nodes, and each data information in the linked list corresponding to the entity nodes to be analyzed are stored in the list, and each entity node stored in the list is traversed in a recursive manner until a node that overlaps with the analyzed node is found as the overlapping node; the ratio of the amount of test data corresponding to the child nodes of the entity nodes to be analyzed in the knowledge graph to the amount of test data corresponding to the entity nodes to be analyzed is used as the affinity value between the nodes, and the problem is located based on the affinity value, and the entity nodes to be analyzed are marked as analyzed nodes.

[0102] Furthermore, if the child node of the entity node to be analyzed is in a modified successor relationship with the entity node to be analyzed, the affinity value is recorded as 1; if the relationship between the test data is a black box relationship, the affinity value is recorded as the inverse of the total number of parent nodes in the black box relationship.

[0103] In the present invention, through the knowledge graph technology, the breadth-first search algorithm can be combined to perform dependency analysis on multiple nodes to check whether there is a causal relationship between the data. The specific process is to start from each node, traverse the child nodes, parent nodes, and nodes on the timestamp data chain in turn, and add these nodes to a list. Each node to be analyzed has its own list. After a round of traversal, repeat the above operation for the traversed nodes until overlapping nodes appear in the record list, or no overlapping nodes are found after traversing the entire graph. This algorithm can effectively analyze the association and affinity between nodes. When there is a problem with the test data, it can quickly check the source of the problem and exclude data that is not related to the problem, providing convenience for the joint test guarantee operation.

[0104] The test data lineage analysis technology involved in the present invention starts from actual needs and aims to address the problem that the existing data lineage management technology cannot cope with the joint test environment with high concurrency and high update frequency. It combines the evolutionary genealogy technology to record the data in the joint test throughout the entire life cycle, including the source, processing, iteration, circulation and relationship construction of the test data in the evolution process, so as to provide functions for the analysis of test results and the traceability of integrated data.

[0105] Data lineage analysis technology supports the construction of relationships among test data sources, processing, iteration, circulation, and evolution, and provides support for full-process tracking and storage of test data, as well as test data quality evaluation, verification, and recovery.

[0106] The experimental data lineage analysis technology based on evolutionary pedigree proposed in the present invention can effectively address the problems of large experimental data throughput and fast data iteration and update in joint experiments, and provide traceable data support for the analysis of experimental results.

[0107] The present invention provides a specific embodiment of a method for managing test data lineage based on a knowledge graph and an evolutionary lineage.

[0108] The test data lineage analysis technology based on evolutionary pedigree proposed in the present invention is used to perform lineage analysis on the data of the measuring equipment in the joint test. Taking radar as an example, for the same target, multiple radars are usually required to conduct joint monitoring, and the data returned by each radar is converted in form, fused, and other operations. If there is no effective method to record the data lineage, it is difficult to trace the data after processing, which brings difficulties to the analysis of the test results.

[0109] like Figure 7 As shown, by constructing an evolutionary genealogy, the mapping relationship is converted into nodes and edges in the graph, and is optimized and maintained in real time as the experiment proceeds.

[0110] The following is a description of the device used to execute the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0111] Optionally, an embodiment of the present application provides a device for managing test data lineage based on a knowledge graph and an evolutionary lineage, the device comprising:

[0112] Triple generation module: configured to trace the current test data, and generate a mapping relationship set of the test data according to the traceability result, wherein the test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data generates the current test data through conversion or operation; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships and black box relationships; based on the data source initially generating data related to the test data, each intermediate data, the current test data and the mapping relationship set, multiple triplets in the form of <first data, mapping relationship, second data> are generated; the second data is generated by the first data through the operation or conversion represented by the mapping relationship in the triple, the distribution relationship means that after conversion or operation, the second data is a subset of the first data, the aggregation relationship means that after conversion or operation, the first data is a true subset of the second data, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship;

[0113] Knowledge graph construction module: Generate a knowledge graph based on all triples, with the first metadata corresponding to the first data and the second metadata corresponding to the second data serving as entity nodes of the knowledge graph. The mapping relationship in the triples serves as a relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; a link is established between each test data stored in the memory and the current test data based on the knowledge graph.

[0114] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0115] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0116] The above modules can be connected or communicated with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, a ZigBee, or NFC, or any combination thereof. Two or more modules may be combined into a single module, and any one module may be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.

[0117] It should be noted that the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program codes. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0118] Figure 6is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 6 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0119] The present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute the above method embodiment when executed by a processor.

[0120] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0123] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A method for managing test data lineage based on knowledge graph and evolutionary lineage, characterized in that: The method comprises the following steps: Step S1: trace the source of the current test data, and generate a mapping relationship set of the test data according to the traceability result, wherein the test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data is converted or operated to generate the current test data; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships, and black box relationships; Based on the data source, data related to the test data, each intermediate data, the current test data and a set of mapping relationships are initially generated, and a plurality of triplets in the form of <first data, mapping relationship, second data> are generated; the second data is generated by the first data through an operation or conversion represented by the mapping relationship in the triplet, the distribution relationship refers to the second data being a subset of the first data after conversion or operation, the aggregation relationship refers to the first data being a true subset of the second data after conversion or operation, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship; Step S2: Generate a knowledge graph based on all triples, in which the first metadata corresponding to the first data and the second metadata corresponding to the second data are both used as entity nodes of the knowledge graph, and the mapping relationship in the triples is used as the relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; based on the knowledge graph, a link is established between each test data stored in the memory and the current test data.

2. The method according to claim 1, characterized in that The repository has an index table, which indexes the metadata and memory storage location corresponding to each test data; the metadata includes the data identifier of the test data, a timestamp, all mapping relationships related to the data corresponding to the metadata, and the location of the metadata in the knowledge graph, and the timestamp is used to record the collection or generation time of the test data.

3. The method as claimed in claim 2, characterized in that A linked list is established for each entity node. When the test data is modified, the modification time of the test data is used as a timestamp, and various versions of the test data are stored in the linked list in sequence according to the timestamp; the modification refers to: the data source that generates the test data corresponding to the entity node generates new test data over time, and the test object corresponding to the new test data is the same as the test object corresponding to the test data; when the modification occurs, the new test data is added to the test data corresponding to the entity node and stored in the linked list.

4. The method according to claim 3, characterized in that The knowledge graph is traversed regularly, entity nodes in the knowledge graph that have no mapping relationship with any other entity nodes are regarded as isolated entity nodes, and the isolated entity nodes are marked. When the number of markings exceeds a preset number, the isolated entity nodes and the test data of each version corresponding to the isolated entity nodes are deleted.

5. The method according to claim 3, characterized in that When locating the problem, for the knowledge graph, the entity nodes that have been analyzed are marked as analyzed nodes; the entity nodes to be analyzed are obtained, a list is generated for the entity nodes to be analyzed, the child nodes of the entity nodes to be analyzed in the knowledge graph, the parent nodes in the knowledge graph, and each data information in the linked list corresponding to the entity nodes to be analyzed are stored in the list, and each entity node stored in the list is traversed in a recursive manner until a node that overlaps with the analyzed node is found as the overlapping node; the ratio of the data volume of the test data corresponding to the child nodes of the entity nodes to be analyzed in the knowledge graph to the data volume of the test data corresponding to the entity nodes to be analyzed is used as the affinity value between the nodes, the problem is located based on the affinity value, and the entity nodes to be analyzed are marked as analyzed nodes.

6. A device for managing test data lineage based on knowledge graph and evolutionary lineage, characterized in that: The device comprises: Triple generation module: configured to trace the current test data, and generate a mapping relationship set of the test data according to the traceability result, wherein the test data is formed from the data source initially generating data related to the test data, through multiple conversions or operations, each conversion or operation generates an intermediate data, and the last intermediate data generates the current test data through conversion or operation; the mapping relationship set includes mapping relationships representing the conversions or operations from the data source initially generating data related to the test data to the current test data, and the mapping relationships include distribution relationships, aggregation relationships and black box relationships; based on the data source initially generating data related to the test data, each intermediate data, the current test data and the mapping relationship set, multiple triplets in the form of <first data, mapping relationship, second data> are generated; the second data is generated by the first data through the operation or conversion represented by the mapping relationship in the triple, the distribution relationship means that after conversion or operation, the second data is a subset of the first data, the aggregation relationship means that after conversion or operation, the first data is a true subset of the second data, and the black box relationship represents the conversion or operation other than the distribution relationship and the aggregation relationship; Knowledge graph construction module: Generate a knowledge graph based on all triples, with the first metadata corresponding to the first data and the second metadata corresponding to the second data serving as entity nodes of the knowledge graph. The mapping relationship in the triples serves as a relationship in the knowledge graph, and the first metadata and the second metadata corresponding to the mapping relationship are connected in the form of edges; a link is established between each test data stored in the memory and the current test data based on the knowledge graph.

7. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images