A data processing method, device, equipment and storage medium

Abstract

The present application discloses a data processing method, device, equipment and storage medium, the method comprising: for the scheme of the present application, based on at least two data models included in the database and the upstream and downstream relationships between the data models, a first relationship map is established; the first relationship map includes at least two nodes and at least one directed edge; based on the at least two nodes and the at least one directed edge, a first core node in the first relationship map is determined; the data model pointed to by the first core node is compared to obtain a first comparison result for the database; based on the first comparison result, a target comparison result for the database is obtained. When performing the comparison, the scheme of the present application can have the characteristics of high automation, high accuracy, small data processing volume and high comparison efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and is related to but not limited to data processing methods, devices, equipment and storage media. Background Art

[0002] With the rapid development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] Faced with massive data tables in the database, under the premise that the calculation logic does not change, if there is a tool upgrade, machine migration, scheduling tool change, etc., it is necessary to quickly compare the data volume and content of the data table before and after the upgrade or migration to determine whether the data volume and content of the data table before and after the upgrade or migration are consistent.

[0004] The comparison schemes in the related art may include the following two schemes:

[0005] Solution 1: Use traversal to fully compare all data tables in the database until the comparison results of all data tables are obtained;

[0006] Solution 2: Manually sort out and screen the core tables in the database, then compare the core tables in the database one by one. Based on the comparison results, manually determine the upstream and downstream tables of the core table that need to be compared, and then compare the upstream and downstream tables of the core table until the comparison results of all data tables are obtained.

[0007] It can be seen that for Solution 1, all data tables in the database need to be compared. The data processing volume of the comparison process is large and the comparison process is time-consuming, resulting in low comparison efficiency. For Solution 2, manual participation in the screening of core tables is required, which consumes a lot of human resources, and the manual processing efficiency is low and prone to errors. This results in low comparison accuracy, low efficiency, and high manpower consumption. Summary of the invention

[0008] The present application provides a data processing method and apparatus, equipment, and storage medium, which can simultaneously have the characteristics of high automation, high accuracy, small data processing volume, and high comparison efficiency when performing comparison.

[0009] The technical solution of this application is implemented as follows:

[0010] The present application provides a data processing method, the method comprising:

[0011] A first relationship graph is established based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, one of the directed edges points to the second node through the first node, and the directed edge is used to indicate that the first node is an upstream node of the second node;

[0012] Based on the at least two nodes and the at least one directed edge, determining a first core node in the first relationship graph; the first core node is a node in the first relationship graph whose importance satisfies a first condition;

[0013] Compare the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state;

[0014] Based on the first comparison result, a target comparison result for the database is obtained; in the target comparison result, the sub-comparison result of each data model has the same content or different content.

[0015] The present application provides a data processing device, which includes an establishing unit, a determining unit, a comparing unit and a processing unit.

[0016] An establishing unit, configured to establish a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is an upstream node of the second node;

[0017] A determination unit, configured to determine a first core node in the first relationship graph based on the at least two nodes and the at least one directed edge; the first core node is a node in the first relationship graph whose importance satisfies a first condition;

[0018] A comparison unit, configured to compare the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state;

[0019] A processing unit is used to obtain a target comparison result for the database based on the first comparison result; in the target comparison result, the sub-comparison result of each data model has the same content or different content.

[0020] The present application also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned data processing method when executing the program.

[0021] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0022] The data processing method, apparatus, device and storage medium provided in the present application include: establishing a first relationship map based on at least two data models included in a database and the upstream and downstream relationships between the data models; the first relationship map includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, and one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is the upstream node of the second node; based on the at least two nodes and the at least one directed edge, determining a first core node in the first relationship map; the first core node is a node in the first relationship map whose importance meets a first condition; comparing the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state; based on the first comparison result, obtaining a target comparison result for the database; in the target comparison result, the sub-comparison result of each data model is the same content or different content. For the solution of this application, when performing the comparison, a relationship map is first established. From the relationship map, the upstream and downstream relationships between the nodes can be clearly obtained, so as to quickly locate the core node, compare the core node first, and then obtain the target comparison result of the database based on the comparison result of the core node; there is no need to compare all data models in the database, and the comparison results of all data models can be obtained. It can be seen that, on the one hand, the entire comparison process does not require human participation, with a high degree of automation and high accuracy; on the other hand, the non-traversal comparison method is adopted, the data processing volume is small, and the comparison efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 An optional structural diagram of a data processing system provided in an embodiment of the present application;

[0024] Figure 2 An optional flow chart of the data processing method provided in the embodiment of the present application

[0025] Figure 3 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0026] Figure 4 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0027] Figure 5 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0028] Figure 6 An optional structural diagram of a relationship map provided in an embodiment of the present application;

[0029] Figure 7 An optional structural diagram of a relationship map provided in an embodiment of the present application;

[0030] Figure 8 An optional flow chart of the relationship map provided in the embodiment of the present application;

[0031] Fig. 9 An optional structural diagram of a data processing device provided in an embodiment of the present application;

[0032] Fig.10 An optional structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0034] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order for the objects, nor do they have a limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0037] The embodiments of the present application may provide a data processing method and apparatus, a device and a storage medium. In practical applications, the data processing method may be implemented by a data processing apparatus, and each functional entity in the data processing apparatus may be collaboratively implemented by hardware resources of the electronic device, such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).

[0038] The data processing method provided in the embodiment of the present application is applied to a data processing system, and the data processing system includes a data processing end and a data storage end.

[0039] The data storage end is used to store the database.

[0040] The data processing end is used to compare the database, specifically to execute: establishing a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, and one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is the upstream node of the second node; based on the at least two nodes and the at least one directed edge, determining the first core node in the first relationship graph; the first core node is a node in the first relationship graph whose importance meets the first condition; comparing the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state; based on the first comparison result, obtaining a target comparison result for the database; in the target comparison result, the sub-comparison result of each data model is the same content or different content.

[0041] It should be noted that the data processing end and the data storage end can be deployed on an independent electronic device, or can be deployed on different electronic devices respectively.

[0042] As an example, the structure of data processing system 10 may be as follows: Figure 1 As shown, it includes: a data storage end 101 and a data processing end 102; data interaction can be performed between the data storage end 101 and the data processing end 102.

[0043] The data storage end 101 is used to store a database 1011. The database 1011 includes a plurality of data models 1011a.

[0044] The data storage end 101 may be an electronic device with data storage capability, for example, a memory, a server, and the like.

[0045] The data processing end 102 is used to execute: establishing a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, and one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is the upstream node of the second node; determining a first core node in the first relationship graph based on the at least two nodes and the at least one directed edge; the first core node is a node in the first relationship graph whose importance meets a first condition; comparing the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state; based on the first comparison result, obtaining a target comparison result for the database; in the target comparison result, the sub-comparison result of each of the data models is the same content or different content.

[0046] The data processing end 102 may be an electronic device with data processing capability. For example, the data storage end may be a processor, a server, and the like.

[0047] Next, combine Figure 1 The schematic diagram of the data processing system shown illustrates various embodiments of the data processing method and apparatus, device and storage medium provided in the embodiments of the present application.

[0048] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a data processing device; wherein the data processing device can be deployed in Figure 1 The data processing end 102 in the embodiment of the present application is described below.

[0049] Figure 2 The flowchart of an optional data processing method is shown in the figure. The data processing method provided in the embodiment of the present application is used to compare the database to determine whether the data content of each data model in the database in the first state and the second state is consistent. Specifically, the method may include but is not limited to Figure 2 S201 to S204 shown.

[0050] S201. The data processing end establishes a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models.

[0051] The first relationship graph includes at least two nodes and at least one directed edge; a node is used to uniquely point to one data model among at least two data models, a directed edge points to the second node through the first node, and the directed edge is used to characterize that the first node is the upstream node of the second node.

[0052] Data model refers to data stored in a certain format. The embodiment of the present application does not limit the specific form of the data model, and can be configured according to actual needs. For example, the data model can be a table data (data table), a document data or a graphic data, etc.

[0053] A node is used to point to a data model in a relationship graph. A node uniquely points to a data model. The embodiment of the present application does not limit the specific shape of the node, and can be configured according to actual needs. For example, the node can be in the shape of a circle, rectangle, dot, five-pointed star, etc.

[0054] The upstream and downstream relationship between data models is used to characterize the relationship that the downstream data model depends on the upstream data model. Example A: If data model A is obtained based on the content of data model B, it is considered that there is an upstream and downstream relationship between data model A and data model B. Specifically, data model A can be regarded as a downstream data model, and data model B can be regarded as an upstream data model.

[0055] A directed edge is used to point to a second node through a first node, indicating that the first node is an upstream node and the second node is a downstream node. The embodiment of the present application does not limit the specific shape of the directed edge, and can be configured according to actual needs. For example, the directed edge can be in the shape of a solid arrow, a hollow arrow, etc.

[0056] The database includes at least two data models and upstream and downstream relationships between the data models. The embodiment of the present application does not limit the number and content of the data models included in the database and the upstream and downstream relationships between the data models, and can be specifically configured according to actual conditions.

[0057] Example B: The database includes data model A and data model B, and the upstream and downstream relationship between data model A and data model B is 1 (data model A is the downstream data model, and data model B is the upstream data model).

[0058] The relationship map is used to graphically display the data models included in the database and the upstream and downstream relationships between the data models. The first relationship map corresponds to displaying all the data models in the database and the upstream and downstream relationships between all the data models.

[0059] S201 can be implemented as follows: the data processing end establishes a node pointing to the data model for each data model based on at least two data models included in the database and the upstream and downstream relationships between the data models, and establishes directed edges between the nodes for the upstream and downstream relationships between each data model, thereby obtaining a first relationship graph corresponding to the database.

[0060] Example C: Based on Example B, the corresponding first relationship graph includes node A and node B, and directed edge 1; wherein node A points to data model A, node B points to data model B, and the directed edge points from node B to node A.

[0061] S202. The data processing end determines a first core node in the first relationship graph based on the at least two nodes and the at least one directed edge.

[0062] The first core node is a node in the first relationship graph whose importance meets the first condition.

[0063] Importance is used to measure the degree of association between a node in the first relationship graph and other nodes. The embodiment of the present application does not limit the method for determining importance, and can be configured according to actual needs.

[0064] In a possible implementation, the importance of a node may be determined by the number of directed edges connected to the node.

[0065] In another possible implementation, the importance of a node may be determined by the number of downstream nodes of the node and the number of shortest paths through the node.

[0066] The first condition is used to assist in screening the first core node. The embodiment of the present application does not impose a unique limitation on the content of the first condition and can be configured according to actual needs.

[0067] In a possible implementation, the first condition may be: a node with the greatest importance.

[0068] In another possible implementation, the first condition may be: the first N nodes sorted in descending order of importance.

[0069] S202 can be implemented as follows: the data processing end determines the importance of each of the at least two nodes based on the at least two nodes and the at least one directed edge, and determines the node of the at least two nodes that satisfies the first condition of importance as the first core node in the first relationship graph.

[0070] It should be noted that the first core node obtained by the data processing end in S202 may be one or more. If the first core node obtained by the data processing end in S202 is more than one, the following S203 is executed for each first core node.

[0071] S203: The data processing end compares the data model pointed to by the first core node to obtain a first comparison result for the database.

[0072] The comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state.

[0073] The first state is different from the second state. The embodiments of the present application do not limit the specific contents of the first state and the second state, and can be configured according to actual needs. For example, the first state can be before the upgrade, and the corresponding second state can be after the upgrade; or the first state can be before the device migration, and the corresponding second state can be after the device migration; or the first state can be before the change, and the corresponding second state can be after the change.

[0074] S203 can be implemented as follows: the data processing end calls the comparison function, compares the data model pointed to by the first core node to see whether the content of the data model in the first state is the same as the content of the data model in the second state, and obtains a first sub-comparison result (the content is the same or different); then, based on the first sub-comparison result, updates the sub-comparison results corresponding to other data models in the database, thereby obtaining a first comparison result for the database.

[0075] Among them, the comparison result is for the database, and the sub-comparison result is for the data model; the comparison result includes the sub-comparison results corresponding to all data models in the database.

[0076] S204: The data processing end obtains a target comparison result for the database based on the first comparison result.

[0077] In the target comparison result, the sub-comparison result of each data model has the same content or different content; in other words, there is no data model without a sub-comparison result in the target comparison result.

[0078] S204 can be implemented as follows: the data processing end determines, based on the first comparison result, whether the sub-comparison results of each data model in the first comparison result are the same or different in content. If so, the first comparison result is used as the target comparison result; if not, the target comparison result is further determined.

[0079] The data processing scheme provided in the embodiment of the present application includes: establishing a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, and one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is the upstream node of the second node; based on the at least two nodes and the at least one directed edge, determining the first core node in the first relationship graph; the first core node is a node in the first relationship graph whose importance meets the first condition; comparing the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state; based on the first comparison result, obtaining a target comparison result for the database; in the target comparison result, the sub-comparison result of each data model is the same content or different content. For the solution of this application, when performing the comparison, a relationship map is first established. From the relationship map, the upstream and downstream relationships between the nodes can be clearly obtained, so as to quickly locate the core node, compare the core node first, and then obtain the target comparison result of the database based on the comparison result of the core node; there is no need to compare all data models in the database, and the comparison results of all data models can be obtained. It can be seen that, on the one hand, the entire comparison process does not require human participation, with a high degree of automation and high accuracy; on the other hand, the non-traversal comparison method is adopted, the data processing volume is small, and the comparison efficiency is high.

[0080] Next, the process of establishing a first relationship map based on at least two data models included in the database and the upstream and downstream relationships between the data models at the data processing end in S201 is described. This process may include but is not limited to Figure 3 S2011 to S2013 shown.

[0081] S2011. The data processing end establishes a node pointing to each of the at least two data models to obtain the at least two nodes.

[0082] S2011 can be implemented as follows: the data processing end establishes a node pointing to each data model in at least two data models, and obtains at least two nodes.

[0083] The embodiment of the present application does not limit the method of establishing a node pointing to the data model, and can be configured according to actual needs. Exemplarily, the data processing end can establish a node identifier (such as a circle) based on a data model (such as data model 1), and name the node identifier, such as node 1, and establish an association relationship between node 1 and data model 1 (such as a table correspondence relationship), so as to realize pointing to data model 1 through node 1.

[0084] S2012. The data processing end establishes a directed edge from the first node to the second node based on the upstream and downstream relationship between the data models, and obtains the at least one directed edge.

[0085] The first node is used to point to the first data model, and the second node is used to point to the second data model; the upstream and downstream relationship is used to represent that the first data in the second data model is obtained according to the first data model.

[0086] The first data may be all or part of the data in the second data model.

[0087] S2012 can be implemented as follows: the data processing end establishes a directed edge from the first node to the second node for the upstream and downstream relationship between each data model, thereby obtaining at least one directed edge.

[0088] The embodiment of the present application does not limit the method of establishing a directed edge from a first node to a second node, and can be configured according to actual needs. Example D: Based on Example B, if node A points to data model A, and node B points to data model B, the data model establishes a directed edge (e.g., an arrow) from node B to node A. Through the directed edge (e.g., an arrow), it can be known that node A is a downstream node and node A is an upstream node.

[0089] S2013. The data processing end obtains the first relationship graph based on at least the at least two nodes and the at least one directed edge.

[0090] The implementation of S2013 may include but is not limited to the following method 1 or method 2:

[0091] Method 1: obtaining a first relationship graph based on at least two nodes and at least one directed edge;

[0092] Method 2: Obtain a first relationship graph based on at least two nodes, point attributes of the nodes, at least one directed edge, and edge attributes of the directed edge.

[0093] For method 1, the data processing end automatically generates a first relationship graph after establishing nodes for all data models in the database and establishing directed edges for upstream and downstream relationships between all data models.

[0094] For method 2, before executing S2013, the data processing method provided in the embodiment of the present application may also include but is not limited to S2014 and S2015.

[0095] S2014. The data processing end determines the point attributes of each of the at least two nodes.

[0096] The point attribute is used to characterize the comparison result of the data model pointed to by the node. The embodiment of the present application does not limit the specific determination method of the point attribute, and it can be configured according to actual needs.

[0097] For example, the point attribute value of each node is initialized to a first value (for example, -1), that is, the first value indicates that there is no comparison result; if the comparison result of the data model pointed to by the node is that the content is the same, the comparison result of the node is configured to be a second value (for example, 0); if the comparison result of the data model pointed to by the node is that the content is different, the comparison result of the node is configured to be a third value (for example, 1).

[0098] S2014 can be implemented as follows: the data processing end determines the comparison result (same content, different content or not compared) of each node in at least two nodes, and adds the point attribute of the node at the corresponding position of the first node according to the comparison result.

[0099] S2015. The data processing end determines an edge attribute of each directed edge in at least one directed edge.

[0100] The upstream and downstream relationship is used to characterize that the first data in the second data model is obtained according to the first data model, and the edge attribute is used to point to the first data.

[0101] S2015 can be implemented as follows: the data processing end determines the first data in the downstream data model for each directed edge of at least one directed edge, and adds the edge attribute of the directed edge at the corresponding position of the directed edge according to the first data.

[0102] Correspondingly, S2013 can be implemented as follows: the data processing end establishes nodes for all data models in the database, and adds the point attributes of the node at each node; after establishing directed edges in the upstream and downstream relationships between all data models, the edge attributes of the directed edges are added at each directed location, and then the first relationship graph is automatically generated.

[0103] Next, the process of determining the first core node in the first relationship graph based on the at least two nodes and the at least one directed edge at the data processing end in S202 is described. The process may include but is not limited to any one of the following methods A1 to A3:

[0104] Method A1: Determine the first core node based on the degree centrality of the node and the betweenness centrality of the node;

[0105] Method A2: Determine the first core node based on the degree centrality of the node;

[0106] Method A3: Determine the first core node based on the betweenness centrality of the node.

[0107] Next, the implementation process of method A1, determining the first core node based on the degree centrality of the node and the betweenness centrality of the node, is described. This process may include but is not limited to Figure 4 S2021 to S2024 shown.

[0108] S2021. The data processing end determines the degree centrality of each of the at least two nodes based on the at least two nodes and the at least one directed edge.

[0109] The degree centrality of a node is used to characterize the number of downstream nodes of the node in the first relationship graph.

[0110] In a possible implementation, the number of direct downstream nodes of the node in the first relationship graph can be used as the degree centrality of the node. In this possible implementation, S2021 can be implemented as follows: the data processing end counts the number of direct downstream nodes of each node as the degree centrality of the node based on at least two nodes and at least one directed edge included in the first relationship graph, thereby determining the degree centrality of each node.

[0111] In a possible implementation, the number of extended downstream nodes of the node in the first relationship graph (the number of all downstream nodes) can be used as the degree centrality of the node. In this possible implementation, S2021 can be implemented as follows: the data processing end counts the number of all downstream nodes of each node as the degree centrality of the node based on at least two nodes and at least one directed edge included in the first relationship graph for each node; thereby determining the degree centrality of each node.

[0112] S2022. The data processing end determines the betweenness centrality of each of the at least two nodes based on the at least two nodes and the at least one directed edge.

[0113] Betweenness centrality is used to characterize the number of times a node appears on the shortest path of the first relationship graph.

[0114] S2022 can be implemented as follows: the data processing end obtains all shortest paths of the first relationship graph based on the shortest path algorithm (such as the Dijkstra algorithm), and then for each node, counts the number of times each node appears on the shortest path as the betweenness centrality of the node; thereby obtaining the betweenness centrality of each node.

[0115] S2023: The data processing end determines the importance of each of the at least two nodes.

[0116] The importance of a node is related to the degree centrality of the node and the betweenness centrality of the node.

[0117] S2023 can be implemented as follows: for each of the at least two nodes, the data processing end performs a first operation on the degree centrality of the node and the betweenness centrality of the node to obtain the importance of the node, thereby obtaining the importance of each node.

[0118] The first operation is not specifically limited in the embodiment of the present application and can be configured according to actual needs. The exemplary first operation can be direct summation or weighted summation.

[0119] S2024. The data processing end determines, among the at least two nodes, a node whose importance satisfies the first condition as the first core node.

[0120] Exemplarily, the data processing end determines the most important node among at least two nodes as the first core node.

[0121] Next, the implementation process of determining the degree centrality of each of the at least two nodes based on the at least two nodes and the at least one directed edge at the S2021 data processing end is described.

[0122] Specifically, for each node, the data processing end calculates the degree centrality of the node according to the first formula.

[0123] The first formula can be: t i Represents the node; C D (t i ) represents the degree centrality (direct degree centrality) of the node; t j t i Downstream node of x ij Indicates the existence of i Point to t j The directed edge of .

[0124] Optionally, the first formula can also be t i Represents the node; C U (t i) represents the degree centrality (extended degree centrality) of the node, C D (t i ) represents the direct degree centrality of the node; x ij Indicates the existence of i to j Directed edge of C D (t j ) means (t j )The direct degree centrality of the node.

[0125] Next, the implementation process of determining the betweenness centrality of each of the at least two nodes based on the at least two nodes and the at least one directed edge at the data processing end in S2022 is described.

[0126] For each node, the data processing end calculates the betweenness centrality of the node according to the second formula.

[0127] The second formula includes t i Represents the node; t j ,t k For at least two nodes except t i Nodes outside C B (t i ) represents the betweenness centrality of the node, and p(j, i, k) represents t i At t j to k The number of shortest paths, Indicates that in the first relationship graph t j to k The number of all shortest paths.

[0128] Next, the implementation process of determining the importance of each of the at least two nodes at the data processing end in S2023 is described.

[0129] For each of the nodes, the data processing end calculates the importance of the node according to the third formula.

[0130] The third formula includes: t i Indicates the node; TI(t i ) indicates the importance of the node; C B (t i ) represents the betweenness centrality of the node; represents the mean of the betweenness centrality of all nodes in the first relationship graph, sd(C B (t)) represents the standard deviation of the betweenness centrality of all nodes in the first relationship graph; C U (t i) represents the degree centrality of the node; represents the mean of the degree centrality of all nodes in the first relationship graph; sd(C U (t)) represents the standard deviation of the degree centrality of all nodes in the first relationship graph.

[0131] For method A2, the first core node is determined based on the degree centrality of the node.

[0132] The specific implementation process can refer to the description of method A1. It should be noted that in method A2, the degree centrality of the node is regarded as the importance of the node.

[0133] Method A3: Determine the first core node based on the betweenness centrality of the node.

[0134] The specific implementation process can refer to the description of method A1. It should be noted that in method A3, the betweenness centrality of a node is regarded as the importance of the node.

[0135] Next, the implementation process of S203, in which the data processing end compares the data model pointed to by the first core node to obtain the first comparison result for the database, is described. This process may include but is not limited to Figure 5 S2031 to S2033 shown.

[0136] S2031. The data processing end compares the data model pointed to by the first core node to obtain a first sub-comparison result.

[0137] The data processing end calls the comparison function to compare the data model pointed to by the first core node to see whether its content in the first state and the second state is the same, and obtains the first sub-comparison result (the content is the same or different).

[0138] S2032. If the first sub-comparison result is identical in content, the data processing end updates the sub-comparison result of the data model pointed to by the upstream node of the first core node in the first relationship graph based on the first sub-comparison result to obtain the first comparison result.

[0139] S2032 can be implemented as follows: if the first sub-comparison result is the same content, the data processing end updates the sub-comparison result of the data model pointed to by the upstream node of the first core node in the first relationship graph to obtain the first comparison result. Specifically, if the original comparison result of the upstream node is not compared, it is updated to the same content; if the original comparison result of the upstream node is the same content, it remains unchanged; if the original comparison result of the upstream node is different content, the comparison result of different content is still maintained.

[0140] Optionally, if the first relationship graph includes point attributes of nodes, the point attributes of the first core node and the upstream nodes of the first core node are updated.

[0141] S2033. If the first sub-comparison result is different content, the data processing end obtains different fields, and updates the sub-comparison result of the data model pointed to by the node related to the different fields in the downstream node of the first core node in the first relationship graph to different content, thereby obtaining the first comparison result.

[0142] S2033 can be implemented as follows: if the first sub-comparison result is different content, the data processing end updates the first relationship graph, and the sub-comparison result of the data model pointed to by the nodes related to different fields in the downstream nodes of the first core node is updated to different content.

[0143] Optionally, if the first relationship graph includes edge attributes of directed edges, the edge attributes of related directed edges are updated based on different fields.

[0144] Next, the implementation process of obtaining the target comparison result for the database based on the first comparison result at the data processing end in S204 is described. This process may include but is not limited to the following S2041 to S2044.

[0145] S2041. The data processing end determines whether the first comparison result satisfies the sub-comparison result of each data model as having the same content or different content.

[0146] In other words, the data processing end determines whether there is a data model whose comparison result is an unmatched data model in the first comparison result.

[0147] If the first comparison result cannot satisfy that the sub-comparison result of each data model is the same content or different content, execute the following S2042 to S2044; if the first comparison result satisfies that the sub-comparison result of each data model is the same content or different content, end; directly determine the first comparison result as the target comparison result.

[0148] S2042. The data processing end determines a second relationship graph based on the first core node.

[0149] The second relationship graph is a sub-graph of the first relationship graph.

[0150] The data processing end determines the second relationship graph based on the first core node. Specifically, the data processing end may use the downstream nodes of the first core node and the directed edges between the downstream nodes in the first relationship graph as the second relationship graph; or the data processing end may use the upstream nodes of the first core node and the directed edges between the upstream nodes in the first relationship graph as the second relationship graph.

[0151] S2043. The data processing end determines a second core node in the second relationship graph, and compares the data model pointed to by the second core node to obtain a second comparison result for the database.

[0152] The process of determining the second core node in the second relationship graph by the data processing end can refer to the detailed description of S202, which will not be repeated here. The difference from S202 is that S202 determines the first core node in the first relationship graph, and S2043 determines the second core node in the second relationship graph.

[0153] The data processing end compares the data model pointed to by the second core node to obtain the second comparison result for the database. The detailed description of S203 can be referred to, and will not be repeated here. Different from S203, S2043 updates the second comparison result based on S203, that is, based on the first comparison result.

[0154] S2044. The data processing end uses the second comparison result as the new first comparison result, and determines whether the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, until the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, and uses the new first comparison result that satisfies the sub-comparison result of each data model that the content is the same or the content is different as the target comparison result.

[0155] Next, the process of determining the second relationship graph based on the first core node by the data processing end in S2042 is described.

[0156] If the first sub-comparison result of the data model pointed to by the first core node is the same in content, the downstream nodes of the first core node and the directed edges between the downstream nodes in the first relationship graph are used as the second relationship graph.

[0157] If the first sub-comparison result is different in content, the upstream node of the first core node and the directed edges between the upstream nodes in the first relationship graph are used as the second relationship graph.

[0158] The following uses an example in which the data model is a data table to illustrate the data processing method provided in the embodiment of the present application through an embodiment.

[0159] To facilitate understanding, some terms involved in the following embodiments are explained.

[0160] Relation Graph, also known as Relation Network, refers to a graph consisting of nodes (Vertex) and edges (Edge), which is used to represent the data structure of nodes and their relationships. The application of relation graph provides an effective way to express, organize, manage and utilize massive and dynamic big data on the Internet.

[0161] Graph processing refers to the way of displaying data models and the relationship between data models in a graphical way. Related methods in graph processing can include: shortest path, connected components, node properties (such as degree centrality, betweenness centrality) and other calculation methods.

[0162] Social network analysis (SNA) refers to the relationship network between users presented in a graph, which is a special case of a complex network. The focus of social network analysis is on relationships and relationship patterns, mining specific user groups, etc.

[0163] Faced with massive data tables in the database, under the premise that the calculation logic does not change, if there is a tool upgrade, machine migration, scheduling tool change, etc., it is necessary to quickly compare the data volume and content of the data table before and after the upgrade or migration to determine whether the data volume and content of the data table before and after the upgrade or migration are consistent.

[0164] The comparison schemes in the related art may include the following two schemes:

[0165] Solution 1: Use traversal to fully compare all data tables in the database;

[0166] Solution 2: Manually sort out and screen the core tables in the database, then compare the core tables in the database one by one. Based on the comparison results, manually determine the upstream and downstream tables of the core table that need to be compared, and then compare the upstream and downstream tables of the core table.

[0167] The comparison scheme of the related art has the following disadvantages:

[0168] For Solution 1, all data tables are traversed for consistency comparison. The entire comparison process is time-consuming and inefficient. Especially when the data tables in the database are large, the time-consuming and efficiency problems are more obvious, and may even be impossible to achieve.

[0169] For Solution 2, on the one hand, the core table is manually sorted out, which requires a lot of manpower for verification and investigation; for example, multiple data table creators or users who are familiar with the calculation logic and data characteristics of the data table are required to verify and investigate; on the other hand, the upstream and downstream tables of the core table that needs to be compared are manually searched layer by layer, and then the upstream and downstream tables of the core table are compared; in this way, the entire comparison process is cumbersome and prone to omissions, resulting in a long comparison process and the inability to guarantee the accuracy of the comparison results.

[0170] This embodiment of the present application has the following features:

[0171] (1) Based on the idea of ​​network analysis in graph computing, a relationship graph of all data tables in the database is constructed according to the calculation statements. Combined with the centrality and other characteristics of the nodes in the relationship graph, an algorithm is used to automatically identify the core tables in the database, reducing the manual screening process;

[0172] (2) Combined with the directed relationship graph formed by the upstream and downstream tables, the upstream and downstream tables that need to be further compared can be automatically identified through the preset algorithm process according to the comparison results of the core table, thereby narrowing the comparison scope, reducing the number of comparison tables and the time consumption, and improving the comparison efficiency;

[0173] (3) When constructing a relationship graph, the associated fields of the upstream and downstream tables are recorded in the edge attributes. When it is necessary to locate the upstream and downstream problem tables of the inconsistent table, the upstream tables related to the inconsistent fields can be quickly screened out through the field information in the edge attributes as the upstream and downstream problem tables of the inconsistent table, further narrowing the comparison scope, reducing the time consumption of the comparison process, and improving the comparison efficiency.

[0174] The principles of the embodiments of the present application are briefly described below.

[0175] In the Internet industry, the speed of tool iteration is gradually accelerating. When facing tool upgrades and machine migration, enterprises need to always ensure data consistency to prevent inconsistent data content due to incompatibility between tool versions. Currently, data consistency comparison relies more on manual screening of data tables for comparison, which often takes a lot of time and manpower to sort out, investigate and locate problems, and is prone to omissions.

[0176] In social network analysis (SNA), centrality is often used to locate nodes or people in a central position in the entire network to determine the importance of the node in the entire network. Based on the idea of ​​network analysis in graph computing, in the data comparison process, this embodiment of the present application first builds a network (relationship map) based on the data table, then determines the core table based on the network, and further compares the upstream and downstream tables according to the actual needs of data comparison, forming a set of fast data comparison solutions based on graph computing, which reduces the time cost and labor cost in the data comparison process.

[0177] Specifically, this embodiment of the present application may include the following stages:

[0178] Data table relationship graph establishment stage: Based on the SQL statements calculated by the data table, a reasonable point-edge relationship is constructed based on the data table and the relationship between the data tables to obtain a relationship graph. The point is the data table name, the edge is the directed edge from the upstream table to the downstream table, the edge attribute records the field names related to the tables, and the point attribute records the comparison results of the tables.

[0179] Core table (equivalent to the core data model mentioned above) identification stage: automatically identify the data table at the core position in the relationship graph (hereinafter referred to as the core table or core data table). For example, based on the idea of ​​network analysis in graph computing, calculate the degree centrality (equivalent to the degree centrality of the node), betweenness centrality and other properties of each data table in the graph, and measure the importance of the data table according to the node characteristics to obtain the core table.

[0180] Data table consistency comparison stage: Compare each data table through the built-in data table consistency comparison function and data inconsistent field acquisition function. Optionally, based on the data comparison results of the core table, some upstream / downstream tables that do not need to be compared in the next round can be automatically excluded. After locating the specific inconsistent field names of the data table, the upstream and downstream tables related to the inconsistent fields can be further filtered out according to the edge attributes in the relationship graph, and the comparison results of the upstream and downstream tables related to the inconsistent fields can be directly obtained.

[0181] The implementation process of each stage is described in detail below.

[0182] The data table relationship map establishment phase can include:

[0183] According to the SQL statements calculated by the data tables, a directed isomorphic relationship graph G (T, E, L) with node and edge attributes composed of the data tables can be constructed.

[0184] Where T = {t1, t2, t3, ..., t n} is a finite set of vertices; E = {(t1, t2), (t2, t4), … (t1, t n )} is a finite set of edges connecting a pair of vertices, and the vertex pairs in E are ordered; L = {(t1, f1), (t2, f2), …, ((t1, t2), {x1, x2, x3}), ((t2, t4), {x5})…} is a set of key-value pairs about edges or fixed points, and the values ​​in the key-value pairs are the attribute values ​​of the vertices or edges, among which the attribute values ​​x1, x2, x3… of the edges represent the relevant fields in the data table, and the attribute values ​​f1, f2, … of the fixed points represent the comparison results of the data table.

[0185] Specifically, the construction of the data table relationship graph may include: point set construction, edge set construction, and attribute set construction.

[0186] Point set construction: The vertex entities in T are homogeneous entities (i.e. nodes of the same type, all data tables), and all are data table names, such as t i .

[0187] Edge set construction: If the data table t is found by parsing the SQL statement j The calculation process involves the data table t i , then add (t i , t j ) indicates that t i Yes j The direct upstream table of the i Point to t j The side relationship.

[0188] Attribute set construction can include point attribute construction and edge attribute construction.

[0189] Point attribute construction:

[0190] For the data table that needs to be compared, i , record the comparison result as f i As a data table i The attribute of the node to which it belongs. The comparison result can be described by the following formula (1).

[0191]

[0192] In formula (1), f i represents the attribute of the node, that is, the comparison result of the node; in formula (1), if t i The node to which it belongs has not been compared, then the attribute of the node is -1; if t i If the comparison result of the node to which it belongs is that the data remains consistent, the attribute of the node is 0; if the comparison result of the node is inconsistent, the attribute of the node is 1.

[0193] Edge attribute construction:

[0194] By parsing the SQL statement, you can get the data table t j The calculation process is related to the data table t i A collection of related field names. For example, if the data table t j The calculation and data table of x1, x2, x5 in i Related, that is, t j The fields x1, x2, and x5 in the table are dependent on the upstream data table t when they are calculated. i , then in L is the edge (t i , t j )Add the attributes of {x1, x2, x5}.

[0195] For example, t j x1 in the calculation is used to calculate t i The number of duplicate ID cards in the j The calculation of x1 in t i Related, take x1 as the edge (t i , t j ) attributes.

[0196] The core table identification stage is described below. This stage may include core table identification feature calculation and core table screening.

[0197] The process of calculating the core table identification features may include:

[0198] For each connected subgraph in the data table relationship graph, calculate the node features such as degree centrality and betweenness centrality of each node in the graph.

[0199] Degree centrality: It is used to measure the out-degree of a node, that is, the number of downstream nodes a node has.

[0200] In a possible implementation, degree centrality can be understood as direct degree centrality, which is represented by the number of direct downstream tables of a data table. For example, the direct degree centrality of a node can be calculated by the following formula (2) (equivalent to the first formula above).

[0201]

[0202] In formula (2), C D (t i ) means (t i ) belongs to the direct degree centrality of the node; x ij Indicates the existence ofi to j The directed edge of .

[0203] Optionally, degree centrality can be understood as extended degree centrality, and extended degree centrality can also be represented by the number of all downstream tables of the data table. For example, the extended degree centrality of a node can be calculated by the following formula (3).

[0204]

[0205] In formula (3), C U (t i ) means (t i ) belongs to the node expansion degree centrality, C D (t i ) means (t i ) belongs to the direct degree centrality of the node; x ij Indicates the existence of i to j Directed edge of C D (t j ) means (t j ) belongs to the direct degree centrality of the node.

[0206] Betweenness centrality: It is used to measure the number of times a node appears in the shortest path between any two nodes. That is, the role of a data table as a "bridge" in the graph can be evaluated by betweenness centrality. For example, the betweenness centrality of a node can be calculated by the following formula (4) (equivalent to the second formula above).

[0207]

[0208] In formula (4), C B (t i ) means (t i ) belongs to the node betweenness centrality, p(j, i, k) represents t i At t j to k The number of shortest paths, Indicates t j to k The number of all shortest paths.

[0209] Next, the implementation of core table filtering is described.

[0210] After completing the feature calculation of each node in the data table relationship map, combined with the upstream and downstream characteristics of the table, the data tables corresponding to the nodes with high importance (high betweenness centrality and high degree centrality) are prioritized. The importance of the node can be calculated by the following formula (5) (equivalent to the third formula above).

[0211]

[0212] In formula (5), TI(t i ) means (t i ) belongs to the importance of the node; C B (t i ) means (t i ) belongs to the betweenness centrality of the node; represents the mean of betweenness centrality, sd(C B (t)) represents the standard deviation of betweenness centrality; C U (t i ) means (t i ) belongs to the degree centrality of the node; represents the mean of degree centrality; sd(C U (t)) represents the standard deviation of degree centrality.

[0213] In this embodiment, the two centrality indices are standardized so that they are on the same basis when compared, and TI (t i ) is robust. Among them, TI(t i ) is larger, indicating that in this relationship graph, i The more central the location, the higher the priority for comparison.

[0214] For example, in Figure 6 In the relationship diagram shown, table_2_2, table_2_3, table_3_1, and table_3_3 are of higher importance. Table_2_2, table_2_3, table_3_1, and table_3_3 can be used as core tables and compared first.

[0215] Next, the implementation of the data table consistency comparison phase is described.

[0216] In this stage, the relationship graph sub-graph segmentation and data table consistency comparison process are automatically cycled according to the consistent or inconsistent comparison results of the core table, as follows:

[0217] If the comparison results of the core tables are consistent, the upstream table exits the comparison process, and the relationship graph subgraph formed by the downstream table is extracted for the next round of comparison.

[0218] For example, Figure 7In the example, the subgraph formed by the downstream tables of table_2_2 will be compared with the next round of core tables.

[0219] If the comparison result of the core table is inconsistent, the comparison result label of the downstream table related to the inconsistent comparison field in the downstream table is updated to "inconsistent", then the downstream table exits the comparison process, and the relationship graph subgraph formed by the upstream table related to the inconsistent comparison field is extracted for the next round of comparison.

[0220] For example, Figure 8 In the comparison, the downstream table of table_2_2 exits the comparison process, and the upstream table table_1_1 related to the inconsistent column x_3 of table_2_2 proceeds to the next round of comparison.

[0221] For example, suppose that the comparison of table A is inconsistent, specifically, the average age field in table A is inconsistent; table A depends on the upstream tables B and C, that is, the age field in table A is obtained by averaging the age in table B and the age in table C. Then, it is inferred that there may be a problem with the age field in table B or table C, so the upstream table of A needs to be compared to see whether table B or table C has a problem.

[0222] For the downstream table of table A, such as table D, suppose there is a field for calculating the maximum value of the average age. Since the average age field in table A is inconsistent, table D must also be inconsistent. There is no need to compare, and the comparison result of table D can be directly obtained as inconsistent.

[0223] Furthermore, if table A is consistent, but table D depends on the calculation process of table A, there may be problems, so table D may still be inconsistent (for example, if a row is missed when reading table A for some reason, the calculated value will also be incorrect). Therefore, the consistency of the upstream table does not necessarily mean that the downstream table is consistent, and further comparison is needed.

[0224] During the comparison process, update the comparison result label f ​​of the data table i When the inference result and f i The maximum value between the original values. That is, if the original f i =-1, when the data table is in an unmatched state, it will be updated to 0 or 1 according to the comparison result or the comparison result of the upstream and downstream tables; f i = 1 does not need to be compared again and cannot be updated to f i = 0, and f i = 0 may be updated to 1 in subsequent further comparisons based on its own comparison results or the comparison results of other upstream and downstream tables.

[0225] The main reason here is that when updating the consistency of the upstream table, it is actually an inference. Because the table is consistent, it is inferred that the upstream table is consistent. But for example, the upstream table A has two fields, age and gender, and tables B and C are its two downstream core tables, which rely on these two fields respectively. When the comparison of table B shows consistency, it is first inferred that table A is consistent (in fact, it can only mean that the age field in table A is consistent, but the gender field is actually to be determined); but when comparing table C, if inconsistency is found, then it is necessary to confirm whether the inconsistency of the gender field in table A caused the inconsistency of table C, so the inferred consistency of table A may be updated to a confirmed inconsistency.

[0226] On the other hand, if table C is compared first, then if table A is already confirmed to be inconsistent, even if table B is consistent, this upstream table cannot be updated to be consistent.

[0227] In addition, if there are multiple connected subgraphs in the original relationship graph, it can also be divided into multiple subgraphs before the core table comparison is performed; when only one data table remains in the subgraph, the data table is directly compared.

[0228] Optionally, we can also pre-determine the approximate proportion threshold r of the core tables in the database through expert experience, r∈(0,1), calculate the importance of the data tables, sort the importance, and select the data tables with the highest importance as the core tables. For example, if there are 100 tables in the database, and the core table proportion threshold is 20%, then according to the above calculation, select the top 20 data tables with the highest importance as the core tables in the database, and continue the subsequent comparison process.

[0229] It should be noted that the size of the threshold will not affect the final comparison result, but may affect the comparison speed. Because the core table will be directly compared in the first round, if this threshold is 1, it is equivalent to all data tables being directly compared. Generally speaking, it is more reasonable to give a threshold below 50% so that the number of core tables that can be compared in parallel in the first round will not be too large, and some upstream and downstream tables that need to be compared later can be automatically excluded based on the comparison results of the first round.

[0230] Exemplarily, the specific algorithm flow is as follows:

[0231]

[0232]

[0233] Among them, F = {f_1, f_2, f_3, ..., f_n} = {-1, -1, -1, ..., -1}; represents the initialization comparison result.

[0234] while-1∈F: It means that there are still tables that may not be correct, so the comparison process continues.

[0235] for g_i∈G: indicates that the following process is performed on each connected subgraph formed by the relationship table (for the connected subgraph, assuming that the four tables ABCD are upstream and downstream tables of each other, and EFGH are upstream and downstream tables of each other, and have no relationship with ABCD, then ABCD will constitute a connected subgraph, and EFGH will constitute another connected subgraph. The following process will first be executed on the subgraph formed by ABCD, for example, it will be found that B is an inconsistent table; and then it will be executed on the subgraph formed by EFGH, for example, they are all consistent).

[0236] if|T_i|==1: [f]_i=Compare(t_i), t_i∈g_i; it means that if there is only one node, that is, only one table, then this table is directly compared.

[0237] else: T_core = Top_(

|T_i|*r

[0238] for t_j∈T_core, where f_j≠1: means that for each selected core table, those that are inconsistent in comparison are excluded and the subsequent process is carried out.

[0239] if f_j=Compare(t_j)==0: indicates that the comparison result of this core table is consistent.

[0240] f_k=max(f_k, 0), where x_jk≠1; indicates updating the comparison result of its upstream table, taking the maximum value of the original value and 0, that is, if the original value is -1 which may not be correct, it is updated to a consistent 0, and if the original value is an inconsistent 1, it remains 1.

[0241] g_m→G, where x_jm=1; it means that the subgraph formed by its downstream table is also added to the comparison process.

[0242] if f_j=Compare(t_j)==1: indicates that the comparison result of this core table is inconsistent.

[0243] f_k=1,where x_jk=1and Diff(t_j)∈L((

t_j,t

[0244] It means updating the comparison result of the downstream table that is not related to the inconsistent fields of the core table, taking the maximum value of the original value and 0, that is, if it was originally -1 which may not be correct, it will be updated to a consistent 0, and if it was originally an inconsistent 1, it will remain 1.

[0245] It means updating the comparison result of the upstream table that is not related to the inconsistent fields of the core table, taking the maximum value of the original value and 0, that is, if it was originally -1 which may not be correct, it will be updated to a consistent 0, and if it was originally an inconsistent 1, it will remain 1.

[0246]

g

t_m,t

[0247] End indicates the end of the comparison process.

[0248] In the second aspect, in order to implement the above data processing method, a data processing device according to an embodiment of the present application is provided as follows: Fig. 9 The structure of the data processing device is described below.

[0249] like Fig. 9 As shown, the data processing device 90 includes: an establishing unit 901, a determining unit 902, a comparing unit 903 and a processing unit 904. Among them:

[0250] Establishing unit 901, used to establish a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, one of the directed edges points to the second node through the first node, and the directed edge is used to characterize that the first node is an upstream node of the second node;

[0251] A determination unit 902 is used to determine a first core node in the first relationship graph based on the at least two nodes and the at least one directed edge; the first core node is a node in the first relationship graph whose importance satisfies a first condition;

[0252] A comparison unit 903 is used to compare the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state;

[0253] The processing unit 904 is used to obtain a target comparison result for the database based on the first comparison result; in the target comparison result, the sub-comparison result of each of the data models has the same content or different content.

[0254] In some embodiments, the establishing unit 901 is specifically configured to:

[0255] For each of the at least two data models, a node pointing to the data model is established to obtain the at least two nodes;

[0256] For the upstream and downstream relationship between the data models, a directed edge is established from the first node to the second node to obtain the at least one directed edge; the first node is used to point to the first data model, and the second node is used to point to the second data model; the upstream and downstream relationship is used to represent that the first data in the second data model is obtained according to the first data model;

[0257] The first relationship graph is obtained based on at least the at least two nodes and the at least one directed edge.

[0258] In some embodiments, the determining unit 902 is specifically configured to:

[0259] Based on the at least two nodes and the at least one directed edge, determining the degree centrality of each of the at least two nodes; the degree centrality is used to characterize the number of downstream nodes of the node in the first relationship graph;

[0260] Based on the at least two nodes and the at least one directed edge, determining the betweenness centrality of each of the at least two nodes; the betweenness centrality is used to characterize the number of times the node appears on the shortest path of the first relationship graph;

[0261] Determining the importance of each of the at least two nodes; the importance of the node is related to the degree centrality of the node and the betweenness centrality of the node;

[0262] Among the at least two nodes, the node whose importance satisfies the first condition is determined as the first core node.

[0263] In some embodiments, the determining unit 902 is further configured to:

[0264] For each of the nodes, calculating the degree centrality of the node according to a first formula;

[0265] Among them, the first formula includes The i represents the node; the C D (ti ) represents the degree centrality of the node; the t j For the t i The downstream node of x ij Indicates the existence of i Point to t j The directed edge of .

[0266] In some embodiments, the determining unit 902 is further configured to:

[0267] For each of the nodes, calculating the betweenness centrality of the node according to a second formula;

[0268] The second formula includes The i represents the node; the t j , the k is the number of nodes except the t i Nodes outside the C B (t i ) represents the betweenness centrality of the node, and p(j, i, k) represents the t i In the said j to the k The number of shortest paths, Indicates that in the first relationship graph, the t j to the k The number of all shortest paths.

[0269] In some embodiments, the determining unit 902 is further configured to:

[0270] For each of the nodes, calculating the importance of the node according to a third formula;

[0271] The third formula includes: The i represents the node; the TI(t i ) represents the importance of the node; the C B (t i ) represents the betweenness centrality of the node; represents the mean value of the betweenness centrality of all nodes in the first relationship graph, the sd(C B (t)) represents the standard deviation of the betweenness centrality of all nodes in the first relationship graph; U (t i ) represents the degree centrality of the node; represents the mean of the degree centrality of all nodes in the first relationship graph; the sd(C U(t)) represents the standard deviation of the degree centrality of all nodes in the first relationship graph.

[0272] In some embodiments, the comparison unit 903 is specifically used to:

[0273] Comparing the data model pointed to by the first core node to obtain a first sub-comparison result;

[0274] If the first sub-comparison result is identical in content, based on the first sub-comparison result, the sub-comparison result of the data model pointed to by the upstream node of the first core node in the first relationship graph is updated to obtain the first comparison result;

[0275] If the first sub-comparison result is different content, obtain different fields, and update the sub-comparison result of the data model pointed to by the node related to the different fields in the downstream node of the first core node in the first relationship graph to different content to obtain the first comparison result.

[0276] In some embodiments, the processing unit 904 is specifically configured to:

[0277] Determining whether the first comparison result satisfies the sub-comparison result of each of the data models as having the same content or different content;

[0278] If the first comparison result cannot satisfy that the sub-comparison result of each of the data models is the same content or different content; determine a second relationship graph based on the first core node;

[0279] Determine a second core node in the second relationship graph, and compare the data model pointed to by the second core node to obtain a second comparison result for the database;

[0280] The second comparison result is used as the new first comparison result, and whether the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, until the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, and the new first comparison result that satisfies the sub-comparison result of each data model that the content is the same or the content is different is used as the target comparison result.

[0281] In some embodiments, the processing unit 904 is further configured to:

[0282] If the first sub-comparison result of the data model pointed to by the first core node is the same in content, the downstream nodes of the first core node and the directed edges between the downstream nodes in the first relationship graph are used as the second relationship graph;

[0283] If the first sub-comparison result is different in content, the upstream node of the first core node and the directed edges between the upstream nodes in the first relationship graph are used as the second relationship graph.

[0284] It should be noted that the data processing device provided in the embodiment of the present application includes the various units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Micro Processor Unit), a digital signal processor (DSP, Digital Signal Processor) or a field programmable gate array (FPGA, Field-Programmable Gate Array), etc.

[0285] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0286] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0287] In the third aspect, to implement the above-mentioned data processing method, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the data processing method provided in the above-mentioned embodiment are implemented.

[0288] Combine the following Fig.10 The electronic device 100 shown is a block diagram of the electronic device.

[0289] In one example, the electronic device 100 may be the above-mentioned data processing terminal. Fig.10 As shown, the electronic device 100 includes: a processor 1001, at least one communication bus 1002, a user interface 1003, at least one external communication interface 1004 and a memory 1005. The communication bus 1002 is configured to achieve connection and communication between these components. The user interface 1003 may include a display screen, and the external communication interface 1004 may include a standard wired interface and a wireless interface.

[0290] The memory 1005 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or processed by the processor 1001 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0291] In a fourth aspect, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the above embodiment are implemented.

[0292] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0293] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0294] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0295] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0296] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0297] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0298] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0299] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0300] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: A first relationship graph is established based on at least two data models included in the database and the upstream and downstream relationships between the data models; the first relationship graph includes at least two nodes and at least one directed edge; one of the nodes is used to uniquely point to one of the at least two data models, one of the directed edges points to the second node through the first node, and the directed edge is used to indicate that the first node is an upstream node of the second node; Based on the at least two nodes and the at least one directed edge, determining a first core node in the first relationship graph; the first core node is a node in the first relationship graph whose importance satisfies a first condition; Compare the data model pointed to by the first core node to obtain a first comparison result for the database; the comparison is used to determine whether the content of the data model pointed to by the first core node is the same in the first state and the second state; Based on the first comparison result, a target comparison result for the database is obtained; in the target comparison result, the sub-comparison result of each data model has the same content or different content.

2. The method according to claim 1, characterized in that The step of establishing a first relationship graph based on at least two data models included in the database and the upstream and downstream relationships between the data models includes: For each of the at least two data models, a node pointing to the data model is established to obtain the at least two nodes; For the upstream and downstream relationship between the data models, a directed edge is established from the first node to the second node to obtain the at least one directed edge; the first node is used to point to the first data model, and the second node is used to point to the second data model; the upstream and downstream relationship is used to represent that the first data in the second data model is obtained according to the first data model; The first relationship graph is obtained based on at least the at least two nodes and the at least one directed edge.

3. The method according to claim 1, characterized in that The determining, based on the at least two nodes and the at least one directed edge, a first core node in the first relationship graph includes: Based on the at least two nodes and the at least one directed edge, determining the degree centrality of each of the at least two nodes; the degree centrality is used to characterize the number of downstream nodes of the node in the first relationship graph; Based on the at least two nodes and the at least one directed edge, determining the betweenness centrality of each of the at least two nodes; the betweenness centrality is used to characterize the number of times the node appears on the shortest path of the first relationship graph; Determining the importance of each of the at least two nodes; the importance of the node is related to the degree centrality of the node and the betweenness centrality of the node; Among the at least two nodes, the node whose importance satisfies the first condition is determined as the first core node.

4. The method according to claim 3, characterized in that: The step of determining the degree centrality of each of the at least two nodes based on the at least two nodes and the at least one directed edge comprises: For each of the nodes, calculating the degree centrality of the node according to a first formula; Among them, the first formula includes ; represents the node; represents the degree centrality of the node; Indicates existence point to The directed edge of For the downstream node.

5. The method according to claim 3, characterized in that: The determining, based on the at least two nodes and the at least one directed edge, the betweenness centrality of each of the at least two nodes comprises: For each of the nodes, calculating the betweenness centrality of the node according to a second formula; The second formula includes ; represents the node; represents the betweenness centrality of the node, Indicates the In arrive The number of shortest paths, In the first relationship graph, to the The number of all shortest paths, , The at least two nodes except the Nodes outside.

6. The method according to claim 3, characterized in that The determining the importance of each of the at least two nodes comprises: For each of the nodes, calculating the importance of the node according to a third formula; The third formula includes: ; represents the node; Indicates the importance of the node; represents the betweenness centrality of the node; represents the mean value of the betweenness centrality of all nodes in the first relationship graph, represents the standard deviation of the betweenness centrality of all nodes in the first relationship graph; represents the degree centrality of the node; represents the mean value of the degree centrality of all nodes in the first relationship graph; Represents the standard deviation of the degree centrality of all nodes in the first relationship graph.

7. The method according to claim 1, characterized in that The comparing the data model pointed to by the first core node to obtain a first comparison result for the database includes: Comparing the data model pointed to by the first core node to obtain a first sub-comparison result; If the first sub-comparison result is identical in content, based on the first sub-comparison result, the sub-comparison result of the data model pointed to by the upstream node of the first core node in the first relationship graph is updated to obtain the first comparison result; If the first sub-comparison result is different content, obtain different fields, and update the sub-comparison result of the data model pointed to by the node related to the different fields in the downstream node of the first core node in the first relationship graph to different content to obtain the first comparison result.

8. The method according to claim 1, characterized in that: The obtaining, based on the first comparison result, a target comparison result for the database, comprises: Determining whether the first comparison result satisfies the sub-comparison result of each of the data models as having the same content or different content; If the first comparison result cannot satisfy that the sub-comparison result of each of the data models is the same content or different content; determine a second relationship graph based on the first core node; Determine a second core node in the second relationship graph, and compare the data model pointed to by the second core node to obtain a second comparison result for the database; The second comparison result is used as the new first comparison result, and whether the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, until the new first comparison result satisfies the sub-comparison result of each data model that the content is the same or the content is different, and the new first comparison result that satisfies the sub-comparison result of each data model that the content is the same or the content is different is used as the target comparison result.

9. The method according to claim 8, characterized in that The determining a second relationship graph based on the first core node includes: If the first sub-comparison result of the data model pointed to by the first core node is the same in content, the downstream nodes of the first core node and the directed edges between the downstream nodes in the first relationship graph are used as the second relationship graph; If the first sub-comparison result is different in content, the upstream node of the first core node and the directed edges between the upstream nodes in the first relationship graph are used as the second relationship graph.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the data processing method according to any one of claims 1 to 9 when executing the program.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 9 is implemented.

Images