Bloodline data reading and writing method, device, electronic device and readable storage medium
By constructing two graph database collections and managing their marking states, the problem of low blood-related reading and writing efficiency is solved, and unsensed smooth switching and efficient query are achieved, and query interrupts are avoided.
Patent Information
- Application Number
- CN202111272401.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, blood-related reading and writing efficiency is low, especially when there are many upstream or downstream nodes. Recursive query results in too long time, low performance, and looping problems and memory overflow risks.
Build a collection containing at least two graph databases, store real-time blood data in one graph database through marking state management, and query it from another graph database during query, realizing unconscious and smooth switching of data, avoiding writing and Query conflict.
It improves the efficiency of blood-related data query, avoids interruptions in business query, reduces the number of queries, and improves the performance of data query.
Smart Images

Figure CN113918571B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and computer-readable storage medium for reading and writing bloodline data. Background Art
[0002] The business scenario in the metadata system is to display the blood relationship between tables. By displaying the upstream or downstream of the current table in the form of a tree on the front end, there are very important application scenarios. For example, by querying the blood relationship to determine the real-time execution information of the actual job, the job can be monitored and warned in real time; it can help relevant personnel quickly locate the jobs that may have problems, prevent business data delays, and affect the use of the systems of various business parties.
[0003] In the prior art, the reading and writing of blood relationship is to directly store the blood relationship table of blood relationship between tables in the relational database table, wherein each record in the blood relationship table stores the current library table and the relationship of the upstream library table. When querying the multi-level upstream relationship of a table or the multi-level downstream relationship of a table, a recursive query is used to query the upstream or downstream of each node. In this way, when the number of nodes is relatively large, a query is made for each node, which takes too long and has low performance. It is also necessary to deal with the problem of loops. The presence of loops will cause an infinite loop when the front end constructs a tree structure, resulting in memory overflow. The main disadvantage of this processing method is that when there are many upstream or downstream nodes, the query efficiency is low. Summary of the invention
[0004] The present invention provides a method, device, equipment and storage medium for reading and writing bloodline data, the main purpose of which is to solve the problem of low bloodline data query efficiency.
[0005] To achieve the above purpose, the present invention provides a method for reading and writing bloodline data, comprising:
[0006] Constructing a graph database set including at least two graph databases, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state;
[0007] When the real-time blood relationship data is acquired, the real-time blood relationship data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state;
[0008] When a lineage data query request is received, the marking status of the graph database in the graph database set is determined, and the found lineage data is obtained from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, the new real-time lineage data is stored in the second graph database marked as inactive, and the mark of the first graph database is changed to inactive again, and the mark of the second graph database is changed to active.
[0009] Optionally, when the real-time blood relationship data is acquired, storing the real-time blood relationship data in the first graph database includes:
[0010] Obtain a blood relationship table containing real-time blood relationship data updated within a preset period;
[0011] Determining the marking status of a database in the graph database set;
[0012] Determining that the first graph database marked as inactive is a standby graph database;
[0013] The data in the backup map database is cleared, and the real-time blood relationship data in the blood relationship table is stored in the backup map database.
[0014] Optionally,
[0015] The storing of the real-time blood relationship data in the blood relationship table into the backup map database comprises:
[0016] Extracting table fields in the kinship table and reference relationships between the table fields;
[0017] The table fields are taken as nodes, and the reference relationships are taken as relationship edges of the nodes;
[0018] The nodes and the relationship edges corresponding to the nodes are summarized to obtain a blood relationship graph, and all blood relationship graphs are stored in the backup graph database.
[0019] Optionally, acquiring the found bloodline data from the first graph database marked as active according to the bloodline data query request includes:
[0020] Parse the query field in the blood relationship data query request, query the blood relationship graph in the first graph database marked as active according to the query field, calculate the clustering coefficient of the nodes in the queried blood relationship graph, and generate a query document for the blood relationship data query request according to the clustering coefficient;
[0021] Calculating the average path length between any two query nodes in the query document, and supplementing the query document with nodes by random addition to generate a supplementary document;
[0022] The supplementary document is modified by nodes to generate a modified document, and the modified document is parsed to obtain the bloodline data.
[0023] Optionally, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state, comprises:
[0024] Determine the usage status of at least two graph databases in the graph database set according to a preset status field, wherein the usage status includes unused and in use;
[0025] Determine that the graph database whose status field is the first field is a first graph database, the usage status of the first graph database is unused, and mark the first graph database as inactive;
[0026] Determine that the graph database whose status field is the second field is the second graph database, the usage status of the second graph database is in use, and mark the second graph database as active.
[0027] Optionally, the constructing a graph database set including at least two graph databases includes:
[0028] Obtain at least two graph database installation packages and decompress them to obtain at least two graph database decompressed files;
[0029] Obtaining an access mode configuration and a switching mode configuration of a graph database, and generating a configuration file based on the access mode configuration and the switching mode configuration;
[0030] Add the configuration file to the decompressed file, and start the added configuration file to obtain the graph database set.
[0031] Optionally,
[0032] The calculating of the clustering coefficient of the nodes in the queried blood relationship graph includes:
[0033] The clustering coefficient of the node in the queried blood relationship graph is calculated using the following clustering coefficient calculation formula:
[0034]
[0035] Among them, C i Represents node v i The clustering coefficient, k i Represents node v i The number of adjacent nodes, e i Indicates the number of actual relationship edges between adjacent nodes, k i (ki -1) indicates the upper limit of the number of relationship edges between adjacent nodes.
[0036] In order to solve the above problems, the present invention also provides a bloodline data reading and writing device, the device comprising:
[0037] A graph database set construction module, used to construct a graph database set including at least two graph databases, mark a first graph database in the graph database set as an inactive state, and mark a second graph database in the graph database set as an active state;
[0038] A blood relationship data storage module, for storing the real-time blood relationship data in the first graph database when the real-time blood relationship data is acquired, and changing the mark of the first graph database to an active state, and changing the mark of the second graph database to an inactive state;
[0039] A lineage data query module is used to determine the marking status of the graph database in the graph database set when a lineage data query request is received, obtain the found lineage data from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, store the new real-time lineage data in the second graph database marked as inactive, change the mark of the first graph database to inactive again, and change the mark of the second graph database to active.
[0040] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0041] a memory storing at least one computer program; and
[0042] The processor executes the computer program stored in the memory to implement the above-mentioned bloodline data reading and writing method.
[0043] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned bloodline data reading and writing method.
[0044] The present invention constructs a graph database set including at least two graph databases, marks the first graph database in the graph database set as inactive, and marks the second graph database in the graph database set as active, stores real-time blood relationship data according to the marking status of the databases in the graph database set, readjusts the marking status of the first graph database and the second graph database, performs data query according to the graph database after the marking status is adjusted, and realizes the imperceptible and smooth switching of data storage and query in the graph database by continuously adjusting the marking status of the graph database each time data is stored and queried, so that data is read from the graph database with active marking each time when querying, and the graph database with inactive marking status can be written when writing, so that data reading and writing will not conflict, thus preventing business query interruption and improving query efficiency. Moreover, compared with the use of a relational database to query blood relationship data, it is necessary to query each node and then merge the query results, with many queries and low efficiency. By using a graph database, there is no need to query each node once, and the node data of all nodes in the required graph database can be queried at one time, greatly improving query efficiency. Therefore, the bloodline data reading and writing method, device, electronic device and computer-readable storage medium proposed in the present invention can solve the problem of low bloodline data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of a flow chart of a method for reading and writing bloodline data provided by an embodiment of the present invention;
[0046] Figure 2 A functional module diagram of a bloodline data reading and writing device provided by an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of the structure of an electronic device for implementing the bloodline data reading and writing method provided in one embodiment of the present invention.
[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0050] The embodiment of the present application provides a method for reading and writing bloodline data. The execution subject of the bloodline data reading and writing method includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the bloodline data reading and writing method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
[0051] Reference Figure 1 FIG. 1 is a flow chart of a method for reading and writing bloodline data provided by an embodiment of the present invention. In this embodiment, the method for reading and writing bloodline data includes:
[0052] S1. Construct a graph database set including at least two graph databases, mark a first graph database in the graph database set as inactive, and mark a second graph database in the graph database set as active.
[0053] In the embodiment of the present invention, the graph database is a non-relational database (NoSQL database), which uses graph theory to store relationship information between entities. Compared with relational databases storing "relational" data (such as blood data), it is often necessary to create an association table to record the many-to-many relationship between different entities, and these association tables are often not used to record information. If there are multiple relationships between two entities, then multiple association tables need to be created between them, making the query complex, slow, and beyond expectations, and the unique design of the graph database just makes up for this defect. The graph databases in the graph database set can be: Neo4j graph database, FlockDB graph database, AllegroGrap graph database, etc.
[0054] Specifically, the constructing includes a graph database set of at least two graph databases, including:
[0055] Obtain at least two graph database installation packages and decompress them to obtain at least two graph database decompressed files;
[0056] Obtaining an access mode configuration and a switching mode configuration of a graph database, and generating a configuration file based on the access mode configuration and the switching mode configuration;
[0057] Add the configuration file to the decompressed file, and start the added configuration file to obtain the graph database set.
[0058] In an embodiment of the present invention, the graph database installation package may be a Neo4j graph database installation package. The configuration file includes access mode configuration and switching mode configuration, wherein the access mode configuration includes: 1. Client link access; 2. Program API interface access; 3. HTTP access; Switching mode configuration refers to setting the priority of the two graph databases according to the marking status of the two graph databases, for example, setting the database with active marking as the main graph database (i.e., priority query use), and setting the database with inactive marking as the backup graph database (i.e., priority storage use).
[0059] In detail, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state, includes:
[0060] Determine the usage status of at least two graph databases in the graph database set according to a preset status field, wherein the usage status includes unused and in use;
[0061] Determine that the graph database whose status field is the first field is a first graph database, the usage status of the first graph database is unused, and mark the first graph database as inactive;
[0062] Determine that the graph database whose status field is the second field is the second graph database, the usage status of the second graph database is in use, and mark the second graph database as active.
[0063] In an embodiment of the present invention, for example, the use status can be determined by a preset state field state_desc field, wherein the first field offline represents unused, and the second field online represents used. The inactive state can be marked as 0 to indicate which graph database has priority for data storage this time, and the active state can be marked as 1 to indicate which graph database has priority for data query this time.
[0064] S2. When the real-time blood relationship data is obtained, the real-time blood relationship data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state.
[0065] In an embodiment of the present invention, the real-time lineage data refers to a logical relationship formed during the generation, fusion, circulation, and disappearance of data. For example, the real-time lineage data includes the logical relationship between the business system, the database used by the business system, the current library table in the database, and the upstream and downstream library tables.
[0066] Specifically, when the real-time blood relationship data is acquired, storing the real-time blood relationship data in the first graph database includes:
[0067] Obtain a blood relationship table containing real-time blood relationship data updated within a preset period;
[0068] Determining the marking status of a database in the graph database set;
[0069] Determining that the first graph database marked as inactive is a standby graph database;
[0070] The data in the backup map database is cleared, and the real-time blood relationship data in the blood relationship table is stored in the backup map database.
[0071] In the embodiment of the present invention, each record in the lineage table stores the upstream relationship of the current library table at multiple levels, and the downstream relationship of the downstream library table at multiple levels. The preset period may be one day.
[0072] In detail, storing the real-time blood relationship data in the blood relationship table into the backup map database includes:
[0073] Extracting table fields in the kinship table and reference relationships between the table fields;
[0074] The table fields are taken as nodes, and the reference relationships are taken as relationship edges of the nodes;
[0075] The nodes and the relationship edges corresponding to the nodes are summarized to obtain a blood relationship graph, and all blood relationship graphs are stored in the backup graph database.
[0076] In the embodiment of the present invention, the graph database is stored in a graph structure, mainly storing nodes, relationships, etc. Among them, nodes represent entities, and relationship edges represent relationships between entities. Edges can be directed or undirected. The storage of the graph is specially optimized, and nodes and relationship edges are stored separately, which helps to improve the query efficiency of the graph database.
[0077] Specifically, the changing the mark of the first graph database to an active state, and the changing the mark of the second graph database to an inactive state, includes:
[0078] The marking state of the first graph database is set to an active state, so that the first graph database serves as a main graph database, and the marking state of the second graph database is set to an inactive state, so that the second graph database serves as a backup graph database.
[0079] Optionally, at this time, the first graph database is used as the main graph database for the next data query, and the second graph database is used as the backup graph database for the next data storage.
[0080] S3. When a lineage data query request is received, determine the marking status of the graph database in the graph database set, obtain the found lineage data from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, store the new real-time lineage data in the second graph database marked as inactive, change the mark of the first graph database to inactive again, and change the mark of the second graph database to active.
[0081] In the embodiment of the present invention, since the first graph database whose marking state is adjusted is the main graph database, that is, the query database, the bloodline data query request preferentially queries data from the first graph database.
[0082] Specifically, acquiring the found bloodline data from the first graph database marked as active according to the bloodline data query request includes:
[0083] Parse the query field in the blood relationship data query request, query the blood relationship graph in the first graph database marked as active according to the query field, calculate the clustering coefficient of the nodes in the queried blood relationship graph, and generate a query document for the blood relationship data query request according to the clustering coefficient;
[0084] Calculating the average path length between any two query nodes in the query document, and supplementing the query document with nodes by random addition to generate a supplementary document;
[0085] The supplementary document is modified by nodes to generate a modified document, and the modified document is parsed to obtain the bloodline data.
[0086] In the embodiment of the present invention, the calculation of the clustering coefficient of the nodes in the queried blood relationship graph includes:
[0087] The clustering coefficient of the node in the queried blood relationship graph is calculated using the following clustering coefficient calculation formula:
[0088]
[0089] Among them, Ci Represents node v i The clustering coefficient, k i Represents node v i The number of adjacent nodes, e i Indicates the number of actual relationship edges between adjacent nodes, k i (k i -1) indicates the upper limit of the number of relationship edges between adjacent nodes.
[0090] When the clustering coefficient C i When it is greater than a preset first threshold, it indicates that the node v i The closeness between the node and the adjacent node is high, then the node v is retained. i As a query node, when the clustering coefficient is less than a preset first threshold, it means that the node v i If the closeness between the node and the adjacent node is low, the node v is deleted. i , by deleting the corresponding nodes, the query document is generated, and the first threshold preset in the present invention may be 0.5.
[0091] In an optional embodiment of the present invention, the calculating the average path length between any two query nodes in the query document includes:
[0092] The average path is calculated by the following path calculation formula:
[0093]
[0094] Among them, l ij Represents any two query nodes v in the query document i ′ and v j ′, N represents the total number of nodes in the query document, d ij Represents the query node v i ′ and v j The shortest path length between ′.
[0095] The present invention uses a random increase method to perform path search and generate supplementary documents. For example, in the query node v i ′ and v j ′ Randomly increase |(2-l ij )×2| nodes, the random increase method refers to: at the query node v i ′ and v j ′Add |(2-l ij )×2| nodes, or at the query node v i ′ and v j ′Add |(2-l ij)×2| nodes, and at the query node v i ′ and v j ′ is connected into a new path, and the supplementary node of the query document is obtained according to the random addition method to generate the supplementary document.
[0096] Furthermore, by traversing the number of nodes in the supplementary document, and calculating the number of nodes v between any two nodes in the supplementary document i ″ and v j The betweenness C between nodes k B (k) The C B The calculation formula of (k) is as follows:
[0097]
[0098] Wherein, V represents the set of all nodes in the supplementary document; δ ij Represents the node v i ″ and v j The number of shortest paths between ij (k) represents the node v i ″ and v j ″The number of shortest paths passing through node k, when the C B (k) is less than a preset second threshold, the node k and the node v are connected by a connection command. i ″ or v j ″ is merged to complete the modification operation of the node k. After the modification operation is performed on all the nodes in the supplementary document, the modified document is generated. The preset second threshold value may be 0.25.
[0099] In the embodiment of the present invention, compared with using a relational database to query bloodline data, which requires querying each node and then merging the query results, with a large number of queries and low efficiency, a graph database is used, and there is no need to query each node once. All required node data can be queried at one time, greatly improving query efficiency.
[0100] In an optional embodiment of the present invention, due to the frequent changes in the lineage relationships between various systems, databases, and database tables in a production environment. If the fully updated lineage data is updated in a graph database each time, it will lead to low database query efficiency and may cause interruptions in business access and data query. When new real-time lineage data for the next cycle is obtained, the new real-time lineage data is stored in an inactive marked graph database, and after each real-time lineage data storage is completed, the marking status of the two graph databases is readjusted to facilitate the storage of real-time lineage data in subsequent cycles. At the same time, by using two graph databases to continuously switch between the main (query) and backup (storage) graph databases, smooth switching of graph databases without perception can be achieved, which improves data query efficiency and prevents access interruptions to business queries.
[0101] The present invention constructs a graph database set including at least two graph databases, marks the first graph database in the graph database set as inactive, and marks the second graph database in the graph database set as active, stores real-time blood relationship data according to the marking status of the databases in the graph database set, readjusts the marking status of the first graph database and the second graph database, performs data query according to the graph database after the marking status is adjusted, and realizes the imperceptible and smooth switching of data storage and query in the graph database by continuously adjusting the marking status of the graph database each time data is stored and queried, so that data is read from the graph database with active marking each time when querying, and the graph database with inactive marking status can be written when writing, so that data reading and writing will not conflict, thus preventing business query interruption and improving query efficiency. Moreover, compared with the use of a relational database to query blood relationship data, it is necessary to query each node and then merge the query results, with many queries and low efficiency. By using a graph database, there is no need to query each node once, and the node data of all nodes in the required graph database can be queried at one time, greatly improving query efficiency. Therefore, the blood relationship data reading and writing method proposed in the present invention can solve the problem of low blood relationship data query efficiency.
[0102] like Figure 2 , which is a functional module diagram of a bloodline data reading and writing device provided in one embodiment of the present invention.
[0103] The bloodline data reading and writing device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the bloodline data reading and writing device 100 may include a graph database set construction module 101, a bloodline data storage module 102 and a bloodline data query module 103. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.
[0104] In this embodiment, the functions of each module / unit are as follows:
[0105] The graph database set construction module 101 is used to construct a graph database set including at least two graph databases, mark a first graph database in the graph database set as an inactive state, and mark a second graph database in the graph database set as an active state;
[0106] The blood relationship data storage module 102 is used for storing the real-time blood relationship data in the first graph database when acquiring the real-time blood relationship data, and changing the mark of the first graph database to an active state, and changing the mark of the second graph database to an inactive state;
[0107] The lineage data query module 103 is used to determine the marking status of the graph database in the graph database set when receiving a lineage data query request, obtain the found lineage data from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, store the new real-time lineage data in the second graph database marked as inactive, change the mark of the first graph database to inactive again, and change the mark of the second graph database to active.
[0108] In detail, the specific implementation of each module of the bloodline data reading and writing device 100 is as follows:
[0109] Step 1: construct a graph database set including at least two graph databases, mark a first graph database in the graph database set as inactive, and mark a second graph database in the graph database set as active.
[0110] In the embodiment of the present invention, the graph database is a non-relational database (NoSQL database), which uses graph theory to store relationship information between entities. Compared with relational databases storing "relational" data (such as blood data), it is often necessary to create an association table to record the many-to-many relationship between different entities, and these association tables are often not used to record information. If there are multiple relationships between two entities, then multiple association tables need to be created between them, making the query complex, slow, and beyond expectations, and the unique design of the graph database just makes up for this defect. The graph databases in the graph database set can be: Neo4j graph database, FlockDB graph database, AllegroGrap graph database, etc.
[0111] Specifically, the constructing includes a graph database set of at least two graph databases, including:
[0112] Obtain at least two graph database installation packages and decompress them to obtain at least two graph database decompressed files;
[0113] Obtaining an access mode configuration and a switching mode configuration of a graph database, and generating a configuration file based on the access mode configuration and the switching mode configuration;
[0114] Add the configuration file to the decompressed file, and start the added configuration file to obtain the graph database set.
[0115] In an embodiment of the present invention, the graph database installation package may be a Neo4j graph database installation package. The configuration file includes access mode configuration and switching mode configuration, wherein the access mode configuration includes: 1. Client link access; 2. Program API interface access; 3. HTTP access; Switching mode configuration refers to setting the priority of the two graph databases according to the marking status of the two graph databases, for example, setting the database with active marking as the main graph database (i.e., priority query use), and setting the database with inactive marking as the backup graph database (i.e., priority storage use).
[0116] In detail, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state, includes:
[0117] Determine the usage status of at least two graph databases in the graph database set according to a preset status field, wherein the usage status includes unused and in use;
[0118] Determine that the graph database whose status field is the first field is a first graph database, the usage status of the first graph database is unused, and mark the first graph database as inactive;
[0119] Determine that the graph database whose status field is the second field is the second graph database, the usage status of the second graph database is in use, and mark the second graph database as active.
[0120] In an embodiment of the present invention, for example, the use status can be determined by a preset state field state_desc field, wherein the first field offline represents unused, and the second field online represents used. The inactive state can be marked as 0 to indicate which graph database has priority for data storage this time, and the active state can be marked as 1 to indicate which graph database has priority for data query this time.
[0121] Step 2: When the real-time bloodline data is obtained, the real-time bloodline data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state.
[0122] In an embodiment of the present invention, the real-time lineage data refers to a logical relationship formed during the generation, fusion, circulation, and disappearance of data. For example, the real-time lineage data includes the logical relationship between the business system, the database used by the business system, the current library table in the database, and the upstream and downstream library tables.
[0123] Specifically, when the real-time blood relationship data is acquired, storing the real-time blood relationship data in the first graph database includes:
[0124] Obtain a blood relationship table containing real-time blood relationship data updated within a preset period;
[0125] Determining the marking status of a database in the graph database set;
[0126] Determining that the first graph database marked as inactive is a standby graph database;
[0127] The data in the backup map database is cleared, and the real-time blood relationship data in the blood relationship table is stored in the backup map database.
[0128] In the embodiment of the present invention, each record in the lineage table stores the upstream relationship of the current library table at multiple levels, and the downstream relationship of the downstream library table at multiple levels. The preset period may be one day.
[0129] In detail, storing the real-time blood relationship data in the blood relationship table into the backup map database includes:
[0130] Extracting table fields in the kinship table and reference relationships between the table fields;
[0131] The table fields are taken as nodes, and the reference relationships are taken as relationship edges of the nodes;
[0132] The nodes and the relationship edges corresponding to the nodes are summarized to obtain a blood relationship graph, and all blood relationship graphs are stored in the backup graph database.
[0133] In the embodiment of the present invention, the graph database is stored in a graph structure, mainly storing nodes, relationships, etc. Among them, nodes represent entities, and relationship edges represent relationships between entities. Edges can be directed or undirected. The storage of the graph is specially optimized, and nodes and relationship edges are stored separately, which helps to improve the query efficiency of the graph database.
[0134] Specifically, the changing the mark of the first graph database to an active state, and the changing the mark of the second graph database to an inactive state, includes:
[0135] The marking state of the first graph database is set to an active state, so that the first graph database serves as a main graph database, and the marking state of the second graph database is set to an inactive state, so that the second graph database serves as a backup graph database.
[0136] Optionally, at this time, the first graph database is used as the main graph database for the next data query, and the second graph database is used as the backup graph database for the next data storage.
[0137] Step three: when a lineage data query request is received, determine the marking status of the graph database in the graph database set, obtain the found lineage data from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, store the new real-time lineage data in the second graph database marked as inactive, change the mark of the first graph database to inactive again, and change the mark of the second graph database to active.
[0138] In the embodiment of the present invention, since the first graph database whose marking state is adjusted is the main graph database, that is, the query database, the bloodline data query request preferentially queries data from the first graph database.
[0139] Specifically, acquiring the found bloodline data from the first graph database marked as active according to the bloodline data query request includes:
[0140] Parse the query field in the blood relationship data query request, query the blood relationship graph in the first graph database marked as active according to the query field, calculate the clustering coefficient of the nodes in the queried blood relationship graph, and generate a query document for the blood relationship data query request according to the clustering coefficient;
[0141] Calculating the average path length between any two query nodes in the query document, and supplementing the query document with nodes by random addition to generate a supplementary document;
[0142] The supplementary document is modified by nodes to generate a modified document, and the modified document is parsed to obtain the bloodline data.
[0143] In the embodiment of the present invention,
[0144] The calculating of the clustering coefficient of the nodes in the queried blood relationship graph includes:
[0145] The clustering coefficient of the node in the queried blood relationship graph is calculated using the following clustering coefficient calculation formula:
[0146]
[0147] Among them, C i Represents node v i The clustering coefficient, k i Represents node v i The number of adjacent nodes, e i Indicates the number of actual relationship edges between adjacent nodes, k i (k i -1) indicates the upper limit of the number of relationship edges between adjacent nodes.
[0148] When the clustering coefficient C i When it is greater than a preset first threshold, it indicates that the node v i The closeness between the node and the adjacent node is high, then the node v is retained. i As a query node, when the clustering coefficient is less than a preset first threshold, it means that the node v i If the closeness between the node and the adjacent node is low, the node v is deleted. i , by deleting the corresponding nodes, the query document is generated, and the first threshold preset in the present invention may be 0.5.
[0149] In an optional embodiment of the present invention, the calculating the average path length between any two query nodes in the query document includes:
[0150] The average path is calculated by the following path calculation formula:
[0151]
[0152] Among them, l ij Represents any two query nodes v in the query document i ′ and v j ′, N represents the total number of nodes in the query document, d ij Represents the query node v i ′ and v j The shortest path length between ′.
[0153] The present invention uses a random increase method to perform path search and generate supplementary documents. For example, in the query node v i ′ and v j ′ Randomly increase |(2-l ij )×2| nodes, the random increase method refers to: at the query node v i ′ and v j′Add |(2-l ij )×2| nodes, or at the query node v i ′ and v j ′Add |(2-l ij )×2| nodes, and at the query node v i ′ and v j ′ is connected into a new path, and the supplementary node of the query document is obtained according to the random addition method to generate the supplementary document.
[0154] Furthermore, by traversing the number of nodes in the supplementary document, and calculating the number of nodes v between any two nodes in the supplementary document i ″ and v j The betweenness C between nodes k B (k) The C B The calculation formula of (k) is as follows:
[0155]
[0156] Wherein, V represents the set of all nodes in the supplementary document; δ ij Represents the node v i ″ and v j The number of shortest paths between ij (k) represents the node v i ″ and v j ″The number of shortest paths passing through node k, when the C B (k) is less than a preset second threshold, the node k and the node v are connected by a connection command. i ″ or v j ″ is merged to complete the modification operation of the node k. After the modification operation is performed on all the nodes in the supplementary document, the modified document is generated. The preset second threshold value may be 0.25.
[0157] In the embodiment of the present invention, compared with using a relational database to query bloodline data, which requires querying each node and then merging the query results, with a large number of queries and low efficiency, a graph database is used, and there is no need to query each node once. All required node data can be queried at one time, greatly improving query efficiency.
[0158] In an optional embodiment of the present invention, due to the frequent changes in the lineage relationships between various systems, databases, and database tables in a production environment. If the fully updated lineage data is updated in a graph database each time, it will lead to low database query efficiency and may cause interruptions in business access and data query. When new real-time lineage data for the next cycle is obtained, the new real-time lineage data is stored in an inactive marked graph database, and after each real-time lineage data storage is completed, the marking status of the two graph databases is readjusted to facilitate the storage of real-time lineage data in subsequent cycles. At the same time, by using two graph databases to continuously switch between the main (query) and backup (storage) graph databases, smooth switching of graph databases without perception can be achieved, which improves data query efficiency and prevents access interruptions to business queries.
[0159] The present invention constructs a graph database set including at least two graph databases, marks the first graph database in the graph database set as inactive, and marks the second graph database in the graph database set as active, stores real-time blood relationship data according to the marking status of the databases in the graph database set, readjusts the marking status of the first graph database and the second graph database, performs data query according to the graph database after the marking status is adjusted, and realizes the imperceptible and smooth switching of data storage and query in the graph database by continuously adjusting the marking status of the graph database each time data is stored and queried, so that data is read from the graph database with active marking each time when querying, and the graph database with inactive marking status can be written when writing, so that data reading and writing will not conflict, thus preventing business query interruption and improving query efficiency. Moreover, compared with the use of a relational database to query blood relationship data, it is necessary to query each node and then merge the query results, with many queries and low efficiency. By using a graph database, there is no need to query each node once, and the node data of all nodes in the required graph database can be queried at one time, greatly improving query efficiency. Therefore, the blood relationship data reading and writing device proposed in the present invention can solve the problem of low blood relationship data query efficiency.
[0160] like Figure 3 , is a schematic diagram of the structure of an electronic device for implementing a method for reading and writing bloodline data provided by an embodiment of the present invention.
[0161] The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a bloodline data reading and writing program.
[0162] Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the blood data reading and writing program, but also can be used to temporarily store data that has been output or is to be output.
[0163] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules (such as bloodline data reading and writing programs, etc.) stored in the memory 11, and calling data stored in the memory 11.
[0164] The communication interface 12 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.
[0165] The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 13 may be divided into an address bus, a data bus, a control bus, etc. The bus 13 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0166] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0167] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
[0168] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.
[0169] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.
[0170] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0171] The bloodline data reading and writing program stored in the memory 11 in the electronic device is a combination of multiple instructions. When running in the processor 10, it can achieve:
[0172] Constructing a graph database set including at least two graph databases, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state;
[0173] When the real-time blood relationship data is acquired, the real-time blood relationship data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state;
[0174] When a lineage data query request is received, the marking status of the graph database in the graph database set is determined, and the found lineage data is obtained from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, the new real-time lineage data is stored in the second graph database marked as inactive, and the mark of the first graph database is changed to inactive again, and the mark of the second graph database is changed to active.
[0175] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.
[0176] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0177] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:
[0178] Constructing a graph database set including at least two graph databases, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state;
[0179] When the real-time blood relationship data is acquired, the real-time blood relationship data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state;
[0180] When a lineage data query request is received, the marking status of the graph database in the graph database set is determined, and the found lineage data is obtained from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, the new real-time lineage data is stored in the second graph database marked as inactive, and the mark of the first graph database is changed to inactive again, and the mark of the second graph database is changed to active.
[0181] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0182] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0183] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0184] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0185] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.
[0186] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0187] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0188] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0189] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.
[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for reading and writing bloodline data, characterized in that: The method comprises: Constructing a graph database set including at least two graph databases, marking a first graph database in the graph database set as an inactive state, and marking a second graph database in the graph database set as an active state; When the real-time blood relationship data is acquired, the real-time blood relationship data is stored in the first graph database, and the mark of the first graph database is changed to an active state, and the mark of the second graph database is changed to an inactive state; When a lineage data query request is received, the marking state of the graph database in the graph database set is determined, and the found lineage data is obtained from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, the new real-time lineage data is stored in the second graph database marked as inactive, and the mark of the first graph database is changed to inactive again, and the mark of the second graph database is changed to active; Wherein, the obtaining of the found bloodline data from the first graph database marked as active according to the bloodline data query request comprises: parsing the query field in the bloodline data query request, querying the bloodline relationship graph in the first graph database marked as active according to the query field, calculating the clustering coefficient of the nodes in the queried bloodline relationship graph, and generating a query document for the bloodline data query request according to the clustering coefficient; calculating the average path length between any two query nodes in the query document, and supplementing the query document with nodes by random addition to generate a supplementary document; performing node correction on the supplementary document to generate a corrected document, and parsing the corrected document to obtain the bloodline data; The marking of the first graph database in the graph database set as inactive, and the marking of the second graph database in the graph database set as active, comprises: judging the usage status of at least two graph databases in the graph database set according to a preset status field, wherein the usage status includes unused and in use; determining that the graph database whose status field is the first field is the first graph database, the usage status of the first graph database is unused, and marking the first graph database as inactive; determining that the graph database whose status field is the second field is the second graph database, the usage status of the second graph database is in use, and marking the second graph database as active; The step of calculating the clustering coefficient of the nodes in the queried blood relationship graph includes: calculating the clustering coefficient of the nodes in the queried blood relationship graph using the following clustering coefficient calculation formula: in, Representation Node The clustering coefficient, Representation Node The number of adjacent nodes, Indicates the number of actual relationship edges between adjacent nodes. Indicates the upper limit of the number of relationship edges between adjacent nodes.
2. The method for reading and writing bloodline data as claimed in claim 1, characterized in that: When the real-time blood relationship data is acquired, storing the real-time blood relationship data in the first graph database includes: Obtain a blood relationship table containing real-time blood relationship data updated within a preset period; Determining the marking status of a database in the graph database set; Determining that the first graph database marked as inactive is a standby graph database; The data in the backup map database is cleared, and the real-time blood relationship data in the blood relationship table is stored in the backup map database.
3. The method for reading and writing bloodline data as claimed in claim 2, characterized in that: The storing of the real-time blood relationship data in the blood relationship table into the backup map database comprises: Extracting table fields in the kinship table and reference relationships between the table fields; The table fields are taken as nodes, and the reference relationships are taken as relationship edges of the nodes; The nodes and the relationship edges corresponding to the nodes are summarized to obtain a blood relationship graph, and all blood relationship graphs are stored in the backup graph database.
4. The method for reading and writing bloodline data according to any one of claims 1 to 3, characterized in that: The constructing includes a graph database set comprising at least two graph databases, including: Obtain at least two graph database installation packages and decompress them to obtain at least two graph database decompressed files; Obtaining an access mode configuration and a switching mode configuration of a graph database, and generating a configuration file based on the access mode configuration and the switching mode configuration; Add the configuration file to the decompressed file, and start the added configuration file to obtain the graph database set.
5. A bloodline data reading and writing device, used to implement the bloodline data reading and writing method as claimed in any one of claims 1 to 4, characterized in that: The device comprises: A graph database set construction module, used to construct a graph database set including at least two graph databases, mark a first graph database in the graph database set as an inactive state, and mark a second graph database in the graph database set as an active state; A blood relationship data storage module, for storing the real-time blood relationship data in the first graph database when the real-time blood relationship data is acquired, and changing the mark of the first graph database to an active state, and changing the mark of the second graph database to an inactive state; A lineage data query module is used to determine the marking status of the graph database in the graph database set when a lineage data query request is received, obtain the found lineage data from the first graph database marked as active according to the lineage data query request, and when new real-time lineage data is obtained, store the new real-time lineage data in the second graph database marked as inactive, change the mark of the first graph database to inactive again, and change the mark of the second graph database to active.
6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the bloodline data reading and writing method as described in any one of claims 1 to 4.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for reading and writing bloodline data as described in any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Neo4j-based big data consanguinity management method, system, apparatus and storage medium
CN109446279A
Data blood relationship analysis method, device, equipment and system and readable storage medium
CN109582660A
