Method and device for automatically importing asset data into graph database and electronic equipment

By receiving asset data from different data sources and converting them into the format supported by the graph database, and importing the graph database with the storage template mapping relationship, the data structure inconsistency and error problems when importing the graph database of different types of data carriers is solved, and automated identification and efficient implementation Import.

CN119003633APending Publication Date: 2024-11-22DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411022288.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art cannot effectively import asset data of different types of data carriers into the graph database automatically, resulting in inconsistency in data structures, errors and complexity problems when importing data.

Method used

By receiving asset data from different data sources, converting templates according to preset formats to the file format supported by the graph database, and importing entity objects and association relationships based on the storage template mapping relationship of the target graph database, using conflict detection and correction rules to process Data conflict.

Benefits of technology

It realizes automatic identification and import of different types of data carriers, ensures that the imported data structure is consistent, reduces import errors, and improves the efficiency and accuracy of data import.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003633A_ABST
    Figure CN119003633A_ABST
Patent Text Reader

Abstract

The invention provides a method and a device for automatically importing asset data into a graph database and electronic equipment, aiming at to-be-imported asset data of different data sources, according to a preset format conversion template, the to-be-imported asset data is converted into a to-be-imported graph database file in a file format which the graph database supports to access, and the to-be-imported graph database file is accessed to the to-be-imported graph database file. Then, according to a mapping relation specified by a database storage template in a target graph database, the entity objects in the graph database file to be imported are imported into corresponding storage points in the database storage template, and the incidence relation between the entity objects is imported into corresponding edges in the database storage template. When asset data to be imported into the graph database of different data sources needs to be imported into the graph database, unified conversion of the data is quickly realized, and the data is converted into a file format which is supported to be accessed by the target graph database, so that automatic identification and automatic importing of the graph database data of different types of data carriers are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, device and electronic device for automatically importing asset data into a graph database. Background Art

[0002] With the development of big data technology, asset data has increasingly become the cornerstone of survival for enterprises or individuals. Among them, asset data refers to data assets owned or controlled by individuals or enterprises that can bring future economic benefits to individuals or enterprises and are recorded in physical or virtual form. Among them, asset data can be the sum of various physical assets and virtual assets. Physical assets refer to physical assets such as servers, network equipment, and communication equipment. Virtual assets refer to intangible virtual assets such as domain names, IP addresses, network virtual applications, applets, public accounts, and video accounts.

[0003] In order to facilitate asset data management, it is usually necessary to uniformly store and manage the asset data owned or controlled by individuals or enterprises. In the database field, the midpoints in the graph database represent entities, and the edges between the points represent the relationship between entities. Therefore, in the field of asset data management, the special data storage characteristics of the graph database are often used to store and manage asset data in order to clearly and explicitly record the asset data owned by individuals or enterprises, as well as the internal connections between asset data.

[0004] However, asset data often has different data carriers and data formats. Currently, there is no universal data import method for importing asset data of different types of data carriers into a graph database. Summary of the invention

[0005] In view of this, embodiments of the present application provide a method, device, and electronic device for automatically importing asset data into a graph database, so as to automatically import data of different types of data carriers into the graph database.

[0006] In a first aspect, an embodiment of the present application provides a method for automatically importing asset data into a graph database, wherein the method comprises:

[0007] Receiving asset data to be imported, wherein the asset data to be imported comes from different data sources;

[0008] According to a preset format conversion template, the asset data to be imported is converted into a graph database file to be imported, wherein the format of the graph database file is a file format that the target graph database supports access to;

[0009] According to the mapping relationship specified by the database storage template in the target graph database, the entity objects in the graph database file to be imported are imported into the corresponding points in the database storage template, and the association relationship between the entity objects is imported into the corresponding edges in the database storage template.

[0010] In some embodiments, after the step of receiving the asset data to be imported, the method further includes:

[0011] If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a data label to be processed is generated.

[0012] In some embodiments, converting the asset data to be imported into a graph database file to be imported according to a preset format conversion template includes:

[0013] Parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects;

[0014] According to the writing rules for each entity object and the writing rules for the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported.

[0015] In some embodiments, the graph database file to be imported is a JSON format file.

[0016] In some embodiments, after the step of receiving the asset data to be imported, the method further includes:

[0017] Parsing the asset data to be imported, and obtaining the asset data parsing result;

[0018] According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

[0019] In some embodiments, in the process of importing the entity objects in the graph database file to be imported into the corresponding points in the database storage template according to the mapping relationship specified by the database storage template in the target graph database, and importing the association relationship between the entity objects into the corresponding edges in the database storage template, the method further includes:

[0020] Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database;

[0021] If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule;

[0022] The preset correction rules include: priority rules, merging rules, and confidence rules.

[0023] In a second aspect, the present application provides a device for automatically importing asset data into a graph database, the device comprising:

[0024] An asset data receiving module, used for receiving asset data to be imported, wherein the asset data to be imported comes from different data sources;

[0025] An asset data preprocessing module, used to convert the asset data to be imported into a graph database file to be imported according to a preset format conversion template, wherein the format of the graph database file is a file format that the target graph database supports access to;

[0026] The asset data import module is used to import the entity objects in the graph database file to be imported into the corresponding points in the database storage template according to the mapping relationship specified by the database storage template in the target graph database, and import the association relationship between the entity objects into the corresponding edges in the database storage template.

[0027] In some embodiments, the asset data receiving module is further used to:

[0028] If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a data label to be processed is generated.

[0029] In some embodiments, converting the asset data to be imported into a graph database file to be imported according to a preset format conversion template includes:

[0030] Parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects;

[0031] According to the writing rules for each entity object and the writing rules for the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported.

[0032] In some embodiments, the graph database file to be imported is a JSON format file.

[0033] In some embodiments, the asset data preprocessing module is further used to:

[0034] Parsing the asset data to be imported, and obtaining the asset data parsing result;

[0035] According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

[0036] In some embodiments, the asset data import module is further used to:

[0037] Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database;

[0038] If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule;

[0039] The preset correction rules include: priority rules, merging rules, and confidence rules.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0041] Processor; and

[0042] Memory for storing programs,

[0043] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes the method for automatically importing asset data into a graph database as described in the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method of automatically importing asset data into a graph database as described in the first aspect.

[0045] Beneficial effects of this application:

[0046] The present application provides a method, device and electronic device for automatically importing asset data into a graph database. For asset data to be imported from different data sources, a preset format conversion template is used to convert the asset data to be imported into a graph database file to be imported in a file format that the graph database supports access to. Then, according to the mapping relationship specified by the database storage template in the target graph database, the entity objects in the graph database file to be imported are imported into the corresponding storage points in the database storage template, and the association relationship between the entity objects is imported into the corresponding edge in the database storage template.

[0047] In this way, when asset data to be imported into the graph database from different data sources needs to be imported into the graph database, the data can be quickly converted uniformly according to the preset format conversion template, converted into a file format that the target graph database supports access, and then quickly imported the entity objects in the asset data to be imported into the corresponding points according to the mapping relationship specified by the database storage template, and the association relationship between the entity objects in the asset data to be imported into the corresponding edges. This can realize the automatic identification and automatic import of graph database data of different types of data carriers, effectively ensure that the data structure of the imported data is consistent with the data structure specified by the graph database, and reduce the errors existing in the import of asset data from different data sources into the graph database. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A schematic diagram of a process of automatically importing asset data into a graph database provided in an embodiment of the present application is shown;

[0050] Figure 2 Another schematic diagram of the process of automatically importing asset data into a graph database provided in an embodiment of the present application is shown;

[0051] Figure 3 A schematic diagram of a process of an apparatus for automatically importing asset data into a graph database provided in an embodiment of the present application is shown;

[0052] Figure 4 A schematic diagram of a logical structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0054] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0055] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0056] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0057] As described in the background technology, graph databases are used to store externally imported data. Existing graph database data import solutions include: solutions based on files, existing databases, and real-time data streams. However, the data formats transmitted to graph databases by different types of data carriers are not the same. Existing technologies cannot guarantee efficient and accurate data import into graph databases when data sources are diverse. The reasons are as follows:

[0058] Specifically, when importing source data into a graph database, different data models must be mapped and converted. This process may lead to some complex and difficult-to-handle situations. If the data model mapping is incorrect or inconsistent, data loss may occur, or incorrect relationships between entities may be established, or data status may be inconsistent, etc.

[0059] In view of this, in a first aspect, the present application provides a method for automatically importing asset data into a graph database, which is applied to any electronic device that has the function of automatically importing asset data into a graph database, including but not limited to a personal mobile terminal, a computer or a server, etc. As a preferred embodiment, the method for automatically importing asset data into a graph database can be applied to a management device of a graph database, which is used to receive data to be imported into the graph database, and the management device can perform data processing on the imported data, including routing, data addition and deletion, etc.

[0060] In some embodiments, the method for automatically importing asset data into a graph database provided by the present application can be as follows: Figure 1 As shown, the following steps are included:

[0061] S11, receiving asset data to be imported, wherein the asset data to be imported comes from different data sources.

[0062] S12: Convert the asset data to be imported into a graph database file to be imported according to a preset format conversion template, wherein the format of the graph database file is a file format that the target graph database supports access to.

[0063] S13. According to the mapping relationship specified by the database storage template in the target graph database, the entity objects in the graph database file to be imported are imported into the corresponding points in the database storage template, and the association relationships between the entity objects are imported into the corresponding edges in the database storage template.

[0064] By selecting an embodiment of the present application, for the asset data to be imported from different data sources, the asset data to be imported is converted into a graph database file to be imported in a file format that the graph database supports access according to a preset format conversion template, and then according to the mapping relationship specified by the database storage template in the target graph database, the entity objects in the graph database file to be imported are imported into the corresponding storage points in the database storage template, and the association relationship between the entity objects is imported into the corresponding edges in the database storage template.

[0065] In this way, when asset data to be imported into the graph database from different data sources needs to be imported into the graph database, the data can be quickly converted uniformly according to the preset format conversion template, converted into a file format that the target graph database supports access, and then quickly imported the entity objects in the asset data to be imported into the corresponding points according to the mapping relationship specified by the database storage template, and the association relationship between the entity objects in the asset data to be imported into the corresponding edges. This can realize the automatic identification and automatic import of graph database data of different types of data carriers, effectively ensure that the data structure of the imported data is consistent with the data structure specified by the graph database, and reduce the errors existing in the import of asset data from different data sources into the graph database.

[0066] The above steps S11 to S13 are described in detail below:

[0067] In the embodiment of the present application, the asset data to be imported refers to the data that needs to be imported into the graph database, including the asset data that has been imported into the graph database but needs to be updated, and also including the data that has never been saved in the graph database. Among them, the sources of the asset data to be imported are different, the types are different, and the data volume is also different. The data source types of the asset data to be imported include one or more of the following:

[0068] Data transmitted from MySQL database, Redis database, local file or preset API (Application Programming Interface) interface.

[0069] Among them, the Mysql database is a relational database, while the graph database is a NoSQL database, which is a non-relational database. Therefore, the data in the Mysql database cannot be directly imported into the graph database without data conversion. Among them, the Redis database (i.e., remote dictionary service) is a non-relational database, which uses the Key-Value (i.e., key-value pair) data storage method, which is also different from the "point-edge" data storage method used by the graph database. Without converting the data in the Redis database, the data in the Redis database cannot be directly imported into the graph database. Therefore, the data conversion required when importing into the graph database will be different depending on the source of the asset data to be imported.

[0070] Similarly, local files are files pre-stored in local memory or cache to be imported into the graph database. The formats of the local files vary, such as txt text format, MySQL database SQL file, or EXCEL file, etc. Due to the different formats of local files, there is also the problem that they cannot be directly imported into the graph database.

[0071] In the embodiment of the present application, the preset API interface is a data interface for the graph database to access data provided by third-party applications. The graph database can obtain asset data provided by third-party applications, web pages and other data sources through the preset API interface. Since different application development organizations organize the required data differently, the data delivered by the preset API interface also has the problem of not being able to be directly imported into the graph database.

[0072] In view of this, in order to enable asset data from different data sources to be imported into the graph database in a unified format, in some embodiments, when executing step S11, the embodiment of the present application may perform preliminary preprocessing on the asset data to be imported through the following steps:

[0073] S11-1. If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a label of the data to be processed is generated.

[0074] As an implementation method, a data source label can be generated by determining the data source of the asset data to be imported. There are many sources of the asset data to be imported, which can come from any device or database that generates asset data. Based on this, when executing step S11-1, different data source labels can be generated for different data sources. Exemplarily, a first data source label is generated for the asset data to be imported imported from a Mysql database, a second data source label is generated for the asset data to be imported imported from a Redis database, and so on. Different data sources have different corresponding data source labels.

[0075] As another implementation, a data processing label may be generated for whether the data format of the asset data to be imported supports direct import into the target graph database, that is, whether the data format of the asset data to be imported belongs to the data format supported by the graph database for import. If the data format of the asset data to be imported does not belong to the data format supported by the graph database for import, then, exemplarily, for data that supports direct import into the graph database, a first data import label is generated, and for data that does not support direct import into the graph database, a second data import label is generated. The second data import label is used to indicate that direct import into the target graph database is not supported.

[0076] Among them, whether the data format of the asset data to be imported supports direct import into the target graph database can be determined according to the version of the target graph database. Then, the data format of the asset data to be imported is matched with the data format of the asset data to be imported. If they can be completely matched, it indicates that the target graph database supports the import of the asset data to be imported. If they cannot be completely matched, it indicates that the target graph database cannot support the direct import of the asset data to be imported.

[0077] In this way, the asset data to be imported can be diverted based on different data tags. For the asset data to be imported without data tags to be processed, step S13 can be directly executed to import it into the graph database. That is, if the data source of the asset data to be imported belongs to the graph database, or the data format of the asset data to be imported belongs to the data format supported by the graph database, step S13 can be directly executed to import it into the graph database.

[0078] For asset data to be imported with data tags to be processed, data conversion is required to convert it into a data format supported by the graph database for import. That is, for data sources that do not belong to the graph database and / or data formats that do not belong to the data formats supported by the graph database for import, data conversion is required to convert it into a data format supported by the graph database for import, and then step S13 is executed to import it into the graph database.

[0079] In some embodiments, Figure 2 As shown, the following steps can be used to convert the asset data to be imported with data tags to be processed into data that can be imported into the graph database:

[0080] S12-1, parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects;

[0081] S12-2. According to the writing rules for each entity object and the writing rules for the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported.

[0082] In the process of executing step S12-1, the corresponding data parsing method can be determined based on the data tag to be processed generated in step S11-1, especially the data source of the asset data to be imported indicated in the data tag to be processed, or the data format of the asset data to be imported, and then the asset data to be imported can be parsed using the determined data parsing method to obtain each content field in the asset data to be imported.

[0083] If the data source of the asset data to be imported is a MySQL database, the data parsing method of the MySQL data (such as the LOAD_FILE() method, the LOAD DATA INFILE() method, etc.) is called to parse the asset data to be imported and obtain the various content fields contained in the asset data to be imported.

[0084] As an implementation method, the asset data to be imported can be parsed through the following steps to obtain each target entity object contained in the asset data to be imported:

[0085] If the received asset data to be imported is a string, each content field contained therein is obtained by regular matching, and each content field corresponds to each entity object.

[0086] If the received asset data to be imported is a list, each list element is extracted therefrom in a loop through the list data extraction method.

[0087] Among them, each content field contains potential entity objects and the relationship between entity objects. Based on this, the use of the embodiment of the present application is equivalent to breaking up the entire asset data to be imported, and then extracting the target content fields required for importing the graph database from each content field according to the needs. The parsing ideas for the asset data to be imported from other data sources can refer to the parsing ideas for the asset data of the Mysql database above, which will not be repeated in this article.

[0088] As an example, for example, the asset data to be imported is a summary table of communication equipment of Department A, which can be shown in Table 1 below:

[0089] Table 1. Summary of communication equipment of Department A

[0090] Serial number Person in charge Device Name Device serial number Department 1 Zhang San Desktop 111 A 2 Li Si printer 222 A 3 Wang Wu Telephone 333 A

[0091] The content fields that can be parsed from Table 1 through the parsing algorithm include: person in charge, device name, device serial number, department to which it belongs, and the actual content corresponding to each content field, such as Zhang San, Li Si, Wang Wu, desktop, printer, telephone, etc. Among them, each content field corresponds to each target entity object. For some relatively comprehensive asset data to be imported, the association relationship between entity objects can be directly extracted from the asset data to be imported. For example, in Table 1 above, it can be clearly determined that Zhang San, Li Si, and Wang Wu are employees of Department A, the desktop with serial number 111 belongs to the communication equipment of Department A, and the person in charge of the desktop with serial number 111 is Zhang San.

[0092] For some asset data to be imported that is relatively lacking in content, it is necessary to combine other asset data related to the asset data to be imported in different data sources to determine the association relationship between each target entity object. For example, if the department column is missing in Table 1 above, the data associated with the target entity object in Table 1 can be analyzed in combination with different data sources such as employee Zhang San's work history information or equipment purchase statistics to determine the department to which each entity object belongs.

[0093] In some embodiments, after step S12-1, the method further includes the following steps:

[0094] Parsing the asset data to be imported, and obtaining the asset data parsing result;

[0095] According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

[0096] Among them, the specific processing ideas for parsing the asset data to be imported can refer to the above-mentioned processing ideas for parsing the asset data to be imported to obtain the target entity object, and obtaining the association relationship between each target entity object. In the embodiment of the present application, the asset data parsing result can also refer to the content field corresponding to the entity object, or the content field corresponding to the association relationship between entity objects.

[0097] In an embodiment of the present application, the asset data to be imported can be a large asset data packet, or it can be just a row of data as shown in Table 1. The size of the asset data to be imported can be large or small according to the actual situation. In the process of executing step S12-1, when parsing the asset data to be imported, the data can be processed one by one. For example, the data in the second row of Table 1 is regarded as a piece of asset data to be imported and parsed to obtain each entity object therein: person in charge Zhang San, device name Desktop, device serial number 111, department A. According to the header settings of Table 1, the content field corresponding to each column should have corresponding content information. However, due to the uneven quality of the asset data to be imported, some content fields are missing or repeated.

[0098] Based on this, in an embodiment of the present application, duplicate items, missing items, and abnormal items in the asset data analysis results can be identified and processed according to preset data screening rules.

[0099] As an implementation method, the preset data screening rules are obtained according to different data source settings. For example, the data source of the purchase list list must contain five pieces of information: purchase items, purchase prices, purchase personnel, sellers, and purchase dates. If there is data missing in the information in the imported purchase list, such as the lack of purchase personnel or seller information, it indicates that there are missing items in the asset data to be imported. If the purchase price of a certain equipment in the imported purchase list is significantly deviated from the historical purchase price, it can be determined that the purchase price is an abnormal item.

[0100] As an implementation method, whether there are missing items can be determined based on the completeness of each piece of asset data to be imported in the same asset data package to be imported.

[0101] As another implementation method, the values ​​of valid fields contained in similar data from different data sources are often similar. Therefore, after comparing data from all different sources, data with particularly large deviations are selected and discarded as anomalies to avoid affecting the accuracy of the overall data. In an embodiment of the present application, duplicate item screening rules can be set to detect whether the values ​​of each required field are unique, such as distinguishing by name or by house number. If there are two data from different sources containing the same house number, they need to be merged to reduce the storage space occupied by duplicate items in the general map database.

[0102] When executing step S12-2, the preset format conversion template can be a preset graph database file, such as a preset JSON (JavaScript Object Notation, a data exchange format) file, wherein the preset JSON file specifies the writing specifications corresponding to each entity object, wherein the filling specifications specify the writing rules for each entity object and the writing rules for the association relationship between each entity object, and the writing rules specify the writing location and specific writing format of each entity object.

[0103] In this way, according to the writing rules of each entity object specified in the preset format conversion template and the writing rules of the association relationship between each entity object, the preset format conversion template can be filled with the values ​​of each target entity object and the association relationship between the target entity objects to generate the graph database file to be imported. In the embodiment of the present application, the graph database file to be imported is a JSON format file, wherein the association relationship belongs to the relationship data corresponding to the JSON file, and the entity object corresponds to the node data in the JSON file.

[0104] In an embodiment of the present application, the target image database will be divided into multiple independent storage modules, each storage module corresponds to a large asset data sub-map database, that is, each large asset data is independent of each other, wherein different asset data sub-map databases correspond to different database storage templates. At this time, when executing step S13, it is necessary to determine the asset data sub-map database to be imported based on the ownership relationship between the asset data to be imported and the asset data divided in the target map database, and then determine the corresponding database storage template.

[0105] In an embodiment of the present application, the database storage template may be a standardized graph data model, which specifies data import rules for importing into the graph database corresponding to the database storage template, wherein one entity object corresponds to one node, and the node also needs to store the attribute information of the node. During the import process of the graph database file to be imported in step S13, a unique identifier or key field may be used to associate each node and the edges between nodes to correctly associate entity objects and the association relationship between entity objects.

[0106] In an embodiment of the present application, specific fields of the data can be divided as attribute information of the node according to actual needs. For example, all basic information of a person can be used as one attribute information, or the name, gender, occupation, etc. can be divided into different attribute information separately. This depends on the actual application and the importance of each attribute to the subsequent application.

[0107] Among them, the data import process is essentially a data mapping process, and at this time, a value needs to be determined as a bridge between each node and node. For example, for example, each person is assigned a mobile phone number, and the mobile phone number is unique. At this time, two people can only contact each other through the mobile phone number, so the relationship between the two people can be represented by the mobile phone number. When the data source is continuously updated in real time, the embodiment of the present application is selected to continuously update and map the entity objects in the database and the relationship between the entity objects according to the received asset data to be imported.

[0108] In some embodiments, during the execution of step S13, the method further includes the following steps:

[0109] Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database;

[0110] If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule;

[0111] The preset correction rules include: priority rules, merging rules, and confidence rules.

[0112] As an implementation method, the preset conflict detection algorithm can be a hash conflict detection algorithm, which performs conflict detection on the imported graph asset data and the asset data stored in the target database. Specifically, a hash function is used to calculate a piece of data to obtain a unique value, and then the value is used as the unique identifier of the data. When the data is different, the value obtained by the hash function calculation is also different. At this time, when the data is imported, the hash values ​​of each data are compared to determine whether there is conflicting data.

[0113] If the asset data to be imported conflicts with the asset data already stored in the target graph database, the priority rules, merge rules, confidence rules and other rules are used to treat the imported asset data, or to modify the existing asset data that conflicts with the imported asset data. Among them, the priority rule can be set according to actual needs, and different data sources have different priorities. Asset data with high priority can be retained and asset data with low priority can be deleted. The merge rule can be to merge two conflicting asset data into one asset data and delete the duplicate items. The confidence rule can be to set confidence for the value of different asset data, similar to the priority rule, and asset data with high confidence is retained first.

[0114] As an implementation method, after the data import is completed, the data can be checked for consistency, wherein the consistency check method can refer to the existing database data consistency check method to ensure the correctness and integrity of the data. Among them, query and verification rules can be used to compare the imported asset data with the received original asset data to determine whether there are inconsistencies or errors between the imported asset data and the received original asset data, and then the inconsistent or erroneous data can be corrected by rolling back the import, repairing the data, or re-importing. Among them, rolling back the import refers to undoing the local import operation and restoring the data in the graph database to the state before the graph data was imported. Repairing data refers to making separate and partial corrections to some erroneous data, and re-importing refers to not undoing the current import operation and trying to re-import until the imported data is consistent with the original asset data.

[0115] In a second aspect, the present application provides a device for automatically importing asset data into a graph database, wherein, Figure 3 As shown, the device 30 comprises:

[0116] The asset data receiving module 301 is used to receive the asset data to be imported, wherein the asset data to be imported comes from different data sources;

[0117] The asset data preprocessing module 302 is used to convert the asset data to be imported into a graph database file to be imported according to a preset format conversion template, wherein the format of the graph database file is a file format that the target graph database supports access to;

[0118] The asset data import module 303 is used to import the entity objects in the graph database file to be imported into the corresponding points in the database storage template according to the mapping relationship specified by the database storage template in the target graph database, and import the association relationship between the entity objects into the corresponding edges in the database storage template.

[0119] In some embodiments, the asset data receiving module 301 is further used to:

[0120] If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a data label to be processed is generated.

[0121] In some embodiments, converting the asset data to be imported into a graph database file to be imported according to a preset format conversion template includes:

[0122] Parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects;

[0123] According to the writing rules for each entity object and the writing rules for the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported.

[0124] In some embodiments, the graph database file to be imported is a JSON format file.

[0125] In some embodiments, the asset data pre-processing module 302 is further used to:

[0126] Parsing the asset data to be imported, and obtaining the asset data parsing result;

[0127] According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

[0128] In some embodiments, the asset data import module 303 is further used to:

[0129] Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database;

[0130] If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule;

[0131] The preset correction rules include: priority rules, merging rules, and confidence rules.

[0132] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0133] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0134] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0135] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0136] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0137] refer to Figure 4, the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0138] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0139] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0140] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned method of automatically importing asset data into a graph database may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to execute the aforementioned method of automatically importing asset data into a graph database by any other appropriate means (e.g., by means of firmware).

[0141] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0142] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A method for automatically importing asset data into a graph database, characterized in that: The method comprises: Receiving asset data to be imported, wherein the asset data to be imported comes from different data sources; According to a preset format conversion template, the asset data to be imported is converted into a graph database file to be imported, wherein the format of the graph database file is a file format that the target graph database supports access to; According to the mapping relationship specified by the database storage template in the target graph database, the entity objects in the graph database file to be imported are imported into the corresponding points in the database storage template, and the association relationship between the entity objects is imported into the corresponding edges in the database storage template.

2. The method according to claim 1, characterized in that After the step of receiving the asset data to be imported, the method further includes: If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a data label to be processed is generated.

3. The method according to claim 2, characterized in that The step of converting the asset data to be imported into a graph database file to be imported according to a preset format conversion template includes: Parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects; According to the writing rules for each entity object and the writing rules for the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported.

4. The method according to any one of claims 1 to 3, characterized in that: The graph database file to be imported is a JSON format file.

5. The method according to claim 1, characterized in that After the step of receiving the asset data to be imported, the method further includes: Parsing the asset data to be imported, and obtaining the asset data parsing result; According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

6. The method according to claim 1, characterized in that In the process of importing the entity objects in the graph database file to be imported into the corresponding points in the database storage template according to the mapping relationship specified by the database storage template in the target graph database, and importing the association relationship between the entity objects into the corresponding edges in the database storage template, the method further includes: Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database; If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule; The preset correction rules include: priority rules, merging rules, and confidence rules.

7. A device for automatically importing asset data into a graph database, characterized in that: The device comprises: An asset data receiving module, used for receiving asset data to be imported, wherein the asset data to be imported comes from different data sources; An asset data preprocessing module, used to convert the asset data to be imported into a graph database file to be imported according to a preset format conversion template, wherein the format of the graph database file is a file format that the target graph database supports access to; The asset data import module is used to import the entity objects in the graph database file to be imported into the corresponding points in the database storage template according to the mapping relationship specified by the database storage template in the target graph database, and import the association relationship between the entity objects into the corresponding edges in the database storage template.

8. The device according to claim 7, characterized in that The asset data receiving module is also used for: If the data source of the asset data to be imported does not belong to the graph database, and / or the data format of the asset data to be imported does not belong to the data format supported for import by the graph database, a data tag to be processed is generated; The step of converting the asset data to be imported into a graph database file to be imported according to a preset format conversion template includes: Parsing the asset data to be imported with the data tag to be processed, obtaining each target entity object contained in the asset data to be imported, and combining other asset data related to the asset data to be imported in the different data sources to determine the association relationship between the target entity objects; According to the writing rules of each entity object and the writing rules of the association relationship between each entity object specified in the preset format conversion template, each target entity object and the association relationship between each target entity object in the asset data to be imported are filled into the preset format conversion template to generate the graph database file to be imported; The graph database file to be imported is a JSON format file.

9. The device according to claim 7, characterized in that The asset data preprocessing module is also used for: Parsing the asset data to be imported, and obtaining the asset data parsing result; According to preset data screening rules, duplicate items, missing items, and abnormal items in the asset data analysis results are identified and processed, wherein the preset data screening rules are based on the preset fields of the asset data to be imported to identify duplicate items, missing items, and abnormal items.

10. The device according to claim 7, characterized in that The asset data import module is also used for: Using a preset conflict detection algorithm, conflict detection is performed on the imported asset data to be imported and the asset data stored in the target graph database; If the asset data to be imported conflicts with the asset data already stored in the target graph database, the asset data to be imported or the already stored asset data conflicting with the asset data to be imported is corrected using a preset correction rule; The preset correction rules include: priority rules, merging rules, and confidence rules.

11. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

12. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

Citation Information

Cited By

  • Automatic migration method and system for graph database metadata

    CN121350314A