Data processing method, data processing device, electronic equipment and storage medium
By obtaining data mapping rules to generate structured query language information, the data in the to-processed database is migrated to the target map database, which solves the problem of slow data migration speed in the prior art and realizes efficient and accurate data processing.
Patent Information
- Application Number
- CN202411009232.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
When the existing technology stores massive data into the graph database, the migration speed is extremely slow, affecting the data processing efficiency.
By obtaining the data mapping rules of the pending database and the target graph database, generating structured query language information, extracting the target data from the pending database, and adding it to the target graph database, simplifying the migration process and improving data processing efficiency and accuracy .
It greatly simplifies the data migration process, saves time cost and memory resource consumption, and improves data processing efficiency and accuracy.
Smart Images

Figure CN118964327A_ABST
Abstract
Description
Background Art
[0002] With the rapid increase in the amount of data in all walks of life, how to tap the potential value of data and effectively manage and utilize data has received widespread attention.
[0003] In order to better reflect the relationship between data, people often migrate data from databases to graph databases. However, when storing massive data in graph databases, existing technologies require a lot of time to parse and execute complex program statements. Especially when creating a large number of nodes and relationships, the migration speed is extremely slow, affecting data processing efficiency.
[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0005] The present disclosure provides a data processing method, a data processing device, an electronic device and a computer-readable storage medium, thereby overcoming the problem of low data processing efficiency when migrating data to a graph database in the prior art at least to a certain extent.
[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0007] According to one aspect of the present disclosure, there is provided a data processing method, which is applied to a data processing system, and includes: obtaining a database to be processed; the database to be processed includes a data table; obtaining a data mapping rule between the database to be processed and a target graph database, the data mapping rule being used to associate the data table in the database to be processed with the nodes in the target graph database; generating structured query language information according to the data mapping rule, and extracting target data from the database to be processed through the structured query language information; and adding the target data to the target graph database.
[0008] According to one aspect of the present disclosure, there is provided a data processing device, which is applied to a data processing system, and includes: a database acquisition module, which is used to acquire a database to be processed; the database to be processed includes a data table; a mapping rule acquisition module, which is used to acquire data mapping rules between the database to be processed and a target graph database, and the data mapping rules are used to associate the data tables in the database to be processed with the nodes in the target graph database; a target data extraction module, which is used to generate structured query language information according to the data mapping rules, and extract target data from the database to be processed through the structured query language information; and a target data adding module, which is used to add the target data to the target graph database.
[0009] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.
[0010] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above methods is implemented.
[0011] The exemplary embodiments of the present disclosure have the following beneficial effects:
[0012] Obtain a database to be processed; the database to be processed includes a data table; obtain data mapping rules between the database to be processed and the target graph database, the data mapping rules are used to associate the data table in the database to be processed with the nodes in the target graph database; generate structured query language information according to the data mapping rules, and extract the target data from the database to be processed through the structured query language information; add the target data to the target graph database. On the one hand, this exemplary embodiment provides a new data processing method, which can generate structured query language information for extracting data through the acquired data mapping rules, and can migrate the data in the database to be processed to the target graph database according to the mapping relationship based on the structured query statement information, which greatly simplifies the migration process. Compared with the prior art, a large amount of data is stored in the graph database and then time is spent on parsing and determining the relationship, which saves time cost and improves data processing efficiency and accuracy; on the other hand, in the process of migrating the data in the database to be processed to the target graph database, this exemplary embodiment can extract the target data from the database to be processed according to the structured query language information of the data mapping rules and add it to the target graph database. The process is simple and convenient, which improves the data migration speed and effectively saves unnecessary memory resource consumption during the data migration process.
[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0015] Figure 1A flow chart schematically illustrates a data processing method in this exemplary embodiment;
[0016] Figure 2 A schematic diagram schematically shows a structure of a data mapping relationship in this exemplary embodiment;
[0017] Figure 3 A schematic diagram schematically shows another structural diagram of data mapping relationship in this exemplary embodiment;
[0018] Figure 4 A sub-flow chart schematically illustrates a data processing method in this exemplary embodiment;
[0019] Figure 5 A module architecture diagram schematically illustrates a data processing method in this exemplary embodiment;
[0020] Figure 6 A block diagram schematically shows a structure of a data device in this exemplary embodiment;
[0021] Figure 7 An electronic device for implementing the above method in this exemplary embodiment is schematically shown. DETAILED DESCRIPTION
[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0023] An exemplary embodiment of the present disclosure first provides a data processing method, which is applied to a data processing system. The data processing system may include multiple nodes or devices for performing data processing, such as a terminal device where a database to be processed is located, an intermediate device, a terminal device where a target graph database is located, and other multiple nodes.
[0024] The following is combined with Figure 1 This exemplary embodiment is further described as follows. Figure 1 As shown, the data processing method may include the following steps S110 to S140:
[0025] Step S110, obtaining a database to be processed; the database to be processed includes a data table.
[0026] Among them, the database to be processed can be any database that needs to add data to the graph database. The database to be processed may include one or more data tables, and may also include the relationship between data tables. The data table may be a general data table including common data, for example, it may include a student table and a teacher table. The data table may also be a relationship table including general data tables, for example, a relationship table between a student table and a teacher table. The relationship table may reflect the association between the student table and the teacher table. Based on the relationship table, the association between the student information in the student table and the teacher information in the teacher table may be queried. In this exemplary embodiment, the data tables included in the database to be processed and the relationship between the data tables may be added to the target graph database to generate a corresponding graph relationship network.
[0027] Step S120, obtaining data mapping rules between the database to be processed and the target graph database, where the data mapping rules are used to associate the data tables in the database to be processed with the nodes in the target graph database.
[0028] The target graph database refers to the graph database to which the data in the database to be processed needs to be added, which can be any specified graph database, such as a Neo4j graph database. A graph database can include multiple nodes, each of which can be an entity, and the lines between nodes can reflect the relationship between entities. The data mapping rules can reflect the mapping relationship between the data tables in the database to be processed and the nodes in the target graph database. For example, the data tables in the database to be processed can have a mapping relationship with the nodes in the graph database, and the relationship between the data tables in the database to be processed and the relationship between the nodes in the graph database can also have a mapping relationship.
[0029] This exemplary embodiment may configure data mapping rules between the database to be processed and the target database in advance, and the data mapping rules are used to associate the data tables in the database to be processed with the nodes in the target graph database.
[0030] Specifically, in an exemplary embodiment, the data mapping rule includes at least one of the following:
[0031] Data mapping rules between the name of the data table in the database to be processed and the label of the node in the target graph database;
[0032] Data mapping rules between the fields of the data table in the database to be processed and the attributes of the nodes in the target graph database.
[0033] In the target graph database, each node may have a corresponding label to identify the node. The data mapping rules may include mapping rules between the name of the data table in the database to be processed and the label of the node in the target graph database. For example, if the name of the data table is the student table, the label of the corresponding node may be the student node or the node corresponding to the student table, etc. The data mapping rules may also include mapping rules between the fields of the data table in the database to be processed and the attributes of the node in the target graph database, wherein a field refers to a character segment in the data table, which may be an indicator field or an indicator value field. For example, a field may be "name", or the value "xiaoming" corresponding to "name", etc. The attributes of the node refer to information used to reflect the content and characteristics of the node. In this exemplary embodiment, the fields of the data table and the attributes of the node may have a mapping relationship. For example, the "name" field in the student table may be mapped to the attributes of the student node, etc.
[0034] In this exemplary embodiment, the database to be processed may include multiple data tables, and one data table may correspond to multiple nodes. When configuring the mapping rules of the nodes, the user can freely choose to map all or part of the fields in the data table to the attributes of the nodes in the graph database. The fields in the data table and the attributes of the nodes can be a one-to-one relationship. The same node may include multiple attributes. For example, different students in the student table may correspond to different nodes, and the specific field information of a student may correspond to the attribute information under a node, etc. Each node may include multiple attributes. For example, in the student table, there may be multiple fields such as "name" and "age". Accordingly, the mapping node may also have multiple attributes.
[0035] Figure 2 shows a data mapping rule for nodes in a graph database. Figure 2 (a) shows a schematic diagram of the structure of a data table in a database to be processed, including the name and fields of the data table; Figure 2 (b) shows a schematic diagram of a node structure of a Neo4j graph database, including nodes labeled as label 1 corresponding to different attributes; Figure 2 (c) shows the data mapping rules of the node, including mapping the table name to the label, and mapping the field to the attribute.
[0036] In an exemplary embodiment, the fields of a data table include primary keys and / or foreign keys, and the foreign keys are used to represent the association relationship between the data table and other data tables in the database to be processed. The data mapping rules between the primary keys and / or foreign keys of the data tables in the database to be processed and the attributes of the nodes in the target graph database are used to establish association relationships for the nodes in the target graph database.
[0037] Among them, the primary key can be the identification information used to uniquely identify the data table, which can be a name, ID (Identify), etc. In this exemplary embodiment, any one of the multiple fields of the data table can be selected as the primary key, such as "name", "age", "other codes", etc.; the foreign key refers to the information used to reflect the relationship between the data table and the data table. According to the foreign key, the information of other data tables related to the current data table can be found. For example, according to the foreign key, the teacher information associated with the student information in the student table can be found in the teacher table. Both the primary key and the foreign key can be stored in the data table as a field. When configuring the data mapping rules, the primary keys and foreign keys of the two associated data tables can be mapped to the attributes of the nodes of the corresponding data tables in the target graph database respectively, and the relationship between the two nodes is established according to the values of the primary key and the foreign key. For example, the data tables in the database to be processed can be associated through the relationship table, and the two primary keys of the two data tables can be mapped to the attributes of the two nodes in the target graph database respectively. Then, the relationship between the two nodes is established according to the values of the primary key and the foreign key. The label name representing the relationship in the target graph database can be customized according to the actual scenario, and the present disclosure does not make specific restrictions on this.
[0038] In this exemplary embodiment, the data tables included in the database to be processed may include general data tables and relationship tables used to represent the relationship between general data tables. The relationship tables may reflect the relationship between different data tables. The data mapping rules may include node information and node relationship information. Figure 3 It shows a data mapping rule for node relationships in a graph database. Figure 3 (a) shows a schematic diagram of a relational table structure of a relational database, including two data tables named Table 1 and Table 2 and a corresponding relational table, wherein the relational table indicates that in relation 1, field 1 of Table 1 has a relation with field 3 of Table 2; Figure 3 (b) shows the corresponding relationship structure of the relationship labels between nodes in the Neo4j database; Figure 3 (b) shows the data mapping rules of the node relationship, including the relationship mapping information between Table 1 and Table 2.
[0039] Step S130 : generating structured query language information according to the data mapping rule, and extracting target data from the database to be processed through the structured query language information.
[0040] Among them, the structured query language information can be an SQL (Structured Query Language) statement. This exemplary embodiment can generate structured query language information through pre-configured data mapping rules, and then extract target data from the database to be processed through the structured query language information. The target data refers to the data and data relationships used to be added to the target graph database, such as specific information in a data table, and the relationship between different data tables.
[0041] Step S140, adding the target data to the target graph database.
[0042] Finally, the target data can be added to the target graph database to complete the data migration process of the data table with association relationship. After adding it to the target graph database, a graph relationship network corresponding to the target data can be generated, in which the nodes have a corresponding relationship with the data table. Through the graph relationship network, the data table information and the relationship between the data tables can be obtained.
[0043] In an exemplary embodiment, if Figure 4 As shown, after the above step S130, the above data processing method may further include:
[0044] Step S410, saving the target data as a character segmentation value format file according to the data mapping rule;
[0045] Step S140 may include:
[0046] Step S420, adding the character segmentation value format file to the target graph database.
[0047] Among them, the character-separated value format file can be a CSV (Comma-Separated Values) format file, and the CSV format file can store table data in plain text, such as numbers and text. The data extraction process can be performed based on the structured query language information, but the extraction process may not be able to associate and mine data relationships and deep data information. This exemplary embodiment can also save the extracted target data as a character-separated value format file based on data mapping rules. The character-separated value format file can be used to store the mapping relationship between nodes and data tables in the graph database, as well as specific data. The data of each field in the data table can correspond to a column of data in the character-separated value format file. In this exemplary embodiment, the character-separated value format file can add a column of data as needed to store the label information of the target graph database node.
[0048] In the related art, after the user configures the data to be extracted from the database, it is directly stored in the graph database, which is unable to store the relationship between the data well. The present exemplary embodiment saves the target data as a character-separated value format file according to the data mapping rules, which can effectively store the data and the data relationship, and adds the character-separated value format file to the target graph database, which makes it easier for the target graph database to manage the information of the data table in the database to be processed.
[0049] In this exemplary embodiment, in order to further improve the effectiveness and pertinence of the relationship, different types of target data can be saved as different character-separated value format files according to the type of target data, for example, specific field data in the data table is saved in the first CSV format file; the primary key or foreign key representing the data table relationship is saved in the second SCV format file. That is, the data in the data table and the relationship between the data tables can be stored separately, and the association relationship data between two data tables in the database to be processed can be stored in a character-separated value format file, which can mainly include the primary key or foreign key of the two associated data tables, the label name of the target graph database relationship, etc.
[0050] In an exemplary embodiment, the above step S420 may include:
[0051] The character segmentation value format file is transmitted to the terminal device where the target graph database is located, so that when the terminal device where the target graph database is located receives the data execution instruction, the character segmentation value format file is added to the target graph database to generate a graph relationship network.
[0052] The database to be processed and the target graph database can be on two different terminal devices. Then, adding the target data to the target graph database can be by first sending the target data to the terminal device where the target graph database is located, for example, remotely copying the character-separated value format file to the host node where the Neo4j graph database is located, and then uploading the character-separated value format file to the specified directory of the host node where the Neo4j graph database is located through the remote copy command. This allows the terminal device to add the target data to the target graph database, for example, by sending a remote execution command to the host node where the Neo4j graph database is located, the target data of the character-separated value format file can be imported into the Neo4j graph database. The import of the character-separated value format file can be implemented through relevant tools, for example, the CSV format file can be imported into the Neo4j graph database using the neo4j-admin (an import tool) tool.
[0053] Based on the above description, in this exemplary embodiment, a database to be processed is obtained; the database to be processed includes a data table; a data mapping rule between the database to be processed and the target graph database is obtained, and the data mapping rule is used to associate the data table in the database to be processed with the node in the target graph database; according to the data mapping rule, structured query language information is generated, and the target data is extracted from the database to be processed through the structured query language information; and the target data is added to the target graph database. On the one hand, this exemplary embodiment provides a new data processing method, which can generate structured query language information for extracting data through the acquired data mapping rule, and can migrate the data in the database to be processed to the target graph database according to the mapping relationship based on the structured query statement information, which greatly simplifies the migration process. Compared with the prior art, a large amount of data is stored in the graph database and then time is spent on parsing and determining the relationship, which saves time cost and improves data processing efficiency and accuracy; on the other hand, in the process of migrating the data in the database to be processed to the target graph database, the target data can be extracted from the database to be processed according to the structured query language information of the data mapping rule and added to the target graph database. The process is simple and convenient, which improves the data migration speed and effectively saves unnecessary memory resource consumption during the data migration process.
[0054] In an exemplary embodiment, the data processing system includes a processing database node to be processed, an intermediate node, and a target graph database node. After obtaining the database to be processed, the data processing method may further include:
[0055] Configure the first connection information of the database to be processed and the second connection information of the target graph database;
[0056] The first connection information is used to connect the intermediate node with the node of the database to be processed, so as to extract the target data from the database to be processed through the structured query language information; the second connection information is used to connect the intermediate node with the node of the target graph database, so as to transmit the target data to the node where the target graph database is located, and enable the node where the target graph database is located to add the target data to the target graph database.
[0057] After obtaining the database to be processed, this exemplary embodiment can first configure the connection information of the database to be processed and the target database. The first connection information is used for the intermediate node to connect with the node of the database to be processed, so as to extract the target data from the database to be processed through structured query language information; the second connection information is used for the intermediate node to connect with the node of the target graph database, so as to transmit the target data to the node where the target graph database is located, and enable the node where the target graph database is located to add the target data to the target graph database.
[0058] Among them, the node of the database to be processed can be the terminal device where the database to be processed is located, the target graph database node can be the terminal device where the target graph database is located, the intermediate node can be another terminal device different from the node of the database to be processed and the target graph database node, or it can be an independent device or component integrated in the terminal device where the database to be processed is located.
[0059] Figure 5 The module architecture diagram of a data processing method in this exemplary embodiment is shown, which may specifically include a database node 510 to be processed, including a database 511 to be processed, an intermediate node 520 for data migration, and a target graph database node 530, including a target graph database 531. This exemplary embodiment may establish a connection with the intermediate node 520 through the first connection information of the database to be processed, so that the intermediate node 520 can extract target data from the database 511 to be processed of the database node 510 based on an SQL statement, and store it as a CSV format file, and establish a connection with the intermediate node through the second connection information of the target graph database 530, so that the intermediate node 520 can copy the generated CSV format file to the target graph database node 530, and make the target graph database node 530 execute relevant instructions, add the CSV format file to the target graph database 531, and generate a graph relationship network.
[0060] Based on this, in the present exemplary embodiment, during the migration process from the database to be processed to the target graph database, it is only necessary to establish a connection with the database to be processed through an intermediate node, and there is no need to directly establish a connection between the database to be processed and the target graph database. This can significantly reduce the memory resource consumption during the migration process and greatly improve the migration speed.
[0061] In an exemplary embodiment, the first connection information includes the type of database to be processed, such as Mysql, Oracle, PostgreSql, Microsoft SQL Server, Microsoft Access, etc., one or more of the user name, password, connection address of the database to be processed, connection port number of the database to be processed, and instance name of the database to be processed; the second connection information includes the connection information of the terminal where the target graph database is located, which may specifically include the connection address, host port number, user name and password, etc. of the host node where the target graph database is located.
[0062] The following takes the target graph data as Neo4j graph database as an example to illustrate the above data processing method. The database to be processed is obtained, and the pre-configured data mapping rules are obtained. In the database to be processed, there is a dictionary type data table "dict_spec". Now the three fields "dict_spec_id", "dict_spec_nbr", and "dict_spec_name" in this data table need to be migrated to the node with the "dictionary type" label of the Neo4j graph database, which correspond to the "dictionary type id", "dictionary type encoding", and "dictionary type name" attributes of the Neo4j graph database node respectively. The data mapping rules expressed in json format are as follows:
[0063]
[0064] In the data mapping rules of Neo4j graph database nodes, "conceptName" indicates the label of Neo4j graph database nodes; "conceptInfo" contains the mapping relationship between data tables and nodes; "tableName" indicates the name of the data table, "filed" indicates the field of the data table, and "name" indicates the attribute of the Neo4j graph database node. Through such data mapping rules, the data table of the database to be processed can be mapped to the Neo4j graph database node.
[0065] Similarly, in the database to be processed, there is a data table "dict_value", which is mapped to the node with the "dictionary value" label in the Neo4j graph database. The data mapping rules expressed in json format are as follows:
[0066]
[0067] The data tables "dict_spec" and "dict_value" are associated through the relationship table "dict_spec_dict_value_rel". The primary keys of "dict_spec" and "dict_value" are "dict_spec_id" and "dict_value_id" respectively. The association between these two data tables is mapped to the relationship in the Neo4j graph database. The data mapping rules expressed in json format are as follows:
[0068]
[0069]
[0070] In the data mapping rules of Neo4j graph database relationships, because relationships are directional, the head node and tail node are used to represent the outgoing node and incoming node of the relationship respectively. "name" represents the label of the relationship, "fromTable" represents the name of the head node data table, "toTable" represents the name of the tail node data table, "relationTable" represents the name of the relationship table, "fromKey" represents the primary key of the head node data table, "toKey" represents the primary key of the tail node data table, "leftLinkey" represents the key associated with the head node in the relationship table, and "rightLinkey" represents the key associated with the tail node in the relationship table.
[0071] Then, based on the above code implementation, structured query language information, such as SQL statements, is generated according to the data mapping rules. Specifically, a parsing algorithm for generating SQL statements can be used to traverse the nodes.conceptInfo array in the data mapping rules, obtain the data table name according to the tableName field, and obtain the field name corresponding to the data table according to the filed field. According to the above implementation, the automatically generated node information extraction SQL statement is as follows:
[0072] Node extraction SQL statement:
[0073] select dict_spec_id,dict_spec_nbr,dict_spec_name frome dict_spec;
[0074] select dict_value_id,dict_value_nbr,dict_value_name frome dict_value;
[0075] The "links" field in the data migration rule is parsed to generate a relationship query statement for finding the association relationship between the "dict_spec" and "dict_value" table data. According to the above embodiment, the automatically generated data relationship extraction SQL statement is as follows:
[0076] Relation extraction SQL statement:
[0077] select dict_spec.dict_spec_id,dict_value.dict_value_id from dict_spec
[0078] inner join dict_spec_dict_value_rel on dict_spec.dict_spec_id=dict_spec_dict_value_rel.dict_spec_id
[0079] inner join dict_spec_dict_value_rel.dict_value_id=dict_value.dict_value_id
[0080] Furthermore, according to the configured data mapping rules and automatically generated SQL statements, the target data extracted from each SQL statement is stored in a CSV file. According to the above embodiment, the data extracted by the three SQL statements can be stored in three CSV files. The generation rules of CSV files are divided into two categories: node files and relationship files, among which the node files are generated by extracting SQL statements based on node information, and the relationship files are generated by extracting SQL statements based on relationships. Generation rules for node files: The database of each field of the relationship data table corresponds to a column of data in the CSV file, and the CSV file can additionally add a column of data to store the labels of the Neo4j database nodes. Generation rules for relationship files: The CSV file mainly has three columns of data, namely: the primary key or foreign key of the two data tables, and the label name of the Neo4j graph database relationship. According to the above embodiment, the column names of the three CSV files stored in the three SQL statements correspond to:
[0081] Node csv file format:
[0082] dict_spec_id:ID,dic_spec_nbr,dict_spec_name,:Label
[0083] dict_value_id:ID,dic_value_nbr,dict_value_name,:Label
[0084] Relationship csv file format:
[0085] Relationship csv file format:
[0086] :START_ID,:END_ID,:TYPE
[0087] Finally, remotely copy the CSV file to the host node where the Neo4j graph database is located, and upload the connection information of the host node where Neo4j is located and the generated multiple CSV files to the specified directory of the host node where the Neo4j graph database is located through the remote copy command, such as using the scp command for remote copying; send a remote execution command to the host node where the Neo4j graph database is located, and use the neo4j-admin tool to import the CSV file into the Neo4j graph database, such as using the neofj-admin import command to import, and finally generate a graph relationship network about the imported data.
[0088] The exemplary embodiments of the present disclosure also provide a data processing device. Figure 6 The device 600 may include a database acquisition module 610 for acquiring a database to be processed; the database to be processed includes a data table; a mapping rule acquisition module 620 for acquiring data mapping rules between the database to be processed and the target graph database, the data mapping rules being used to associate the data table in the database to be processed with the nodes in the target graph database; a target data extraction module 630 for generating structured query language information according to the data mapping rules, and extracting target data from the database to be processed through the structured query language information; and a target data adding module 640 for adding target data to the target graph database.
[0089] In an exemplary embodiment, the data mapping rules are configured to include at least one of the following: data mapping rules between the name of the data table in the database to be processed and the label of the node in the target graph database; data mapping rules between the field of the data table in the database to be processed and the attribute of the node in the target graph database.
[0090] In an exemplary embodiment, the fields of a data table include primary keys and / or foreign keys, and the foreign keys are used to represent the association relationship between the data table and other data tables in the database to be processed. The data mapping rules between the primary keys and / or foreign keys of the data tables in the database to be processed and the attributes of the nodes in the target graph database are used to establish association relationships for the nodes in the target graph database.
[0091] In an exemplary embodiment, after extracting the target data from the database to be processed through structured query language information, the device may also include: a file saving unit, used to save the target data as a character segmentation value format file according to the data mapping rules; a target data adding module, including: a file adding unit, used to add the character segmentation value format file to the target graph database.
[0092] In an exemplary embodiment, a file adding unit is used to transmit a character segmentation value format file to a terminal device where a target graph database is located, so that when the terminal device where the target graph database is located receives a data execution instruction, the character segmentation value format file is added to the target graph database to generate a graph relationship network.
[0093] In an exemplary embodiment, the data processing system includes a processing database node to be processed, an intermediate node and a target graph database node. After obtaining the database to be processed, the device also includes: a connection information configuration unit, used to configure the first connection information of the database to be processed and the second connection information of the target graph database; the first connection information is used for the intermediate node to connect with the database node to be processed, so as to extract the target data from the database to be processed through structured query language information; the second connection information is used for the intermediate node to connect with the target graph database node, so as to transfer the target data to the node where the target graph database is located, and enable the node where the target graph database is located to add the target data to the target graph database.
[0094] In an exemplary embodiment, the first connection information includes one or more of the type of the database to be processed, user name, password, connection address of the database to be processed, connection port number of the database to be processed, and instance name of the database to be processed; the second connection information includes connection information of the target graph database node.
[0095] The specific details of each module / unit in the above-mentioned device have been described in detail in the embodiments of the method part. The undisclosed details can be found in the embodiments of the method part, so they will not be repeated here.
[0096] Exemplary embodiments of the present disclosure also provide an electronic device capable of implementing the above method.
[0097] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0098] Refer to the following Figure 7 An electronic device 700 according to such an exemplary embodiment of the present disclosure is described below. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0099] like Figure 7As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.
[0100] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can execute Figure 1 Steps shown etc.
[0101] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .
[0102] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0103] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0104] The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0105] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiment of the present disclosure.
[0106] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.
[0107] The exemplary embodiments of the present disclosure also provide a program product for implementing the above method, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0108] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0109] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0110] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0111] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0112] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0113] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0114] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0115] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A data processing method, characterized in that: Applied to data processing systems, including: Obtaining a database to be processed; the database to be processed includes a data table; Obtaining data mapping rules between the database to be processed and the target graph database, wherein the data mapping rules are used to associate a data table in the database to be processed with a node in the target graph database; Generate structured query language information according to the data mapping rule, and extract target data from the database to be processed through the structured query language information; The target data is added to the target map database.
2. The method according to claim 1, characterized in that: The data mapping rule includes at least one of the following: Data mapping rules between the name of the data table in the to-be-processed database and the label of the node in the target graph database; Data mapping rules between the fields of the data table in the to-be-processed database and the attributes of the nodes in the target graph database.
3. The method according to claim 2, characterized in that The fields of the data table include primary keys and / or foreign keys, and the foreign keys are used to represent the association relationship between the data table and other data tables in the database to be processed. The data mapping rules between the primary keys and / or foreign keys of the data tables in the database to be processed and the attributes of the nodes in the target graph database are used to establish association relationships for the nodes in the target graph database.
4. The method according to claim 1, characterized in that: After extracting the target data from the database to be processed by using the structured query language information, the method further includes: According to the data mapping rule, the target data is saved as a character segmentation value format file; The adding the target data to the target graph database comprises: The character segmentation value format file is added to the target graph database.
5. The method according to claim 1, characterized in that The step of adding the character segmentation value format file to the target graph database comprises: The character segmentation value format file is transmitted to the terminal device where the target graph database is located, so that when the terminal device where the target graph database is located receives the data execution instruction, the character segmentation value format file is added to the target graph database to generate a graph relationship network.
6. The method according to claim 1, characterized in that The data processing system includes a processing database node to be processed, an intermediate node and a target graph database node. After obtaining the database to be processed, the method further includes: Configure the first connection information of the database to be processed and the second connection information of the target graph database; The first connection information is used for the intermediate node to connect with the node of the database to be processed so as to extract target data from the database to be processed through the structured query language information; the second connection information is used for the intermediate node to connect with the target graph database node so as to transmit the target data to the node where the target graph database is located, and enable the node where the target graph database is located to add the target data to the target graph database.
7. The method according to claim 6, characterized in that The first connection information includes one or more of the type of the database to be processed, user name, password, connection address of the database to be processed, connection port number of the database to be processed, and instance name of the database to be processed; the second connection information includes connection information of the target graph database node.
8. A data processing device, characterized in that: Applied to data processing systems, including: A database acquisition module is used to acquire a database to be processed; the database to be processed includes a data table; A mapping rule acquisition module, used to acquire data mapping rules between the database to be processed and the target graph database, wherein the data mapping rules are used to associate the data table in the database to be processed with the nodes in the target graph database; A target data extraction module, used to generate structured query language information according to the data mapping rule, and extract target data from the database to be processed through the structured query language information; A target data adding module is used to add the target data to the target graph database.
9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Data migration method and device, computer program product and electronic equipment
CN120407539A
