Data processing method and system based on knowledge graph, and computer program product

Abstract

The invention relates to the technical field of computers, in particular to a data processing method and system based on a knowledge graph, a computer program product, electronic equipment and a storage medium. The method comprises the steps of obtaining to-be-processed data, and performing partition storage on the to-be-processed data according to time attributes; loading the to-be-processed data corresponding to each storage partition in a partition manner, creating an external table, and extracting corresponding relation detail data; performing relation aggregation processing on the relation detail data, and outputting relation result data based on a pre-configured result data standard data structure; and incrementally updating the relationship result data into a graph database, and updating the entity attributes and the relationship attributes based on the entity relationship. The scheme can be applied to relational data processing of a million-level data scale.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and specifically to a data processing method and system based on a knowledge graph, a computer program product, an electronic device, and a storage medium. Background Art

[0002] In related technologies, the use of knowledge graphs to record relational data has been widely used in many fields. In large-scale data scenarios, performance bottlenecks are likely to occur when simply relying on the capabilities of graph databases to perform graph analysis of relational data such as points, edges, and minimum paths. Open source scenarios only support a scale of billions. In the design of graph analysis architecture for known relational data, when it involves tens of billions or even larger scales, methods such as expanding computing nodes, introducing GPU acceleration, or introducing graph processing engines are usually adopted. All of these methods increase performance by adding hardware equipment and using higher costs.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a data processing method and system based on a knowledge graph, a storage medium, a computer program product, and an electronic device, which can be applied to the construction of a knowledge graph of hundreds of billions of data, effectively overcoming the defects existing in the prior art.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to a first aspect of the present disclosure, a data processing method based on a knowledge graph is provided, the method comprising:

[0007] Acquire the data to be processed, and partition and store the data to be processed according to time attributes;

[0008] Partition loading the data to be processed corresponding to each storage partition, creating an external table, and extracting corresponding relationship detail data;

[0009] Performing relationship aggregation processing on the relationship detail data, and outputting relationship result data based on a pre-configured result data standard data structure;

[0010] The relationship result data is incrementally updated to the graph database, and entity attributes and relationship attributes are updated based on entity relationships.

[0011] In some exemplary embodiments, the method further comprises:

[0012] Performing data cleaning on the relationship detail data and defining row-key generation rules;

[0013] Based on the row-key generation rule, the relationship detail data after data cleaning is batch processed and saved to the Hbase database.

[0014] In some exemplary embodiments, the method further comprises:

[0015] Obtaining a relational data query request based on a graph database; wherein the relational data query request includes entity query information and / or relationship query information;

[0016] Perform row-key search in the Hbase database based on the master record ID of the relationship to obtain data search results;

[0017] A graph relationship is constructed based on the data retrieval results and displayed in a graphical user interface.

[0018] In some exemplary embodiments, the relationship data query request further includes: time query information;

[0019] The method further comprises:

[0020] According to the time query information, data is searched in the Hbase database using the time attribute to obtain a data search result.

[0021] In some exemplary embodiments, the obtaining of the data to be processed and partitioning and storing the data to be processed according to time attributes includes:

[0022] The data to be processed is compressed, connected to the HDFS file system in Parquet file format, and partitioned and stored by day.

[0023] In some exemplary embodiments, when extracting the corresponding relationship detail data, the method further includes:

[0024] The relationship detail data is classified according to a pre-configured standardized relationship data structure, so as to perform data aggregation processing according to the classification result; wherein the standardized relationship data structure includes multiple relationship types.

[0025] In some exemplary embodiments, the method further comprises:

[0026] In the graph database, entity types are defined based on a preconfigured standard data structure of result data, and graph definition is performed according to relationship types.

[0027] According to a second aspect of the present disclosure, a knowledge graph-based data processing system is provided, comprising:

[0028] A partition storage module is used to obtain the data to be processed and partition and store the data to be processed according to time attributes;

[0029] A relationship detail data extraction module is used to load the to-be-processed data corresponding to each storage partition in partitions, create an external table, and extract the corresponding relationship detail data;

[0030] An aggregation processing module, used for performing relationship aggregation processing on the relationship detail data, and outputting relationship result data based on a pre-configured result data standard data structure;

[0031] The database update module is used to incrementally update the relationship result data into the graph database, and update the entity attributes and relationship attributes based on the entity relationship.

[0032] According to a third aspect of the present disclosure, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned data processing method based on the knowledge graph is implemented.

[0033] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0034] Processor; and

[0035] A memory, configured to store executable instructions of the processor;

[0036] Among them, the processor is configured to implement the above-mentioned knowledge graph-based data processing method by executing the executable instructions.

[0037] According to a fifth aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned data processing method based on the knowledge graph is implemented.

[0038] The data processing method based on the knowledge graph provided by the embodiment of the present disclosure partitions the newly added data to be processed, and loads the incremental data to be processed in partitions and extracts the corresponding relationship detail data, then aggregates the relationship detail data to output the corresponding relationship result data, and then updates it to the graph database. By partitioning the data to be processed and updating the relationship result data using the computing power of the graph database, the advantages of each component are maximized to complement each other, making up for the performance problem of graph computing with hundreds of billions of data, while also reducing its storage cost.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0041] Figure 1 A schematic diagram schematically illustrates an exemplary system architecture of the technical solution of the exemplary embodiments of the present disclosure;

[0042] Figure 2 A schematic diagram schematically illustrates a data processing method based on a knowledge graph according to an exemplary embodiment of the present disclosure;

[0043] Figure 3 A schematic diagram schematically illustrates a data query method according to an exemplary embodiment of the present disclosure;

[0044] Figure 4 A schematic diagram schematically illustrates a data processing method flow in an exemplary embodiment of the present disclosure;

[0045] Figure 5 A schematic diagram schematically illustrates a data processing system based on a knowledge graph in an exemplary embodiment of the present disclosure;

[0046] Figure 6 The diagram schematically shows the components of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] In related technologies, the characterization of knowledge graphs for relational data has been widely used, which has led to the introduction of databases specifically designed for relational data graph computing, such as Neo4j, JanusGraph, Amazon Neptune, etc. Such databases provide query and storage capabilities specifically for graph data. In large-scale data scenarios, performance bottlenecks are likely to occur when simply relying on the capabilities of graph databases to perform dimensional graph analysis of relational data such as points, edges, and minimum paths. Open source scenarios only support a scale of billions. When it comes to tens of billions or even larger scales, solutions are usually adopted to expand computing nodes, introduce GPU acceleration, and introduce graph processing engines, etc., in exchange for performance at a higher cost.

[0050] In view of the shortcomings and deficiencies of the prior art, this example implementation provides a data processing method based on a knowledge graph, which can solve the problem of relational data processing at a scale of hundreds of billions.

[0051] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0052] like Figure 1 As shown, the system architecture 100 may include an intelligent terminal device 101, a server 103, and a data storage system 104. The intelligent terminal device 101 may be a computer, a mobile phone, a tablet computer, etc. The intelligent terminal device 101 may communicate and transmit data with the server 103 through a communication network 102. The server 103 is connected to the data storage system 104. The communication network 102 may include various connection types, such as a wired communication link, a wireless communication link, etc. Among them, the server 103 may be a server cluster composed of multiple servers. The data storage system 104 may be a database.

[0053] In one embodiment of the present invention, a user can initiate a relational data processing request corresponding to a target service on a smart terminal device. After receiving the data processing request from the user terminal device, the server obtains and processes the incremental data corresponding to the target service, obtains the corresponding relational data, and updates it in the data storage system.

[0054] It should be noted that the method provided in the embodiment of the present invention can be executed by the server 103. However, in other embodiments of the present invention, the intelligent terminal device 101 and the server 103 can also cooperate to implement the data processing method based on the knowledge graph provided in the embodiment of the present invention.

[0055] Exemplary, reference Figure 2 As shown, the data processing method based on the knowledge graph may specifically include:

[0056] Step S11, obtaining the data to be processed, and partitioning and storing the data to be processed according to time attributes;

[0057] Step S12, partition-loading the data to be processed corresponding to each storage partition, creating an external table, and extracting corresponding relationship detail data;

[0058] Step S13, performing relationship aggregation processing on the relationship detail data, and outputting relationship result data based on a pre-configured result data standard data structure;

[0059] Step S14, incrementally updating the relationship result data into the graph database, and updating the entity attributes and relationship attributes based on the entity relationship.

[0060] The data processing method based on knowledge graph provided by the present invention can be applied to relational data processing at the scale of hundreds of billions. By introducing Hadoop, Spark, HBase and Janus Graph, the analysis architecture of graph computing of relational data is rationally planned to meet better query and computing performance. At the same time, it does not require large-scale GPU servers and uses SATA hard disk servers to reduce storage costs.

[0061] Below, each step of the knowledge graph-based data processing method in this example implementation will be described in more detail with reference to the accompanying drawings and examples.

[0062] In step S11, the data to be processed is obtained, and the data to be processed is partitioned and stored according to time attributes.

[0063] In this example implementation, the data to be processed may be business data corresponding to the target business; for example, the business data may be commodity sales data in the field of e-commerce, or financial data, or medical data.

[0064] Exemplarily, the obtaining of the data to be processed and partitioning and storing the data to be processed according to a time attribute include: compressing the data to be processed, accessing the data to an HDFS file system in a Parquet file format, and partitioning and storing the data by day.

[0065] Specifically, the business system corresponding to the target business can obtain the corresponding data to be processed and connect it to the HDFS file system (Hadoop Distributed File System). The Snappy compression tool is used to compress the data to be processed, and the compressed data is partitioned and stored in the HDFS file system by day in the Parquet file format, thereby realizing planned distributed storage of massive data files.

[0066] In step S12, the to-be-processed data corresponding to each storage partition is loaded in partition, an external table is created, and corresponding relationship detail data is extracted.

[0067] In this example implementation, the Spark tool may be called to load partitions of data files in the HDFS file system, create an external table, and extract relational detail data from the loaded data.

[0068] Exemplarily, when extracting the corresponding relationship detail data, the method further includes: classifying the relationship detail data according to a preconfigured standardized relationship data structure for data aggregation processing according to the classification result; wherein the standardized relationship data structure includes multiple relationship types.

[0069] Specifically, a standardized relational data structure can be pre-configured. As shown in Table 1, the standardized relational structure may include multiple key features: row-key, element ID, master record ID, multiple element types and corresponding element values, multiple relationship types, collection time, data source, and collection location; wherein the collection time may be the time when the data was captured; and the collection location may be the city or prefecture corresponding to the data collection. For the currently loaded data to be processed, text recognition may be performed on it, the corresponding feature fields may be extracted, and the standardized relational data structure may be used as a template for filling to obtain standardized relational data. For the extraction of the extracted relational detail data, referring to the data structure described in Table 1, the relational data may be classified according to the first, second, and third level relational types.

[0070] Table 1

[0071]

[0072]

[0073] In step S13, the relationship detail data is subjected to relationship aggregation processing, and the relationship result data is output based on a pre-configured standard data structure of the result data.

[0074] In this example implementation, after acquiring the relationship detail data, it can be aggregated. Referring to the standard data structure of the result data shown in Table 2, the collection days, collection times, first collection time, and last collection time of the same type of relationship data are calculated and used to calculate the intimacy of the displayed relationship.

[0075] Table 2

[0076]

[0077] In step S14, the relationship result data is incrementally updated to the graph database, and entity attributes and relationship attributes are updated based on the entity relationship.

[0078] In this example implementation, the graph database may be a Janus Graph database. The relationship result data generated in step S13 is incrementally updated to the Janus Graph database, and edge attributes are updated based on the primary record ID of the point-to-point relationship. A point may be an entity in a knowledge graph, and an edge may be a relationship in a relationship graph.

[0079] Exemplarily, the method further includes: defining entity types in the graph database based on a preconfigured standard data structure of result data, and performing graph definition according to relationship types.

[0080] Specifically, the schema configuration of the Janus Graph graph database can be customized in advance according to business needs. Specifically, in the schema configuration of the Janus Graph graph database, the point type can be defined based on the feature type contained in the result data standard data structure shown in Table 2, and the graph can be defined according to the relationship type.

[0081] In this example implementation, the method further includes:

[0082] Step S15, performing data cleaning on the relationship detail data and defining row-key generation rules;

[0083] Based on the row-key generation rule, the relationship detail data after data cleaning is batch processed and saved to the Hbase database.

[0084] Specifically, the extracted relationship detail data and / or the standardized relationship data processed based on the standardized relationship data structure can be cleaned. Specifically, the generation rule of its Rowkey can be predefined: (recordid, dataid, latm) separated by '|', where recordid is the unique ID of the same relationship between AB elements, dataid is the different detailed record IDs under this unique ID, and latm is used for time range query of Hbase database.

[0085] You can add all the data to the Hbase database, shard Hbase by relationship type, and define the table sharding strategy based on the scale of relationship detail data. The default is 100.

[0086] In this example implementation, reference Figure 3 As shown, the above method also includes:

[0087] Step S21, obtaining a relational data query request based on a graph database; wherein the relational data query request includes entity query information and / or relationship query information;

[0088] Step S22, performing a row-key search in the Hbase database based on the master record ID of the relationship to obtain a data search result;

[0089] Step S23, constructing a graph relationship based on the data retrieval results and displaying it in a graphical user interface.

[0090] Specifically, the user can initiate a relational data query request for the knowledge graph on the terminal device and send the data query request to the server. The data query request can be a point relationship query for the Janus Graph database. The server can process the data query request based on the Janus Graph database and build a graph relationship display of the elements. When performing data queries, the minimum path and other dimensions can be calculated based on the relationship results of the Janus Graph database; when querying edge attribute details, the Rowkey retrieval query of the Hbase database can be performed based on the primary record ID of the edge.

[0091] In this example implementation, the relationship data query request further includes: time query information. The method further includes:

[0092] According to the time query information, data is searched in the Hbase database using the time attribute to obtain a data search result.

[0093] Specifically, the user's query request may also include a time range; for example, data may be queried in the Hbase database based on the data collection time, and the data retrieval results may be output. In addition, the data retrieval results may be sorted, and the sorted retrieval results may be output and fed back to the terminal device.

[0094] In this example implementation, reference Figure 4 As shown in the figure, for relational data of hundreds of billions of scale, HDFS files are partitioned and stored by day, and Spark batch processing is performed incrementally to extract relational data according to the standardized relational structure. Then, the incremental relational data is aggregated, the vertex and edge attribute results are calculated, and the relational result data is stored in the Janus Graph database to meet the graph expansion calculation of vertex and edge queries. At the same time, the incremental relational detail data can be batch processed and written into the Hbase database for the detailed data service of the graph relationship.

[0095] The method provided by the disclosed embodiment can be applied to PB-level data scales, and is suitable for relational data knowledge graph analysis scenarios in big data projects in various industries. Comprehensively consider the advantages of the Spark engine's computing power in batch processing, while also considering the storage and query advantages of distributed file storage systems and column storage systems, and then combine the computing power of the Janus Graph database itself; through reasonable planning of multiple big data components, avoid its own shortcomings and achieve the best solution. At the same time, the big data components of this solution are all planned based on the Hadoop distributed file system, support distributed structure, and have good horizontal scalability. At the same time, the server configuration requirements are medium, and the server hard disk can choose a large-capacity SATA disk to reduce its investment cost and have a high cost performance. It also reduces its storage cost.

[0096] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0097] For further reference, Figure 5 As shown, in the implementation of this example, a data processing system 50 based on a knowledge graph is also provided, including:

[0098] The partition storage module 501 is used to obtain the data to be processed and partition and store the data to be processed according to time attributes;

[0099] The relationship detail data extraction module 502 is used to load the to-be-processed data corresponding to each storage partition in partitions, create an external table, and extract the corresponding relationship detail data;

[0100] Aggregation processing module 503, used to perform relationship aggregation processing on the relationship detail data, and output relationship result data based on a pre-configured result data standard data structure;

[0101] The database update module 504 is used to incrementally update the relationship result data into the graph database, and update the entity attributes and relationship attributes based on the entity relationship.

[0102] In some exemplary embodiments, the system further comprises:

[0103] The data cleaning module is used to clean the relationship detail data and define a row-key generation rule; based on the row-key generation rule, the cleaned relationship detail data is batch processed and saved to the Hbase database.

[0104] In some exemplary embodiments, the system further comprises:

[0105] A query request processing module is used to obtain a relational data query request based on a graph database; wherein the relational data query request includes entity query information and / or relationship query information; a row-key search is performed in the Hbase database based on the master record ID of the relationship to obtain data retrieval results; a graph relationship is constructed based on the data retrieval results and displayed in a graphical user interface.

[0106] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0107] Figure 6 A schematic diagram of an electronic device suitable for implementing an embodiment of the present invention is shown.

[0108] It should be noted that Figure 6 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0109] like Figure 6 As shown, the electronic device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0110] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0111] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0112] Specifically, the electronic device may be a smart mobile electronic device such as a mobile phone, a tablet computer or a laptop computer, or may be a smart electronic device such as a desktop computer, or a smart wearable device.

[0113] It should be noted that the storage medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any storage medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0114] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0116] It should be noted that, as another aspect, the present application also provides a storage medium, which can be included in an electronic device; or it can exist independently without being installed in the electronic device. The above storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device can implement the following Figure 2 and / or Figure 3 The individual steps of the method are shown.

[0117] In one embodiment, the present application provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0118] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0119] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0120] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method based on knowledge graph, characterized in that: The method comprises: Acquire the data to be processed, and partition and store the data to be processed according to time attributes; Partition loading the data to be processed corresponding to each storage partition, creating an external table, and extracting corresponding relationship detail data; Performing relationship aggregation processing on the relationship detail data, and outputting relationship result data based on a pre-configured result data standard data structure; The relationship result data is incrementally updated to the graph database, and entity attributes and relationship attributes are updated based on entity relationships.

2. The method according to claim 1, characterized in that The method further comprises: Performing data cleaning on the relationship detail data and defining row-key generation rules; Based on the row-key generation rule, the relationship detail data after data cleaning is batch processed and saved to the Hbase database.

3. The method according to claim 2, characterized in that The method further comprises: Obtaining a relational data query request based on a graph database; wherein the relational data query request includes entity query information and / or relationship query information; Perform row-key search in the Hbase database based on the master record ID of the relationship to obtain data search results; A graph relationship is constructed based on the data retrieval results and displayed in a graphical user interface.

4. The method according to claim 3, characterized in that The relationship data query request also includes: time query information; The method further comprises: According to the time query information, data is searched in the Hbase database using the time attribute to obtain a data search result.

5. The method according to claim 1, characterized in that The obtaining of the data to be processed and partitioning and storing the data to be processed according to time attributes includes: The data to be processed is compressed, connected to the HDFS file system in Parquet file format, and partitioned and stored by day.

6. The method according to claim 1, characterized in that When extracting the corresponding relationship detail data, the method further includes: The relationship detail data is classified according to a pre-configured standardized relationship data structure, so as to perform data aggregation processing according to the classification result; wherein the standardized relationship data structure includes multiple relationship types.

7. The method according to claim 1, characterized in that The method further comprises: In the graph database, entity types are defined based on a preconfigured standard data structure of result data, and graph definition is performed according to relationship types.

8. A data processing system based on knowledge graph, characterized in that: The system comprises: A partition storage module is used to obtain the data to be processed and partition and store the data to be processed according to time attributes; A relationship detail data extraction module is used to load the to-be-processed data corresponding to each storage partition in partitions, create an external table, and extract the corresponding relationship detail data; An aggregation processing module, used for performing relationship aggregation processing on the relationship detail data, and outputting relationship result data based on a pre-configured result data standard data structure; The database update module is used to incrementally update the relationship result data into the graph database, and update the entity attributes and relationship attributes based on the entity relationship.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the knowledge graph-based data processing method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the knowledge graph-based data processing method described in any one of claims 1 to 7 by executing the executable instructions.

Images