A data management system and method

Abstract

This application discloses a data management system and method. The data management system includes a data service module, a database change data capture module, a raw database, a graph database, and a search engine. The data service module receives write data through a data write interface, processes the data, and transmits it to the raw database for storage. The write data includes only entity data and relation data. The database change data capture module listens to the raw database and obtains change data, writing the change data to an internal message queue. The graph database and search engine store change messages obtained from the internal message queue. This application abstracts data into two types: "entities" and "relationships," stores the data in the raw database, and achieves data linkage between the graph database and the search engine by listening to the raw database, ensuring data consistency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data management system and method. Background Technology

[0002] For information systems, raw databases are typically used for storage. In recent years, with the development of information technology, we often have the following requirements in practical application scenarios:

[0003] 1. Information systems often need to describe complex relationships between different objects. When storing a large number of nodes and relationships, the retrieval efficiency of raw databases is not high. Therefore, the industry currently widely uses graph databases such as Neo4j to store nodes and relationships to facilitate retrieval.

[0004] 2. Users often need to perform fast full-text searches, retrieving objects whose fields contain certain keywords, and performing paginated or statistical queries. However, raw databases are not very efficient at handling full-text searches, so the industry generally uses search engines (such as Elasticsearch) to implement full-text search for semi-structured data.

[0005] In practice, graph databases and search engines often need to be used together, so the storage and analysis technologies for both types of data must ensure data consistency. However, currently there is no data linkage mechanism between graph databases and search engines, and the data in both graph databases and search engines needs to be maintained separately. Summary of the Invention

[0006] This application provides a data management system and method that abstracts data into two types: "entities" and "relationships," stores the data in the original database, and achieves data linkage between the graph database and the search engine by monitoring the original database, thereby ensuring data consistency.

[0007] This application provides a data management system, including a data service module, a database change data capture module, a raw database, a graph database, and a search engine;

[0008] The data service module receives write data through the data write interface, processes the data, and then transmits it to the original database for storage; the write data includes only entity data and relation data.

[0009] The database change data capture module listens to the original database and obtains the change data, and writes the change data to the internal message queue.

[0010] Graph databases and search engines store change messages obtained from internal message queues.

[0011] Preferably, the graph database stores entity data and relation data, while the search engine only stores entity data that needs to support search functionality.

[0012] Preferably, the graph database has a relational query / graph query interface, and the data service module has a retrieval interface. The data service module forwards the data received by the retrieval interface to the search engine.

[0013] Preferably, the data management system further includes at least one customized message queue, which interacts with external users.

[0014] A custom message queue listens to the internal message queue and generates change messages adapted to the needs of the corresponding external users, which are then used by the external users.

[0015] Preferably, the raw database, graph database, and / or search engine are clustered raw databases, clustered graph databases, and clustered search engines.

[0016] This application also provides a data management method, including:

[0017] The data access module receives the first write data through the data write interface and performs the first data processing to obtain the second write data; wherein, the first write data only includes entity data and relation data;

[0018] The original database receives the second write data;

[0019] The database change data capture module listens to the original database and obtains the change data, and writes the change data to the internal message queue.

[0020] The graph database and search engine write the first change message obtained by performing a second data processing on the changed data in the internal message queue.

[0021] Preferably, the graph database writes the structured data information of the entities, and the search engine writes the unstructured text fields of the entities.

[0022] Preferably, the data management method further includes:

[0023] Based on the needs of external users, the changed data in the internal message queue is processed in the third way to obtain the second changed message and output it to a customized message queue for communication with external users.

[0024] Preferably, the first data processing includes verifying whether both ends of the relational data have entities, and verifying whether each field of the entity data and relational data conforms to a predefined value range.

[0025] Preferably, the second data processing includes:

[0026] Distinguish whether the changed data belongs to entity data changes or relational data changes, obtain attribute change results, and determine the target database to be written to based on the attribute change results. The target database includes graph databases and / or search engines.

[0027] Identify whether the changed data was added, deleted, or modified to determine the type of change.

[0028] Based on the change method, the format of the changed data is converted into a data format that is compatible with the target database to obtain the first change message.

[0029] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0031] Figure 1 A schematic diagram of the structure of the first embodiment of the data management system provided in this application;

[0032] Figure 2 A schematic diagram of the structure of the second embodiment of the data management system provided in this application;

[0033] Figure 3 A flowchart of the data management method provided for this application. Detailed Implementation

[0034] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0035] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0036] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0037] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0038] This application provides a data management system and method that abstracts data into two types: "entities" and "relationships," stores the data in the original database, and achieves data linkage between the graph database and the search engine by monitoring the original database, thereby ensuring data consistency.

[0039] Example 1

[0040] like Figure 1 As shown, the data management system includes a data service module, a database change data capture module, a raw database, a graph database, and a search engine.

[0041] like Figure 1 As shown, the data service module includes a data write interface ( Figure 1 The text appears to be a mix of seemingly unrelated phrases and sentences, making it impossible to translate coherently. It seems to be Figure 1 (The text in the image is an example of a full-text search API.)

[0042] The data service module receives write data through the data write interface, processes the data, and then transmits it to the original database for storage.

[0043] In this application, data objects are abstracted into "entities" and "relationships." An "entity" refers to an object with its own attribute values, such as "user," "organization," "server," "cluster," and "application." A "relationship" describes the state of interaction and mutual influence between entities, such as which organization a user "belongs" to, or which cluster a server "belongs" to. Therefore, the data written only includes entity data and relation data.

[0044] Before writing data, administrators need to pre-define entity types and relationship types in the data service module, including information such as the fields of each entity and relationship and their allowed value ranges. When writing data of relationship type, the data service module needs to verify whether both ends of the relationship have corresponding entities. When writing data of entity type and relationship type, the data service module will verify whether each field conforms to the predefined value range.

[0045] like Figure 1 As shown in the dashed box in the upper left corner, data input through the data write interface enters the ingress message queue. The data written to the ingress message queue needs to be processed by message processing logic to obtain the message format required by the user before it can be transmitted to the original database. This message processing logic is customized based on the requirements corresponding to the business functions of this data management system. The message processing logic differs for different business systems.

[0046] The retrieval interface is used to receive data (mainly keywords) input by the retrieval user and forward it to the search engine to realize functions such as full-text search.

[0047] After receiving the write data from the data service module, the original database stores it. Accordingly, the data in the original database only includes relational data and entity data. It should be noted that the original database supports the extraction of changed data using change data retrieval techniques. As an example, the original database can be a relational database (e.g., MySQL) or a non-relational database (e.g., Cassandra).

[0048] Database Change Data Capture (CDC) module Figure 1 The CDC software (shown in the image) listens to the raw database and obtains changed data, then writes the changed data to an internal message queue. As an example, the CDC software could be software like Debezium or Oracle Golden Gate, which is capable of acquiring changed data.

[0049] The graph database and search engine store change messages obtained from the internal message queue. Specifically, the change log processing module processes the changed data in the internal message queue to obtain change messages corresponding to the graph database and search engine, respectively. Specifically, the change log processing module distinguishes whether the change occurred in entity data or relational data, identifies whether the change was an add, delete, or update, and converts the changed data to the format required by the graph database and search engine before writing it to them. It should be noted that the same changed data must be written to both the graph database and the search engine simultaneously to ensure data consistency; it is not permissible to write only to the graph database or only to the search engine.

[0050] As an example, the graph database writes structured data information of entities, while the search engine writes unstructured text fields of entities.

[0051] In this system, the graph database stores entity and relational data, while the search engine only stores entity data necessary for search functionality. Some large fields are not very meaningful in the graph database and are therefore only stored in the search engine. For example, if "article" is treated as an entity, and the article's body is quite long and occupies a significant amount of space, then the article's body field is only stored in the search engine, while the graph database only stores information such as the article's identifier, title, and author, not the body itself.

[0052] When writing data, it is necessary to ensure that changes to entity data precede changes to relational data. For example, when a user changes their work department, it is necessary to first ensure that the entities of user and work department already exist in the data management system before adding the relationship between the user and work department.

[0053] The graph database has a relational query / graph query interface, supporting queries with complex relational conditions. The data service module has a retrieval interface, which forwards the data received by the retrieval interface to the search engine, which supports keyword-based retrieval.

[0054] Since graph databases and search engines are both non-relational databases, they support dynamic adjustment of data structures and can dynamically change fields, thus offering high flexibility.

[0055] Preferably, the data management system also includes at least one customized message queue, which interacts with external users. The customized message queue listens to the internal message queue and generates change messages adapted to the needs of the corresponding external users, for use by the external users (e.g., Hadoop-based big data platforms). Specifically, such as... Figure 1 As shown, changed data in the internal message queue is processed by message processing logic into changed messages adapted to the needs of external users. This message processing logic is customized based on different business output requirements. For example, the changed data in the internal message queue is converted into the message format required by the data lake and input into the data lake through the message processing logic corresponding to the data lake.

[0056] As an example, when the data management system handles a large volume of data (e.g., hundreds of millions, billions, or tens of billions), the system supports replacing some key components to handle the large data volume.

[0057] As an example, a non-relational database (such as Cassandra) is used as the original database. This type of non-relational database supports increasing data capacity and concurrent processing capabilities by horizontally scaling nodes.

[0058] As another embodiment, the raw database, graph database, and / or search engine can be a clustered raw database, a clustered graph database, or a clustered search engine.

[0059] like Figure 2 As shown, the data management system comprises a cluster of primary databases consisting of multiple relational databases, a graph database cluster consisting of multiple graph databases, and an Elasticsearch cluster consisting of multiple Elasticsearch search engines. Correspondingly, each relational database corresponds to a data service module.

[0060] As an example, an open-source cluster graph database such as Nebula is used, which can increase the number of nodes and edges it can support to the tens of billions.

[0061] As an example, container technology is used to extend the data query module, and a Redis-based memory cache can be added if necessary.

[0062] Based on this architecture, entity and relational data can be categorized at multiple levels. For a given entity object, each data service module specifies the content of its various fields, including but not limited to field names and value ranges. When data is written via the API interface, each data service module automatically verifies whether the written entity and relational data conforms to its predefined rules, such as whether the value range of a certain field matches the predefined rules, thus achieving automatic classification of the written data.

[0063] This improves the data carrying capacity and concurrent processing capability of the data management system, enhances the query performance per second (QPS) metric, and allows the data query interface to further accelerate query speed and performance using memory caching technologies such as Redis and Memcached.

[0064] As an example, a company may have different types of organizational structures, such as administrative organizations and party-building organizations. A user may belong to different types of organizations simultaneously (for example, user U1 may belong to organization A1 administratively, and also belong to party branch B1 as a party member), or may belong to different organizations with different affiliations (user U1 may belong to organization A1 administratively, be seconded to organization A2, and hold a part-time position in organization A3). These three organizations, A1, A2, and A3, all belong to a higher-level organization A0. A user's entity information may include their work experience and date of birth, among other things.

[0065] Therefore, the enterprise includes three types of entities: administrative organizations, party building organizations, and users, as well as three types of relationships: affiliation, secondment, and part-time.

[0066] Therefore, the following process is included when creating the enterprise's data management system:

[0067] 1. Define the above 3 entity types and 3 relationship types in the data service module.

[0068] 2. Write five organizational entities A0, A1, A2, A3, and B1, and one user entity U1 through the data writing API interface.

[0069] 3. Write the relationship information between entities through the data writing API interface, that is, A1, A2, and A3 all belong to A0, U1 belongs to A1, is seconded to A2, works part-time with A3, and belongs to B1.

[0070] After entity and relationship data is written via the API, the data is automatically stored in the graph database and search engine. Then, the following queries can be performed using the query interfaces of the graph database and search engine:

[0071] 1. Search for all staff members under a certain organization who have both subordinate and seconded relationships and belong to at least one party branch.

[0072] 2. Search for all staff members under a specific Party organization whose work experience includes the keyword "xx factory", and sort them by date of birth.

[0073] Example 2

[0074] Based on the aforementioned data management system, this application also provides a data management method. For example... Figure 3 As shown, data management methods include:

[0075] S310: The data access module receives the first write data through the data write interface and performs the first data processing to obtain the second write data; wherein, the first write data only includes entity data and relation data.

[0076] Specifically, the first data processing includes verifying whether both ends of the relational data have entities, and verifying whether each field of the entity data and relational data conforms to a predefined range of values.

[0077] S320: The raw database receives the second write data.

[0078] S330: The database change data capture module listens to the original database and obtains the change data, and writes the change data to the internal message queue.

[0079] S340: The graph database and search engine write the first change message obtained by performing second data processing on the changed data in the internal message queue.

[0080] Specifically, the second data processing includes:

[0081] P1: Identify whether the changed data pertains to entity data or relational data, obtain the attribute change results, and determine the target database to be written to based on the attribute change results. The target database includes graph databases and / or search engines. For example, if there are changes to entity data, the target databases are graph databases and search engines; if there are changes to relational data, the target database is a graph database.

[0082] P2: Identify whether the changed data is added, deleted, or modified, and obtain the type of change.

[0083] P3: Based on the change method, convert the format of the changed data into a data format that is compatible with the target database to obtain the first change message.

[0084] Preferably, the data management method further includes:

[0085] S350: Based on the needs of external users, it performs third-party data processing on the changed data in the internal message queue, obtains the second changed message, and outputs it to a customized message queue for communication with external users.

[0086] The beneficial effects of this application are as follows:

[0087] 1. Universality: The data management system of this application only has two data types, "entity" and "relationship", which are highly abstract. Therefore, this application is a unified underlying data architecture that can express complex relationships and support full-text search. It can be widely applied to a large number of data scenarios and can be reused in multiple scenarios, reducing the complexity of development and improving the development speed.

[0088] 2. This application combines the advantages of graph databases and search engines. Graph databases are highly efficient in performing complex relational queries, such as iterative queries, while search engines are highly efficient in performing full-text searches. Users can seamlessly switch between graph queries and full-text searches according to the needs of the actual scenario, making it suitable for most scenarios.

[0089] 3. Write once, use everywhere: Data is written only through the original database. Subsequent data writing to the graph database and search engine depends on the data change messages of the original database, without the need for separate operations, thus ensuring data consistency.

[0090] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A data management system, characterized by, The system comprises a data service module, a database change data capturing module, an original database, a graph database and a search engine, wherein the graph database stores entity data and relationship data, and the search engine only stores entity data required to support search function; The data service module receives write data through a data write interface and transmits the write data to the original database for storage after data processing; wherein the write data only comprises entity data and relationship data; The database change data capturing module listens to the original database and obtains change data, and writes the change data into an internal message queue; The graph database and the search engine store change messages obtained from the internal message queue, wherein the change data in the internal message queue is processed by a change log processing module to obtain change messages corresponding to the graph database and the search engine respectively, the change log processing module distinguishes whether the entity data or the relationship data has changed, distinguishes which one of addition, deletion and modification the change is, and writes the change data into the graph database and the search engine after converting the format of the change data into the format required by the graph database and the search engine; and the same change data needs to be written into the graph database and the search engine at the same time to ensure data consistency.

2. The data management system of claim 1, wherein, The graph database has a relationship query / graph query interface, and the data service module has a retrieval interface, the data service module forwards data received by the retrieval interface to the search engine.

3. The data management system of claim 1, wherein, Further comprising at least one customized message queue, the customized message queue interacts with external users; The customized message queue listens to the internal message queue and forms change messages adapted to the needs of corresponding external users for use by the external users.

4. The data management system of claim 1, wherein, The original database, the graph database and / or the search engine are cluster type original database, cluster type graph database and cluster type search engine.

5. A data management method characterized by, The system comprises: A data access module receives first write data through a data write interface and performs first data processing to obtain second write data; wherein the first write data only comprises entity data and relationship data; An original database receives the second write data; A database change data capturing module listens to the original database and obtains change data, and writes the change data into an internal message queue; A graph database and a search engine write first change messages obtained by performing second data processing on the change data in the internal message queue, wherein the graph database writes structured data information of entities, and the search engine writes unstructured text fields of entities; The second data processing comprises: Distinguishing whether the change data belongs to change of entity data or change of relationship data, obtaining an attribute change result, determining a target database required to be written according to the attribute change result, the target database comprising the graph database and / or the search engine, if there is change of entity data, the target database is the graph database and the search engine, and if there is change of relationship data, the target database is the graph database; The system comprises: distinguishing whether the change data belongs to addition data, deletion data or modification data, obtaining a change mode; converting a format of the change data into a data format suitable for the target database according to the change mode, and obtaining the first change message.

6. The data management method according to claim 5, characterized by, Further comprising: performing third data processing on the change data in the internal message queue according to a requirement of an external user, obtaining a second change message and outputting the second change message to a customized message queue in communication with the external user.

7. The data management method of claim 5, wherein, The first data processing comprises checking whether both ends of the relationship data have entities, and checking whether each field of the entity data and the relationship data conforms to a pre-defined value range.

Images