Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring specified business data in a target data source, and constructing a graph node and a node relation topological graph for a target entity in the specified business data in a memory; in response to a node change event for the target data source, updating the graph node and / or the node relation topological graph in the memory according to the node content of the change node and the type of the node change event; or, in response to a query instruction for the graph node and / or the node relation topological graph, querying the target graph node and / or the target node relation from the newest node relation topological graph in the memory according to a query condition indicated by the query instruction; and obtaining corresponding target business data and / or target business relation data from the target data source according to the queried target graph node and / or target node relation. The data relation query efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] The data in the information world is complex and intricate, and the relationships between data can reflect many phenomena and trends, such as social relationships and transaction relationships between users, and equity relationships between companies.

[0003] When processing data relational networks, existing technologies usually have difficulty in satisfying real-time queries and have the technical problem of low query efficiency. Summary of the invention

[0004] In view of this, an object of the present disclosure is to provide a data processing method, device, electronic device and storage medium to improve the efficiency of data relationship query.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for processing data, the method comprising: obtaining specified business data in a target data source, and constructing in a memory a graph node and a node relationship topology graph for a target entity in the specified business data; wherein the graph node is used to indicate index information of the target entity in the memory; in response to a node change event for the target data source, updating the graph node and / or the node relationship topology graph in the memory according to the node content of the changed node and the type of the node change event; or, in response to a query instruction for the graph node and / or the node relationship topology graph, querying the target graph node and / or target node relationship from the latest node relationship topology graph in the memory according to the query conditions indicated by the query instruction; and obtaining corresponding target business data and / or target business relationship data from the target data source according to the queried target graph node and / or target node relationship.

[0006] In a second aspect, an embodiment of the present disclosure provides a data processing device, which includes: a construction module, which is used to obtain specified business data in a target data source, and construct a graph node and a node relationship topology map for a target entity in the specified business data in a memory; wherein the graph node is used to indicate index information of the target entity in the memory; an update module, which is used to respond to a node change event for the target data source, and update the graph node and / or the node relationship topology map in the memory according to the node content of the change node and the type of the node change event; or, a query module, which is used to respond to a query instruction for the graph node and / or the node relationship topology map, and query the target graph node and / or target node relationship from the latest node relationship topology map in the memory according to the query condition indicated by the query instruction; an acquisition module, which is used to acquire corresponding target business data and / or target business relationship data from the target data source according to the queried target graph node and / or target node relationship.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned data processing method.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data processing method.

[0009] The embodiments of the present disclosure bring the following beneficial effects:

[0010] The above-mentioned data processing method, device, electronic device and storage medium construct a graph node and node relationship topology diagram of business data in memory and update it synchronously with the data source, so that the query of data or data relationship does not need to directly operate the data source, and can also ensure the real-time and high efficiency of the query. For complex data relationships, it overcomes the disadvantage that the database cannot meet multi-hop queries, and data relationship queries are more real-time and efficient.

[0011] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0012] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart of an embodiment of a method for processing data in an embodiment of the present disclosure;

[0015] Figure 2 A schematic diagram of a method for processing data in an embodiment of the present disclosure;

[0016] Figure 3 is another schematic diagram of the method for processing data in an embodiment of the present disclosure;

[0017] Figure 4 is another schematic diagram of the method for processing data in an embodiment of the present disclosure;

[0018] Figure 5 A schematic diagram of a data processing device provided in an embodiment of the present disclosure;

[0019] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0021] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] For ease of understanding, the specific process of the embodiment of the present disclosure is described below. Figure 1 , an embodiment of the data processing method in the embodiment of the present disclosure includes:

[0023] Step S10: obtaining designated business data in the target data source, and constructing a graph node and a node relationship topology graph for the target entity in the designated business data in the memory; wherein the graph node is used to indicate the index information of the target entity in the memory;

[0024] It can be understood that the target data source may include one or more databases, and the types of databases may be relational databases and non-relational databases, such as relational databases MySQL, PostgresSQL, etc., and non-relational databases such as MongoDB, Elasticsearch, etc. The embodiments of the present disclosure support target data sources of any database type, so that the processing of data relationships can be more flexible.

[0025] In this implementation, the designated business data may be any part / all of the data in the target data source. In one implementation, the data in a designated field in the target data source may be determined as the designated business data. Multiple designated fields may be specified, and the field value of the designated field corresponding to each target entity may be used as the node attribute value of the graph node corresponding to the target entity, so that the designated attributes may also be efficiently queried in memory.

[0026] It should be noted that the target entity refers to the object for which a data relationship needs to be constructed. For example, assuming that the object for which a data relationship needs to be constructed is a user on a social platform, then the target entity is the user, which may specifically refer to the user's identifier (identity document, ID) in the target data source / specified business data. For another example, assuming that the object for which a data relationship needs to be constructed is an account on a trading platform, then the target entity is the account, and the graph nodes of the target entity reflect the index information of each account in the memory, and the node relationship topology graph reflects the association relationship between accounts.

[0027] In this implementation, the target entity is an abstract concept that can include multiple entity objects, where each entity object corresponds to a graph node, and the node relationship topology graph is a relationship graph of the graph nodes of all entity objects. Specifically, the target entity can be an identifier field specified in the target data source, which is unique in the target data source, such as a user ID, account ID, transaction serial number, etc., which is not limited here.

[0028] It should be noted that the node relationship topology graph is a topology graph model, which is a data structure that represents entities and relationships with points (Vertex / Node) and edges (Edge). The graph node is the point in the data structure, which is identified by the index information of the target entity in the memory, where each target entity corresponds to an index information. The index information can be generated when constructing the graph node, or it can be the identifier of the target entity in the target data source, which is not limited here.

[0029] It can be understood that the graph nodes and node relationship topology in the memory only need to be constructed once, that is, this step can be executed once, and the subsequent steps S20-S30 can be executed multiple times, and can be executed as long as the trigger conditions of the corresponding steps are met. This step is actually the process of synchronizing the entire amount of specified business data in the target data source into graph nodes and node relationship topology. Step S20 is an incremental synchronization process.

[0030] Step S20, in response to a node change event for a target data source, updating the graph nodes and / or the node relationship topology graph in the memory according to the node content of the changed node and the type of the node change event; or,

[0031] In this implementation, by monitoring the node change events of the target data source, the update of the graph nodes and / or node relationship topology graph in the memory is triggered, so that the search results of business data or business relationship data will not lag behind, the latest data can be synchronized in real time, and the real-time performance of data relationship queries is improved.

[0032] It should be noted that the types of node change events include node addition events, node update events and node deletion events. Different types of node change events have different corresponding update logics. For example, the node addition event calls the node addition logic, and the update objects include the graph nodes in the memory and the node relationship topology map; the node update event calls the node update logic, and the update objects include the graph nodes in the memory. It can also determine whether to update the node relationship topology map based on the node content of the changed node; and the node deletion event calls the node deletion logic, and the update objects include the graph nodes in the memory and the node relationship topology map.

[0033] It can be understood that the node change event for the target data source can be triggered by any change in the business data in the target data source that is related to the constructed graph node or node relationship topology graph. The type of change in business data is different, and the type of corresponding node change event is also different. For example, if the attribute of a target entity in the business data changes, a node update event can be triggered. If a new target entity is added to the business data, a node new event can be triggered. The specific details are not limited here.

[0034] In this implementation, what is updated are the graph nodes and / or node relationship topology in the memory, so that the graph nodes and node relationship topology in the memory can be kept up to date and updated synchronously with the target data source, thereby improving the real-time performance of data or data relationship queries.

[0035] Step S30, in response to a query instruction for a node relationship topological graph, querying a target graph node and / or a target node relationship from the latest node relationship topological graph in the memory according to a query condition indicated by the query instruction;

[0036] In this implementation, when it is necessary to query the target entity or entity relationship, the target graph nodes and / or target node relationships that meet the query conditions can be queried from the latest node relationship topology graph in the memory through query instructions, so that the query of data relationships does not require complex multi-hop query operations on the source database, thereby improving query efficiency and ensuring the real-time nature of the query.

[0037] It is understandable that the query instruction for the node relationship topology graph can query the target graph nodes and / or target node relationships in the node relationship topology graph that meet the query conditions, and different query instructions can be triggered through different query interfaces. Among them, the target node relationship refers to a node relationship chain in the node relationship topology graph, including at least two graph nodes, which is not specifically limited here.

[0038] Step S40: Acquire corresponding target business data and / or target business relationship data from the target data source according to the queried target graph nodes and / or target node relationships.

[0039] It can be understood that since there is a corresponding relationship between the graph nodes in the memory and the data in the target data source, in this implementation, based on the queried target graph nodes and / or target node relationships, the target business data corresponding to the target graph nodes and / or the target business relationship data corresponding to the target node relationships can be obtained from the target data source or specified business data, wherein the target business relationship data corresponding to the target node relationships include the target business data corresponding to each graph node in the target node relationship, which is not specifically limited here.

[0040] In one implementation, in addition to querying the graph nodes and node relationship topology graph in the memory, other operations may also be performed, such as node relationship definition and query of node relationship definition, etc., which will not be described in detail here.

[0041] The data processing method provided in the above implementation mode constructs a graph node and node relationship topology graph of business data in memory and updates it synchronously with the data source, so that the query of data or data relationship does not need to directly operate the data source, and the real-time and high efficiency of the query can be guaranteed. For complex data relationships, it overcomes the disadvantage that the database cannot meet multi-hop queries, and data relationship queries are more real-time and efficient.

[0042] Next, the specific data processing method is explained.

[0043] In one embodiment, when obtaining specified business data in a target data source and constructing a graph node and a node relationship topology graph for a target entity in the specified business data in memory, data of a specified field in the target data source is obtained to obtain the specified business data; based on the field value of the target field in the specified business data, a graph node corresponding to each entity object is constructed in memory; wherein the target field is used to identify the entity object of the target entity; and based on the relationship between the entity objects in the specified business data, a node relationship topology graph between the graph nodes is constructed in memory.

[0044] It is understandable that when triggering a full synchronization instruction, the fields to be synchronized can be specified, and in response to the full synchronization instruction, the data of the specified fields indicated by the full synchronization instruction in the target data source is obtained, thereby obtaining the specified business data. Among them, the specified fields include the target fields, and the target fields are the identifiers of the entity objects in the target data source as the target entities. For example, if the target entity is a user, then the target field is the ID of the user in the target data source. If the target entity is a trading account, then the target field is the ID of the trading account in the target data source. The specifics are not limited here.

[0045] It should be noted that the designated fields may also include fields for indicating designated business attributes of the target entity, such as the user's age, gender, city, etc. The designated fields may also include the table name of the data table to which the entity object belongs in the target data source, wherein the table name may be combined with the field value of the target field to identify the entity object of the target entity, thereby increasing the probability of uniqueness of the entity object in memory.

[0046] In this implementation, based on the identifier of the entity object in the specified business data, that is, the field value of the target field, a graph node corresponding to each entity object can be created in memory, wherein the identifier of the entity object can also be used as the identifier of the graph node in memory, and the identifier of the graph node in memory can also be other, which is not limited here. After constructing the graph node, the association relationship between the graph nodes can be constructed in memory based on the relationship between the entity objects, thereby obtaining a node relationship topology diagram.

[0047] In one embodiment, after constructing the graph node corresponding to each entity object in memory, it is possible to check whether the number of graph nodes in the memory is consistent with the number of target fields in the specified business data, and it is also possible to check whether the type of the graph node is consistent with the type of the corresponding data table in the specified business data, thereby verifying the consistency of the graph node in the memory with the target data source.

[0048] In one embodiment, when constructing a graph node corresponding to each entity object in memory based on the field value of the target field in the specified business data, memory space is applied for each field value of the target field in the specified business data to obtain a memory address corresponding to each entity object; a pointer is created for each memory address corresponding to the entity object to store the corresponding memory address, and the pointer and field value corresponding to each entity object are stored as a graph node corresponding to the entity object.

[0049] In this implementation, when constructing a graph node corresponding to each entity object in the memory, first apply for storage space in the memory, store the relevant information of the entity object in the applied storage space, obtain the memory address of the storage space, and then create a pointer to point to the memory address, and use the pointer as the identifier of the graph node. The relevant information of the entity object may include the field value of the target field, and may also include other business attribute values ​​of the entity object and update information of the graph node (such as update time), etc., which are not limited here.

[0050] By way of example and not limitation, Figure 2 It is a visual schematic diagram of a graph node, including relevant information of an entity object corresponding to the graph node node1, wherein node1 may be a pointer corresponding to the graph node, which is not specifically limited here.

[0051] In one embodiment, when constructing a node relationship topology graph between graph nodes in memory based on the relationship between entity objects in specified business data, the pointer of each entity object and the pointer of at least one associated entity object are stored in a key-value pair structure based on the relationship between entity objects in the specified business data to obtain a node relationship topology graph between graph nodes.

[0052] In this implementation, based on the association relationship that needs to be established between entity objects in the specified business data, the associated entity object corresponding to each entity object is determined, and then the pointer to the entity object is used as the key in a key-value pair data structure, and the pointer to the associated entity object corresponding to the entity object is used as the value in the key-value pair data structure, thereby obtaining a node relationship topology graph between graph nodes.

[0053] By way of example and not limitation, Figure 3 The figure shows a visual node relationship topology diagram. Figure 4The figure shows a schematic diagram of the node relationship topology stored in memory according to a specific data structure. Figure 3 You can intuitively view the association relationship between graph nodes (arrows indicate association relationships). Figure 4 It is a visual diagram of the storage structure of the node relationship topology graph. The cells on the left represent the pointers of the entity objects, that is, the memory addresses, and the cells on the right represent the pointers of the entity objects associated with the entity objects on the left. Figure 4 It can be seen that the node association relationship adopts a bilateral storage method, that is, the source node and the target node both store the relationship with each other, so that the associated nodes can be mined downward from any direction, improving the query efficiency of the node relationship.

[0054] In one embodiment, in response to a node change event for a target data source, when the graph nodes and / or the node relationship topology diagram in the memory are updated according to the node content of the changed node and the type of the node change event, the node change event of the target data source is subscribed to through a message system; in response to the node change event pushed by the message system, the graph nodes and / or the node relationship topology diagram in the memory are updated according to the node content of the changed node and the type of the node change event.

[0055] In this implementation, in order to synchronize the graph node-related data of the target data source to the memory in real time when it changes, the node change event of the target data source is first subscribed to through the message system, where the message system can be any system with message subscription and consumption functions, such as distributed messaging systems kafka, Jafka, messaging middleware MetaQ, RocketMQ, etc., which are not limited here.

[0056] It can be understood that after subscribing to the node change event of the target data source through the message system, when the data change of the target data source triggers the node change event, the message system will push the changed data in the form of a node change event to the node change service, and update the corresponding graph nodes and / or node relationship topology graph in the memory through the node change service, so that the processing of data relationships is more real-time.

[0057] In one embodiment, the types of node change events include node addition events, node update events, and node deletion events; when updating the graph nodes and / or node relationship topology in the memory according to the node content of the changed node and the type of the node change event, the change data corresponding to the node change event is obtained from the target data source through the adapter corresponding to the database type of the target data source, and the change data is converted into the node content of the changed node; for the node addition event, a graph node corresponding to the change data is created in the memory, and the node relationship topology in the memory is updated according to the graph node corresponding to the change data; for the node update event, the changed node is locked, and according to the node content of the changed node, the change content of the changed node in the memory and the node relationship topology are updated, and the changed node is unlocked; for the node deletion event, the changed node to be deleted is locked, the corresponding change node in the memory and the corresponding change node in the node relationship topology are deleted, and the changed node is unlocked.

[0058] It can be understood that nodes can be added, nodes can be updated, and nodes can be deleted for the graph nodes and node relationship topology graphs in the memory. In this implementation, different types of node change events trigger different node change logics, and different node change logics can correspond to different node change interfaces, which are not specifically limited here.

[0059] In this implementation, when responding to a node change event pushed by a message system, the adapter corresponding to the node change event is called to obtain the corresponding change data from the target data source, and then convert it into the node content of the graph node and store it in the memory. It can also be converted into a node relationship topology graph and stored in the memory, so that the disclosed embodiment can adapt to different database types and make data relationship processing more flexible.

[0060] For new node events, first check whether there is an identical graph node in the memory, specifically a graph node with the same identifier of the entity object. If not, create a new graph node in the memory in the same way as the above graph node construction method. If it exists, first lock the changed node, find the corresponding identical graph node from the memory according to the identifier of the changed node, update the necessary attributes, then find the associated node set corresponding to the changed node from the node relationship topology diagram and update it, and release the lock after the update is completed.

[0061] For node update events, first lock the changed node, find the corresponding identical graph node from the memory according to the identifier of the changed node, update the necessary attributes, then find the associated node set corresponding to the changed node from the node relationship topology graph and update it, and release the lock after the update is completed. The identifier of the changed node can be the identifier of the entity object corresponding to the changed node in the target data source, or the identifier of the changed node in the memory, which is not limited here.

[0062] For node deletion events, first lock the changed node, find the pointer from the memory according to the identifier of the changed node, then find the associated node set corresponding to the changed node from the node relationship topology graph and delete it, then delete the changed node, delete the corresponding record of the graph node in the memory, and finally release the lock.

[0063] It can be understood that the query instructions for the graph nodes and / or the node relationship topology graph include node query instructions, node relationship query instructions, and defined node relationship query instructions, etc., which are described in detail below.

[0064] In one embodiment, for a node query instruction, the identifier, business attribute, type, etc. of the graph node to be queried can be specified through query conditions. The target graph node that meets the query conditions and the graph nodes associated with the target graph node can be queried from the latest graph node in the memory, and the target business data of the target graph node in the target data source and the target business relationship data of the graph nodes associated with the target graph node can be output.

[0065] In one embodiment, for a node relationship query instruction, in response to a query instruction for a graph node and / or a node relationship topology graph, when querying a target graph node and / or a target node relationship from the latest node relationship topology graph in the memory according to the query condition indicated by the query instruction, in response to a node relationship query instruction for a graph node and / or a node relationship topology graph, query starting point data is obtained from the query condition indicated by the query instruction; according to the query starting point data, a target identifier of the entity object to be queried in the target data source is obtained from the target data source; a target graph node corresponding to the target identifier is queried from the latest graph node in the memory, and an associated graph node corresponding to the target graph node is queried from the latest node relationship topology graph in the memory; and the associated graph nodes are screened according to the node association condition indicated by the query condition to obtain the target node relationship.

[0066] When querying node relationships, the query conditions can be used to specify query starting point data, node filtering conditions, query node depth, query termination conditions, query algorithms and other information. The query algorithm can be a depth-first traversal algorithm, a breadth-first traversal algorithm, a shortest path algorithm and other node traversal algorithms, which are not specifically limited here.

[0067] In this implementation, the corresponding entity object is found from the target data source based on the query starting point data, and the target identifier of the corresponding entity object is obtained, such as the user's ID in the target data source. The target identifier is then used to find the pointer to the corresponding target graph node from the latest graph node in the memory. The pointer to the target graph node is used to find the pointer set of graph nodes associated with the target graph node from the latest node relationship topology graph in the memory. The pointer set is traversed to find the node content corresponding to each pointer, and check whether the node content meets the node filtering conditions. If it meets the conditions, the node is placed in the result set. If it does not meet the conditions, the node is skipped and the next pointer in the pointer set is checked. When the query termination conditions are met or the query node depth is met, the query is terminated and the result set is returned as the target node relationship.

[0068] It should be noted that if a defined node relationship needs to be queried, the node relationship in the memory must be defined first. In one embodiment, in response to a definition instruction for a node relationship topology graph, the latest node relationship topology graph in the memory is defined for a specified node relationship according to the node links and / or node filtering conditions indicated by the definition instruction, and an identifier for the specified node relationship is obtained.

[0069] For example, suppose a node relationship topology graph is A->B->C->D->E->F. Then, when only the node relationship of C->D->E needs to be queried, a node relationship topology graph can be defined with only the nodes C->D->E. The defined topology view can also be called a subgraph of the entire graph.

[0070] It should be noted that each time a topological view of a node relationship is defined, an identifier corresponding to the topological view is generated, so that when the identifier of the specified topological view is subsequently queried, the node and relationship data corresponding to the topological view can be queried, avoiding the need to pass the definition of the topological view every time a query is made.

[0071] Furthermore, after defining the node relationship, the defined node relationship can be queried through the defined node relationship query instruction, that is, by specifying the identifier of the node relationship to be queried through the defined node relationship query instruction, the node and relationship data of the corresponding node relationship can be obtained, including the target business data and the target business relationship data.

[0072] Corresponding to the above method embodiment, see Figure 5A schematic diagram of a data processing device is shown, the device comprising: a construction module 501, used to obtain specified business data in a target data source, and construct a graph node and a node relationship topology map for a target entity in the specified business data in a memory; wherein the graph node is used to indicate the index information of the target entity in the memory; an update module 502, used to respond to a node change event for the target data source, and update the graph node and / or the node relationship topology map in the memory according to the node content of the change node and the type of the node change event; or, a query module 503, used to respond to a query instruction for the graph node and / or the node relationship topology map, and query the target graph node and / or the target node relationship from the latest node relationship topology map in the memory according to the query condition indicated by the query instruction; an acquisition module 504, used to obtain corresponding target business data and / or target business relationship data from the target data source according to the queried target graph node and / or target node relationship.

[0073] The processing device for the above data constructs a graph node and node relationship topology graph of the business data in the memory and updates it synchronously with the data source, so that the query of data or data relationship does not need to directly operate the data source, and the real-time and high efficiency of the query can be guaranteed. For complex data relationships, it overcomes the disadvantage that the database cannot meet multi-hop queries, and the data relationship query is more real-time and efficient.

[0074] Optionally, the above-mentioned construction module 501 includes: a data acquisition unit, used to acquire data of a specified field in the target data source to obtain specified business data; a first construction unit, used to construct a graph node corresponding to each entity object in the memory according to the field value of the target field in the specified business data; wherein the target field is used to identify the entity object of the target entity; and a second construction unit, used to construct a node relationship topology graph between the graph nodes in the memory according to the relationship between the entity objects in the specified business data.

[0075] Optionally, the above-mentioned first construction unit is specifically used to: apply for memory space for each field value of the target field in the specified business data to obtain the memory address corresponding to each entity object; create a pointer for the memory address corresponding to each entity object to store the corresponding memory address, and store the pointer and field value corresponding to each entity object as a graph node corresponding to the entity object.

[0076] Optionally, the above-mentioned construction module 501 also includes: an associated storage unit, which is used to obtain at least one business attribute value of each entity object from the specified business data, and associate and store the business attribute value with the corresponding target graph node.

[0077] Optionally, the second construction unit is specifically used to: store the pointer of each entity object and the pointer of at least one associated entity object in a key-value pair structure according to the relationship between entity objects in the specified business data, and obtain a node relationship topology graph between the graph nodes.

[0078] Optionally, the above-mentioned update module 502 is specifically used to: subscribe to the node change event of the target data source through the message system; in response to the node change event pushed by the message system, update the graph node and / or the node relationship topology graph in the memory according to the node content of the changed node and the type of the node change event.

[0079] Optionally, the types of the node change event include node addition events, node update events and node deletion events; the above-mentioned update module 502 is specifically used to: obtain the change data corresponding to the node change event from the target data source through the adapter corresponding to the database type of the target data source, and convert the change data into the node content of the change node; for the node addition event, create a graph node corresponding to the change data in the memory, and update the node relationship topology map in the memory according to the graph node corresponding to the change data; for the node update event, lock the change node, update the change content of the change node in the memory and update the node relationship topology map according to the node content of the change node, and unlock the change node; for the node deletion event, lock the change node to be deleted, delete the corresponding change node in the memory and the corresponding change node in the node relationship topology map, and unlock the change node.

[0080] Optionally, the above-mentioned query module 503 is specifically used to: in response to a node relationship query instruction for the graph node and / or the node relationship topological graph, obtain query starting point data from the query condition indicated by the query instruction; obtain a target identifier of the entity object to be queried in the target data source from the target data source according to the query starting point data; query the target graph node corresponding to the target identifier from the latest graph node in the memory, and query the associated graph node corresponding to the target graph node from the latest node relationship topological graph in the memory; filter the associated graph nodes according to the node association conditions indicated by the query conditions to obtain the target node relationship.

[0081] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above data processing method. The electronic device can be a server or a terminal device.

[0082] See also Figure 6As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine executable instructions that can be executed by the processor 100. The processor 100 executes the machine executable instructions to implement the above data processing method.

[0083] Further, Figure 6 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0084] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware, for example:

[0086] Obtain specified business data from a target data source, and construct in memory a graph node and a node relationship topology for a target entity in the specified business data; wherein the graph node is used to indicate index information of the target entity in memory; in response to a node change event for the target data source, update the graph node and / or the node relationship topology in memory according to the node content of the changed node and the type of the node change event; or, in response to a query instruction for the graph node and / or the node relationship topology, query the target graph node and / or the target node relationship from the latest node relationship topology in memory according to the query conditions indicated by the query instruction; and according to the queried target graph node and / or target node relationship, obtain corresponding target business data and / or target business relationship data from the target data source.

[0087] In this method, by building a graph of business data nodes and node relationship topology in memory and updating it synchronously with the data source, the query of data or data relationships does not need to directly operate the data source, and the real-time and high-efficiency of the query can be guaranteed. For complex data relationships, it overcomes the disadvantage that the database cannot meet multi-hop queries, and data relationship queries are more real-time and efficient.

[0088] Optionally, the steps of obtaining specified business data in the target data source and constructing in memory the graph nodes and node relationship topology diagram for the target entity in the specified business data include: obtaining data of a specified field in the target data source to obtain the specified business data; constructing in memory a graph node corresponding to each entity object according to the field value of the target field in the specified business data; wherein the target field is used to identify the entity object of the target entity; and constructing in memory a node relationship topology diagram between graph nodes according to the relationship between entity objects in the specified business data.

[0089] Optionally, the step of constructing a graph node corresponding to each entity object in memory according to the field value of the target field in the specified business data includes: applying for memory space for each field value of the target field in the specified business data to obtain the memory address corresponding to each entity object; creating a pointer for the memory address corresponding to each entity object to store the corresponding memory address, and storing the pointer and field value corresponding to each entity object as the graph node corresponding to the entity object.

[0090] Optionally, after creating a pointer to store the corresponding memory address for each entity object, and storing the corresponding pointer and field value of each entity object as a graph node corresponding to the entity object, it also includes: obtaining at least one business attribute value of each entity object from specified business data, and associating the business attribute value with the corresponding target graph node for storage.

[0091] Optionally, the step of constructing a node relationship topology graph between graph nodes in memory according to the relationship between entity objects in specified business data includes: according to the relationship between entity objects in specified business data, storing the pointer of each entity object and the pointer of at least one associated entity object in a key-value pair structure to obtain a node relationship topology graph between graph nodes.

[0092] Optionally, in response to a node change event for a target data source, the step of updating the graph nodes and / or the node relationship topology diagram in the memory according to the node content of the changed node and the type of the node change event includes: subscribing to the node change event of the target data source through a message system; in response to the node change event pushed by the message system, updating the graph nodes and / or the node relationship topology diagram in the memory according to the node content of the changed node and the type of the node change event.

[0093] Optionally, the types of node change events include node addition events, node update events and node deletion events; the step of updating the graph nodes and / or the node relationship topology map in the memory according to the node content of the change node and the type of the node change event includes: obtaining the change data corresponding to the node change event from the target data source through the adapter corresponding to the database type of the target data source, and converting the change data into the node content of the change node; for the node addition event, creating a graph node corresponding to the change data in the memory, and updating the node relationship topology map in the memory according to the graph node corresponding to the change data; for the node update event, locking the change node, updating the change content of the change node in the memory and the node relationship topology map according to the node content of the change node, and unlocking the change node; for the node deletion event, locking the change node to be deleted, deleting the corresponding change node in the memory and the corresponding change node in the node relationship topology map, and unlocking the change node.

[0094] Optionally, in response to a query instruction for a graph node and / or a node relationship topology graph, the step of querying the target graph node and / or target node relationship from the latest node relationship topology graph in the memory according to the query conditions indicated by the query instruction includes: in response to a node relationship query instruction for a graph node and / or a node relationship topology graph, obtaining query starting point data from the query conditions indicated by the query instruction; obtaining a target identifier of the entity object to be queried in the target data source from the target data source according to the query starting point data; querying the target graph node corresponding to the target identifier from the latest graph node in the memory, and querying the associated graph node corresponding to the target graph node from the latest node relationship topology graph in the memory; screening the associated graph nodes according to the node association conditions indicated by the query conditions to obtain the target node relationship.

[0095] This embodiment further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above data processing method, for example:

[0096] Obtain specified business data from a target data source, and construct in memory a graph node and a node relationship topology for a target entity in the specified business data; wherein the graph node is used to indicate index information of the target entity in memory; in response to a node change event for the target data source, update the graph node and / or the node relationship topology in memory according to the node content of the changed node and the type of the node change event; or, in response to a query instruction for the graph node and / or the node relationship topology, query the target graph node and / or the target node relationship from the latest node relationship topology in memory according to the query conditions indicated by the query instruction; and according to the queried target graph node and / or target node relationship, obtain corresponding target business data and / or target business relationship data from the target data source.

[0097] In this method, by building a graph of business data nodes and node relationship topology in memory and updating it synchronously with the data source, the query of data or data relationships does not need to directly operate the data source, and the real-time and high-efficiency of the query can be guaranteed. For complex data relationships, it overcomes the disadvantage that the database cannot meet multi-hop queries, and data relationship queries are more real-time and efficient.

[0098] Optionally, the steps of obtaining specified business data in the target data source and constructing in memory the graph nodes and node relationship topology diagram for the target entity in the specified business data include: obtaining data of a specified field in the target data source to obtain the specified business data; constructing in memory a graph node corresponding to each entity object according to the field value of the target field in the specified business data; wherein the target field is used to identify the entity object of the target entity; and constructing in memory a node relationship topology diagram between graph nodes according to the relationship between entity objects in the specified business data.

[0099] Optionally, the step of constructing a graph node corresponding to each entity object in memory according to the field value of the target field in the specified business data includes: applying for memory space for each field value of the target field in the specified business data to obtain the memory address corresponding to each entity object; creating a pointer for the memory address corresponding to each entity object to store the corresponding memory address, and storing the pointer and field value corresponding to each entity object as the graph node corresponding to the entity object.

[0100] Optionally, after creating a pointer to store the corresponding memory address for each entity object, and storing the corresponding pointer and field value of each entity object as a graph node corresponding to the entity object, it also includes: obtaining at least one business attribute value of each entity object from specified business data, and associating the business attribute value with the corresponding target graph node for storage.

[0101] Optionally, the step of constructing a node relationship topology graph between graph nodes in memory according to the relationship between entity objects in specified business data includes: according to the relationship between entity objects in specified business data, storing the pointer of each entity object and the pointer of at least one associated entity object in a key-value pair structure to obtain a node relationship topology graph between graph nodes.

[0102] Optionally, in response to a node change event for a target data source, the step of updating the graph nodes and / or the node relationship topology diagram in the memory according to the node content of the changed node and the type of the node change event includes: subscribing to the node change event of the target data source through a message system; in response to the node change event pushed by the message system, updating the graph nodes and / or the node relationship topology diagram in the memory according to the node content of the changed node and the type of the node change event.

[0103] Optionally, the types of node change events include node addition events, node update events and node deletion events; the step of updating the graph nodes and / or the node relationship topology map in the memory according to the node content of the change node and the type of the node change event includes: obtaining the change data corresponding to the node change event from the target data source through the adapter corresponding to the database type of the target data source, and converting the change data into the node content of the change node; for the node addition event, creating a graph node corresponding to the change data in the memory, and updating the node relationship topology map in the memory according to the graph node corresponding to the change data; for the node update event, locking the change node, updating the change content of the change node in the memory and the node relationship topology map according to the node content of the change node, and unlocking the change node; for the node deletion event, locking the change node to be deleted, deleting the corresponding change node in the memory and the corresponding change node in the node relationship topology map, and unlocking the change node.

[0104] Optionally, in response to a query instruction for a graph node and / or a node relationship topology graph, the step of querying the target graph node and / or target node relationship from the latest node relationship topology graph in the memory according to the query conditions indicated by the query instruction includes: in response to a node relationship query instruction for a graph node and / or a node relationship topology graph, obtaining query starting point data from the query conditions indicated by the query instruction; obtaining a target identifier of the entity object to be queried in the target data source from the target data source according to the query starting point data; querying the target graph node corresponding to the target identifier from the latest graph node in the memory, and querying the associated graph node corresponding to the target graph node from the latest node relationship topology graph in the memory; screening the associated graph nodes according to the node association conditions indicated by the query conditions to obtain the target node relationship.

[0105] The data processing method, device, electronic device and computer program product of the storage medium provided in the embodiments of the present disclosure include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0107] In addition, in the description of the embodiments of the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0108] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0109] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0110] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above embodiments, those skilled in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining designated business data from a target data source, and constructing a graph node and a node relationship topology graph for a target entity in the designated business data in memory; wherein the graph node is used to indicate index information of the target entity in memory; In response to a node change event for the target data source, the graph node and / or the node relationship topology graph in the memory is updated according to the node content of the changed node and the type of the node change event; or, In response to a query instruction for the graph node and / or the node relationship topology graph, querying a target graph node and / or a target node relationship from the latest node relationship topology graph in the memory according to a query condition indicated by the query instruction; According to the queried target graph nodes and / or target node relationships, corresponding target business data and / or target business relationship data are acquired from the target data source.

2. The method according to claim 1, characterized in that The step of obtaining the specified business data in the target data source and constructing the graph nodes and node relationship topology graph for the target entity in the specified business data in the memory includes: Get the data of the specified field in the target data source to obtain the specified business data; According to the field value of the target field in the specified business data, a graph node corresponding to each entity object is constructed in the memory; wherein the target field is used to identify the entity object of the target entity; According to the relationship between the entity objects in the specified business data, a node relationship topology graph between the graph nodes is constructed in the memory.

3. The method according to claim 2, characterized in that The step of constructing a graph node corresponding to each entity object in memory according to the field value of the target field in the specified business data includes: Applying memory space for each field value of the target field in the specified business data to obtain the memory address corresponding to each entity object; A pointer is created for the memory address corresponding to each entity object to store the corresponding memory address, and the pointer and field value corresponding to each entity object are stored as the graph node corresponding to the entity object.

4. The method according to claim 3, characterized in that After the steps of creating a pointer to store the corresponding memory address for each entity object, and storing the corresponding pointer and field value of each entity object as a graph node corresponding to the entity object, the method further includes: At least one business attribute value of each entity object is obtained from the specified business data, and the business attribute value is associated with a corresponding target graph node and stored.

5. The method according to claim 3, characterized in that: The step of constructing a node relationship topology graph between the graph nodes in the memory according to the relationship between the entity objects in the specified business data includes: According to the relationship between entity objects in the specified business data, a pointer to each entity object and a pointer to at least one associated entity object are stored in a key-value pair structure to obtain a node relationship topology graph between the graph nodes.

6. The method according to claim 1, characterized in that The step of updating the graph node and / or the node relationship topology graph in the memory in response to the node change event for the target data source according to the node content of the changed node and the type of the node change event includes: Subscribe to the node change event of the target data source through the message system; In response to the node change event pushed by the message system, the graph node and / or the node relationship topology graph in the memory is updated according to the node content of the changed node and the type of the node change event.

7. The method according to claim 1, characterized in that The types of node change events include node addition events, node update events and node deletion events; The step of updating the graph node and / or the node relationship topology graph in the memory according to the node content of the changed node and the type of the node change event includes: Obtaining change data corresponding to the node change event from the target data source through an adapter corresponding to the database type of the target data source, and converting the change data into node content of the change node; For the newly added event of the node, a graph node corresponding to the changed data is created in the memory, and the node relationship topology graph in the memory is updated according to the graph node corresponding to the changed data; In response to the node update event, the changed node is locked, and according to the node content of the changed node, the changed content of the changed node in the memory is updated and the node relationship topology diagram is updated, and the changed node is unlocked; In response to the node deletion event, the changed node to be deleted is locked, the corresponding changed node in the memory and the corresponding changed node in the node relationship topology diagram are deleted, and the changed node is unlocked.

8. The method according to claim 1, characterized in that The step of querying the target graph node and / or the target node relationship from the latest node relationship topology graph in the memory according to the query condition indicated by the query instruction in response to the query instruction for the graph node and / or the node relationship topology graph comprises: In response to a node relationship query instruction for the graph node and / or the node relationship topology graph, obtaining query starting point data from a query condition indicated by the query instruction; According to the query starting point data, obtaining from the target data source a target identifier of the entity object to be queried in the target data source; Query the target graph node corresponding to the target identifier from the latest graph node in the memory, and query the associated graph node corresponding to the target graph node from the latest node relationship topology graph in the memory; According to the node association condition indicated by the query condition, the association graph nodes are screened to obtain the target node relationship.

9. A data processing device, characterized in that: The device comprises: A construction module, used to obtain the specified business data in the target data source, and construct the graph nodes and node relationship topology graph for the target entity in the specified business data in the memory; wherein the graph nodes are used to indicate the index information of the target entity in the memory; An update module, configured to update the graph nodes and / or the node relationship topology graph in the memory in response to a node change event for the target data source, according to the node content of the changed node and the type of the node change event; or, A query module, configured to respond to a query instruction for the graph node and / or the node relationship topology graph, and query a target graph node and / or a target node relationship from the latest node relationship topology graph in the memory according to a query condition indicated by the query instruction; The acquisition module is used to obtain corresponding target business data and / or target business relationship data from the target data source according to the queried target graph nodes and / or target node relationships.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the data processing method according to any one of claims 1 to 8.

Images