Graph data processing method and device, equipment and medium

Abstract

The embodiment of the invention discloses a graph data processing method and device, equipment and a medium. The method comprises the following steps: acquiring to-be-processed graph data, and classifying the to-be-processed graph data to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; determining a reference graph data partition of the candidate graph data from the candidate graph data partitions according to a preset candidate data partition algorithm, and storing the candidate graph data in the corresponding reference graph data partition; according to the candidate graph thread corresponding to the reference graph data partition, processing the candidate graph data in the reference graph data partition to obtain reference graph data, and storing the reference graph data in a corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; and updating the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue. According to the scheme, the real-time performance of graph data processing is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of graph database technology, and in particular, to a graph data processing method, device, equipment and medium. Background Art

[0002] In the current environment, graph data plays an important role in the field of big data. With the vigorous development of the Internet and the widespread use of various applications, the scale of graph data has grown exponentially. Today's graph data can contain billions of nodes and tens of billions of edges, and the scale of these data is still increasing over time.

[0003] In many application scenarios, graph data needs to be processed in real time. However, traditional graph data processing methods may not meet the real-time requirements, resulting in high latency. Therefore, it is crucial to improve the real-time performance of graph data processing. Summary of the invention

[0004] The present invention provides a graph data processing method, device, equipment and medium to improve the real-time performance of graph data processing.

[0005] According to one aspect of the present invention, a graph data processing method is provided, comprising:

[0006] Acquire graph data to be processed, and classify the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data;

[0007] According to a preset candidate data partitioning algorithm, determining a reference graph data partition of the candidate graph data from the candidate graph data partitions, and storing the candidate graph data in the corresponding reference graph data partitions;

[0008] According to the candidate graph thread corresponding to the reference graph data partition, the candidate graph data in the reference graph data partition is processed to obtain reference graph data, and the reference graph data is stored in a corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread;

[0009] According to the reference graph data in the reference time queue, the corresponding original graph data in the preset graph database is updated.

[0010] According to another aspect of the present invention, there is provided a graph data processing apparatus, comprising:

[0011] A candidate graph data acquisition module is used to acquire graph data to be processed, and classify the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data;

[0012] A partition storage data module, used to determine the reference graph data partition of the candidate graph data from the candidate graph data partition according to a preset candidate data partition algorithm, and store the candidate graph data in the corresponding reference graph data partition;

[0013] A queue storage data module is used to process the candidate graph data in the reference graph data partition according to the candidate graph thread corresponding to the reference graph data partition to obtain reference graph data, and store the reference graph data in a corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread;

[0014] The data updating module is used to update the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue.

[0015] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0016] one or more processors;

[0017] A memory for storing one or more programs;

[0018] When one or more programs are executed by one or more processors, the one or more processors can execute any one of the graph data processing methods provided by the embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement any one of the graph data processing methods provided by the embodiments of the present invention when executed.

[0020] An embodiment of the present invention provides a graph data processing scheme, which obtains graph data to be processed and classifies the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; according to a preset candidate data partitioning algorithm, the reference graph data partition of the candidate graph data is determined from the candidate graph data partition, and the candidate graph data is stored in the corresponding reference graph data partition; according to the candidate graph thread corresponding to the reference graph data partition, the candidate graph data in the reference graph data partition is processed to obtain reference graph data, and the reference graph data is stored in the corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; according to the reference graph data in the reference time queue, the corresponding original graph data in the preset graph database is updated. The above scheme, by introducing candidate graph threads, and one candidate data partition corresponds to one candidate graph thread, realizes processing of graph data in the corresponding data partition by dedicated threads, improves the consistency of processing data in the data partition, improves the efficiency and accuracy of graph data processing, and thus improves the real-time performance of graph data processing.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flowchart of a graph data processing method provided by Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a graph data processing method provided by Embodiment 2 of the present invention;

[0025] Figure 3 It is a schematic diagram of a correspondence between a candidate graph data partition and a candidate graph thread provided by the third embodiment of the present invention;

[0026] Figure 4 It is a structural schematic diagram of a graph data processing device provided by Embodiment 4 of the present invention;

[0027] Figure 5 It is a structural schematic diagram of an electronic device for implementing a graph data processing method provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0029] In modern applications, graph data often needs to be accessed, updated, and written concurrently by multiple users and tasks. However, concurrent updates and writes may lead to data consistency issues and conflicts, such as data loss, deadlocks, performance degradation, etc. Traditional database management systems may have some bottlenecks and defects in handling the concurrency of graph data.

[0030] In the existing technology, there are many ways to handle this demand, such as transaction management: Graph data transaction processing is very important for data consistency and reliability. Therefore, most graph databases have transaction-related processing. However, for large-scale graph data transaction processing, the graph database system may not be able to provide efficient transaction processing capabilities, resulting in performance degradation and consistency issues. However, for the processing of large concurrent operation updates and writes, various graph databases have different emphases and cannot balance performance and consistency issues well.

[0031] Such as stream data processing: use stream data processing tools such as flink (open source stream processing framework) and Spark (open source distributed query and processing engine) to process data with low latency ingestion and write it to the graph database. This provides low latency and high throughput, but there are also challenges. There are two problems. Let's take flink as an example: a: flink is a powerful and complex distributed computing framework. Setting up and maintaining a flink cluster requires expertise. This may increase the complexity of deployment and maintenance, and flink clusters usually require a lot of computing and memory resources, so it is necessary to invest in hardware and cloud computing resources, which may pose a challenge to costs. b: Although data consistency issues and conflicts can be handled through time windows for concurrent operations of multiple users and tasks, the real-time performance of data queries will be reduced accordingly.

[0032] Embodiment 1

[0033] Figure 1 It is a flowchart of a graph data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of processing graph data. The method can be executed by a graph data processing device. The device can be implemented in software and / or hardware and can be configured in an electronic device that carries the graph data processing function.

[0034] See also Figure 1 The graph data processing method shown includes:

[0035] S110, obtaining graph data to be processed, and classifying the graph data to be processed to obtain candidate graph data.

[0036] The graph data to be processed refers to the graph data to be processed. The candidate graph data refers to the data obtained after classifying the graph data to be processed. Exemplarily, the candidate graph data is candidate point data or candidate edge data. The candidate point data refers to the node data in the graph data to be processed. The candidate edge data refers to the edge data in the graph data to be processed. The graph data includes point data (i.e., node data) and edge data.

[0037] Exemplarily, according to the data structure of the graph data to be processed, the graph data to be processed is classified to obtain candidate point data and candidate edge data. The data structure of the candidate point data may be data with only point identifiers. The point identifier may be a point ID (Identity), that is, data used to characterize the identity of the point data. The data structure of the candidate edge data may be data with a starting point identifier, a target point identifier, and a relationship type between two points. The starting point identifier may be a starting point ID, that is, data used to characterize the identity of the starting point of the edge data. The target point identifier may be a target point ID, that is, data used to characterize the identity of the end point of the edge data. The relationship type between two points refers to the category of the relationship between the starting point identifier and the target point identifier.

[0038] S120. Determine a reference image data partition of the candidate image data from the candidate image data partition according to a preset candidate data partition algorithm, and store the candidate image data in the corresponding reference image data partition.

[0039] Among them, the candidate data partitioning algorithm refers to a pre-set algorithm for determining the partition where the candidate graph data is located. The embodiment of the present invention does not impose any limitation on the setting of the candidate data partitioning algorithm, which can be set by a technician based on experience. Exemplarily, the candidate data partitioning algorithm may include a point data partitioning algorithm and an edge data partitioning algorithm. The point data partitioning algorithm refers to a pre-set algorithm for determining the partition where the point data is located. The edge data partitioning algorithm refers to a pre-set algorithm for determining the partition where the edge data is located.

[0040] Among them, the candidate graph data partition refers to a pre-set graph data partition. Exemplarily, the candidate graph data partition can be a candidate point data partition into a candidate edge data partition. The candidate point data partition refers to a pre-set partition for storing point data. The candidate edge data partition refers to a pre-set partition for storing edge data. It should be noted that the number of candidate point data partitions and candidate edge data partitions can be multiple.

[0041] The reference graph data partition refers to a candidate graph data partition for storing candidate graph data. Exemplarily, the reference graph data partition may be a reference point data partition or a reference edge data partition. The reference point data partition refers to a candidate point data partition for storing candidate point data. The reference edge data partition refers to a candidate edge data partition for storing candidate edge data.

[0042] In an optional embodiment, according to a preset candidate data partitioning algorithm, a reference graph data partition of the candidate graph data is determined from the candidate graph data partitions, including: according to the data category of the candidate graph data, selecting a corresponding reference data partitioning algorithm from the preset candidate data partitioning algorithm; according to the reference data partitioning algorithm, determining the reference graph data partition of the candidate graph data from the candidate graph data partitions.

[0043] The data category refers to the data type of the candidate graph data. Exemplarily, the data category may be a point data category or an edge data category. The reference data partitioning algorithm refers to an algorithm for determining the partition where the candidate graph data is located. Exemplarily, the reference data partitioning algorithm may be a point data partitioning algorithm or an edge data partitioning algorithm.

[0044] Exemplarily, if the data category of the candidate graph data is the point data category, that is, the candidate graph data is candidate point data, then the reference data partitioning algorithm is the point data partitioning algorithm; according to the point data partitioning algorithm, the reference point data partition of the candidate graph data is determined from the candidate point data partition.

[0045] Exemplarily, if the data category of the candidate graph data is the edge data category, that is, the candidate graph data is candidate edge data, then the reference data partitioning algorithm is the edge data partitioning algorithm; according to the edge data partitioning algorithm, the reference edge data partition of the candidate graph data is determined from the candidate edge data partitions.

[0046] It can be understood that determining the reference data partitioning algorithm according to the data category improves the accuracy of the determined reference data partitioning algorithm.

[0047] Exemplarily, if the candidate graph data is candidate point data, the candidate graph data is stored in a corresponding reference point data partition; if the candidate graph data is candidate edge data, the candidate edge data is stored in a corresponding reference edge data partition.

[0048] S130. According to the candidate image thread corresponding to the reference image data partition, the candidate image data in the reference image data partition is processed to obtain reference image data, and the reference image data is stored in a corresponding reference time queue.

[0049] Among them, the candidate graph thread refers to the thread corresponding to the preset candidate graph data partition. Exemplarily, one candidate graph data partition corresponds to one candidate graph thread. The candidate graph thread can be a candidate point thread or a candidate edge thread. The candidate point thread refers to the thread corresponding to the candidate point data partition. The candidate edge thread refers to the thread corresponding to the candidate edge data partition.

[0050] The reference graph data refers to the data obtained after the candidate graph thread corresponding to the reference graph data partition processes the candidate graph data. Exemplarily, the reference graph data may be reference point data or reference edge data. The reference point data refers to the data obtained after the candidate point thread corresponding to the reference point data partition processes the candidate point data in the reference point data partition. The reference edge data refers to the data obtained after the candidate edge thread corresponding to the reference edge data partition processes the candidate edge data in the reference edge data partition.

[0051] The reference time queue refers to a queue pre-set for storing reference graph data. Exemplarily, the reference time queue may be a reference point time queue or a reference edge time queue. The reference point time queue refers to a queue for storing reference point data. The reference edge time queue refers to a queue for storing reference edge data.

[0052] Specifically, the candidate graph thread corresponding to the reference graph data partition is used as the reference graph thread; the candidate graph data in the reference graph data partition is processed according to the reference graph thread to obtain the reference graph data; and the reference graph data is stored in the corresponding reference time queue.

[0053] The reference graph thread refers to the candidate graph thread corresponding to the reference graph data partition. Exemplarily, the reference graph thread may be a reference point thread or a reference graph thread. The reference point thread refers to the candidate point thread corresponding to the reference point data partition. The reference edge thread refers to the candidate edge thread corresponding to the reference edge data partition.

[0054] Exemplarily, if the candidate graph data is candidate point data, the candidate point thread corresponding to the reference point data partition is used as the reference point thread; the candidate point data in the reference point data partition is processed according to the reference point thread to obtain the reference point data; and the reference point data is stored in the corresponding reference point time queue.

[0055] Exemplarily, if the candidate graph data is candidate edge data, the candidate edge thread corresponding to the reference edge data partition is used as the reference edge thread; the candidate edge data in the reference edge data partition is processed according to the reference edge thread to obtain the reference edge data; and the reference edge data is stored in the corresponding reference edge time queue.

[0056] S140. Update the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue.

[0057] The preset graph database refers to a pre-set database for storing graph data. The embodiment of the present invention does not impose any limitation on the setting of the preset graph database, which can be set by technicians based on experience.

[0058] The original graph data refers to the graph data originally stored in the preset graph database. Exemplarily, the original graph data may be original point data or original edge data. The original point data refers to the point data originally stored in the preset graph database. The original edge data refers to the edge data originally stored in the preset graph database.

[0059] Exemplarily, if the candidate graph data is candidate point data, the corresponding original point data in the preset graph database is updated according to the reference point data in the reference point time queue.

[0060] Exemplarily, if the candidate graph data is candidate edge data, the corresponding original edge data in the preset graph database is updated according to the reference edge data in the reference edge time queue.

[0061] An embodiment of the present invention provides a graph data processing scheme, which obtains graph data to be processed and classifies the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; according to a preset candidate data partitioning algorithm, the reference graph data partition of the candidate graph data is determined from the candidate graph data partition, and the candidate graph data is stored in the corresponding reference graph data partition; according to the candidate graph thread corresponding to the reference graph data partition, the candidate graph data in the reference graph data partition is processed to obtain reference graph data, and the reference graph data is stored in the corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; according to the reference graph data in the reference time queue, the corresponding original graph data in the preset graph database is updated. The above scheme, by introducing candidate graph threads, and one candidate data partition corresponds to one candidate graph thread, realizes processing of graph data in the corresponding data partition by dedicated threads, realizes synchronous processing of data in each data partition, improves the efficiency and accuracy of graph data processing, and thus improves the real-time performance of graph data processing.

[0062] Embodiment 2

[0063] Figure 2 This is a flowchart of a graph data processing method provided by the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the operation of "storing the reference graph data in the corresponding reference time queue" into "determining the time-related data of the reference graph data, and determining the reference time queue from the candidate time queues based on the time-related data; based on the preset graph data deduplication algorithm, storing the reference graph data in the reference time queue" to improve the storage mechanism of the reference time queue. It should be noted that for the parts not described in detail in the embodiments of the present invention, please refer to the descriptions of other embodiments.

[0064] See also Figure 2 The graph data processing method shown includes:

[0065] S210: Obtain graph data to be processed, and classify the graph data to be processed to obtain candidate graph data.

[0066] Wherein, the candidate graph data is candidate point data or candidate edge data;

[0067] S220. Determine the reference image data partition of the candidate image data from the candidate image data partition according to a preset candidate data partition algorithm, and store the candidate image data in the corresponding reference image data partition.

[0068] S230. According to the candidate image thread corresponding to the reference image data partition, the candidate image data in the reference image data partition is processed to obtain the reference image data.

[0069] Among them, one candidate graph data partition corresponds to one candidate graph thread.

[0070] S240: Determine time association data of the reference graph data, and determine a reference time queue from the candidate time queues based on the time association data.

[0071] Among them, time-related data refers to time data associated with reference graph data. Candidate time queue refers to a queue pre-set for storing candidate graph data. Exemplarily, the candidate time queue can be a candidate point time queue and a candidate edge time queue. Candidate point time queue refers to a time queue pre-set for storing point data. Candidate edge time queue refers to a time queue pre-set for storing edge data.

[0072] It should be noted that the setting of the candidate time queue can be determined according to the data generation time. Exemplarily, the time window is determined according to the data generation time, and the candidate time queue is determined according to the time window. Among them, the time window can be used to determine the candidate time queue.

[0073] For example, if the data generation time includes 12:00:00, 12:00:02, 12:00:03, and 12:00:03, the time window can be 2s, then [12:00:00-12:00:02) corresponds to candidate time queue A, and [12:00:02-12:00:04) corresponds to candidate time queue B.

[0074] It should be noted that, in the embodiment of the present invention, the settings of the candidate time queues can be independent of each other, that is, one candidate image data partition can correspond to multiple candidate time queues, and the reference image data corresponding to each candidate image data in the candidate image data partition can only be stored in the candidate time queue corresponding to the candidate image data partition; or, the settings of the candidate time queues can be interconnected, that is, each candidate image data partition can correspond to one or more candidate time queues, and the reference image data corresponding to each candidate image data in each candidate image data partition can be stored in the candidate time queue corresponding to each candidate image data partition.

[0075] Exemplarily, if the candidate graph data is candidate point data, then time association data of the reference point data is determined; and based on the time association data, a reference point time queue is determined from the candidate point time queue.

[0076] Exemplarily, if the candidate graph data is candidate edge data, then time association data of the reference edge data is determined; and according to the time association data, a reference edge time queue is determined from the candidate edge time queues.

[0077] In an optional embodiment, if the time-related data includes data generation time and data collection time, then after determining the reference time queue from the candidate time queue, the method also includes: taking the difference between the data generation time and the data collection time as the data delay value; and judging whether the reference image data has the authority to be stored in the reference time queue based on the data delay value and a preset data delay threshold.

[0078] Among them, data generation time refers to the generation time of candidate graph data. Data acquisition time refers to the time when candidate graph data is acquired. The data delay value can be used to characterize the length of time of data transmission delay. The embodiment of the present invention does not impose any limitation on the size of the preset data delay threshold, which can be set by technicians based on experience or needs, or repeatedly determined through a large number of experiments. Exemplarily, the preset data delay threshold can be 1s.

[0079] Exemplarily, if the candidate map data is candidate point data, the reference map data is reference point data, and the data generation time and data collection time of the reference point data are determined; the difference between the data generation time and the data collection time of the reference point data is used as the data delay value of the reference point data; based on the preset data delay threshold and the data delay value of the reference point data, it is determined whether the reference point data has the authority to be stored in the reference point time queue.

[0080] Exemplarily, if the candidate graph data is candidate edge data, the reference graph data is reference edge data, and the data generation time and data collection time of the reference edge data are determined; the difference between the data generation time and the data collection time of the reference edge data is used as the data delay value of the reference edge data; based on the preset data delay threshold and the data delay value of the reference edge data, it is determined whether the reference edge data has the authority to be stored in the reference edge time queue.

[0081] It can be understood that by introducing data delay values ​​and preset data delay thresholds to determine whether the reference image data has the authority to be stored in the reference time queue, the comprehensiveness of the reference image data in the reference time queue is improved on the basis of considering the data transmission delay; at the same time, it avoids storing outdated reference image data in the reference time queue, thereby improving the accuracy and real-time performance of the data stored in the reference time queue.

[0082] In an optional embodiment, whether the reference image data has the permission to be stored in the reference time queue is determined based on the data delay value and the preset data delay threshold, including: if the data delay value is less than or equal to the preset data delay threshold, then it is determined that the reference image data has the permission to be stored in the reference time queue; if the data delay value is greater than the preset data delay threshold, then it is determined that the reference image data does not have the permission to be stored in the reference time queue.

[0083] Exemplarily, if the candidate map data is candidate point data, the reference map data is reference point data, and if the data delay value of the reference point data is less than or equal to the preset data delay threshold, it is determined that the reference point data has the authority to be stored in the reference point time queue; if the data delay value of the reference point data is greater than the preset data delay threshold, it is determined that the reference point data does not have the authority to be stored in the reference point time queue.

[0084] Exemplarily, if the candidate graph data is candidate edge data, the reference graph data is reference edge data, and if the data delay value of the reference edge data is less than or equal to the preset data delay threshold, it is determined that the reference edge data has the authority to be stored in the reference edge time queue; if the data delay value of the reference edge data is greater than the preset data delay threshold, it is determined that the reference edge data does not have the authority to be stored in the reference edge time queue.

[0085] It can be understood that by storing reference image data with data delay values ​​less than or equal to the preset data delay threshold in the reference time queue, and prohibiting the storage of reference image data with data delay values ​​greater than the preset data delay threshold in the reference time queue, the accuracy of storing reference image data in the reference time queue is improved.

[0086] S250: Based on a preset graph data deduplication algorithm, the reference graph data is stored in a reference time queue.

[0087] The graph data deduplication algorithm refers to a pre-set algorithm for deduplicating reference graph data stored in a reference time queue. Exemplarily, the graph data deduplication algorithm may be an edge data deduplication algorithm or a point data deduplication algorithm. The point data deduplication algorithm may be used to deduplicate reference point data stored in a reference point time queue. The edge data deduplication algorithm may be used to deduplicate reference edge data stored in a reference edge time queue.

[0088] Exemplarily, if the candidate graph data is candidate point data, the reference graph data is reference point data, and based on a preset point data deduplication algorithm, the reference point data is stored in a reference point time queue.

[0089] Exemplarily, if the candidate graph data is candidate edge data, the reference graph data is reference edge data, and based on a preset edge data deduplication algorithm, the reference edge data is stored in a reference edge time queue.

[0090] In an optional embodiment, based on a preset graph data deduplication algorithm, the reference graph data is stored in a corresponding reference time queue, including: based on the preset graph data deduplication algorithm, determining whether the reference graph data is updated graph data; if so, updating the graph data to be updated in the reference time queue to the reference graph data; if not, directly storing the reference graph data in the reference time queue.

[0091] The updated graph data may be used to update the corresponding original stored graph data in the reference time queue. Exemplarily, the updated graph data may be updated point data or updated edge data. The updated point data may be used to update the corresponding original stored point data in the reference point time queue. The updated edge data may be used to update the corresponding original stored edge data in the reference edge time queue.

[0092] The graph data to be updated refers to the original stored graph data that needs to be updated in the reference time queue. Exemplarily, the graph data to be updated may be point data to be updated or edge data to be updated. The point data to be updated refers to the original stored point data that needs to be updated in the reference point time queue. The edge data to be updated refers to the original stored edge data that needs to be updated in the reference edge time queue.

[0093] Specifically, based on a preset graph data deduplication algorithm, determine whether the reference graph data is updated graph data; if so, determine the to-be-updated graph data corresponding to the updated graph data in the reference time queue, and update the to-be-updated graph data to the reference graph data; if not, directly store the reference graph data in the reference time queue.

[0094] Exemplarily, if the candidate map data is candidate point data, the reference map data is reference point data, and based on a preset point data deduplication algorithm, it is determined whether the reference point data is update point data; if so, the to-be-updated point data corresponding to the update point data in the reference point time queue is determined, and the to-be-updated point data is updated to the reference point data; if not, the reference point data is directly stored in the reference point time queue.

[0095] Exemplarily, if the candidate graph data is candidate edge data, the reference graph data is reference edge data, and based on a preset edge data deduplication algorithm, it is determined whether the reference edge data is updated edge data; if so, the to-be-updated edge data corresponding to the updated edge data in the reference edge time queue is determined, and the to-be-updated edge data is updated to the reference edge data; if not, the reference edge data is directly stored in the reference edge time queue.

[0096] It can be understood that by introducing the update graph data and determining the storage method of the reference graph data, it is avoided that a lot of irrelevant data is stored in the reference time queue, the performance of the reference time queue is improved, and the waste of resources is reduced.

[0097] S260: Update the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue.

[0098] The embodiment of the present invention provides a graph data processing scheme, which refines the operation of storing reference graph data in a corresponding reference time queue into determining time-related data of the reference graph data, and determines the reference time queue from the candidate time queue based on the time-related data; based on a preset graph data deduplication algorithm, the reference graph data is stored in the reference time queue, thereby improving the storage mechanism of the reference time queue. The above scheme improves the accuracy of the determined reference time queue by introducing time-related data to determine the reference time queue; at the same time, by introducing a graph data deduplication algorithm to store the reference graph data, the occupation of resources in the reference time queue is reduced.

[0099] The processing of candidate point data and candidate edge data in the embodiment of the present invention may be performed simultaneously.

[0100] Embodiment 3

[0101] Based on the above embodiments, the embodiment of the present invention provides a method for processing a large amount of graph data.

[0102] The main purpose of the present invention is to deal with data consistency problems and conflicts that may arise when being accessed, updated and written concurrently by multiple users and tasks, while ensuring the real-time nature of data queries. The main functions of this tool include solving the real-time nature and concurrent processing problems of data queries. The acquired graph data to be processed is classified to obtain candidate graph data; the candidate graph data is pushed to Kafka (open source messaging system), and the tool consumes the candidate graph data in Kafka to process concurrent operations and query data in real time. Data from log files to other databases and data warehouses can be easily handled. One of the main advantages of this tool is high-performance data loading, memory preprocessing of data, caching of real-time queries, and flexible configuration of conditional triggers for persistent data operations, thereby supporting fast data analysis and query. In addition, data format conversion and preprocessing can be performed to ensure that the imported data matches the table structure and data type of the graph database, while using the algorithmic capabilities of the graph database itself to perform better data calculations and other operations. Data consistency is also the focus of its attention. Even if there is an interruption or error during the import process, appropriate processing and data recovery can be performed to ensure data integrity.

[0103] The embodiments of the present invention implement applications based on Kafka data sources and graph database writing and querying, which require consideration of key elements such as multi-threaded processing, data updating, real-time query, data buffering, data verification, writing retries, and exception handling.

[0104] Exemplarily, this tool consists of two parts, one is the client responsible for writing candidate graph data to Kafka, referred to as "Client", and the other is an application used to handle concurrent writing, updating and real-time query of candidate graph data, referred to as "Server". Among them, Client implements the ability to send the specified candidate graph data to the partition of the specified topic (i.e., the reference graph data partition) through the specified algorithm (i.e., the candidate data partitioning algorithm). Among them, Server first uses the Kafka consumer library to create a consumer to subscribe to the specified topic, which can be called "consumer". At the same time, it obtains and listens to the number of partitions of this topic through the Kafka client. Based on the number of partitions and the tool, it automatically allocates the number of threads to be created to consume the specified partition data. Each separate candidate graph thread can be called "Single-partition" for short sp. Because the client Client guarantees that the same point / edge data will definitely be written to the same candidate graph data partition, the candidate graph data in the candidate graph data partition is naturally ordered. This partition-level processing method helps improve concurrency and performance, because each sp thread is only responsible for processing the data of a specific partition, thus avoiding lock competition and resource contention; in addition, when the number of partitions increases, new sp threads can be created dynamically to ensure that data processing will not be hindered by the introduction of new topics. This design method enables the system to adapt to changing data streams and maintain high performance and scalability. At the same time, you can customize the time window and water level (that is, the preset data delay threshold) to ensure concurrent writes and update operations that cause data consistency problems due to network problems. For example: a business platform that uses a graph database to implement business link queries and real-time risk detection. Such as Figure 3 As shown in the figure, point data is published to the topic "topic1" and edge data is published to the topic "topic2". Each partition of these two topics has a corresponding consumer thread sp (i.e., candidate graph thread) to consume and process data. The candidate point thread is stored in consumer1 and the candidate edge thread is stored in consumer2.

[0105] Exemplarily, the working principle of the candidate graph thread may include parsing the candidate graph data, that is, the sp thread (i.e., the candidate graph thread) obtains the candidate graph data from the specified partition (i.e., the reference graph data partition) of kafka. These candidate graph data exist as binary data in protobuf format, and sp will parse these binary data into reference graph data.

[0106] Exemplarily, the working principle of the candidate graph thread may include storing the candidate graph data, storing the reference graph data in the reference time queue corresponding to the time window, and the time window may be determined according to the data generation time or the time when the offset of Kafka is generated (i.e., the data collection time), and the water level may be customized. For example, the time window may be set to 2s and the water level may be set to 1s.

[0107] For example, since the candidate graph data is ordered in Kafka itself, it is divided into multiple windows according to the data generation time. Taking into account the delay of network transmission, the window from 00 seconds to 02 seconds will be written to the disk at 03 seconds and destroyed in the memory. Due to the mechanism of Kafka, a single partition only allows one thread to consume. This can solve the problem of data concurrency well. In the memory, the later value can directly overwrite the previous value according to the data generation time or data collection time.

[0108] For example, in a single time window, the reference graph data is a set of point models with a deduplication of Set <vertexmodel>Or the edge model's deduplication set Set <edgemodel>This storage structure uses a graph data deduplication algorithm to achieve real-time update operations for the same point / edge according to the specified time column.

[0109] For example, if there are too many points to be written, or the data concurrency of the same point / edge is too large. For the former, only the number of Kafka partitions needs to be expanded horizontally to meet the real-time requirements. For the latter, the sp thread and the client can use the same allocation algorithm to expand a single partition into a group of multiple data for consumption and processing. Specifically, when the number of candidate point data is large, the number of candidate point data partitions can be directly increased, and the number of candidate point threads can be increased at the same time; when the concurrency of candidate point data or candidate edge data is too large, multiple candidate graph data in a single candidate graph data partition can be treated as a group, and a group of candidate graph data corresponds to one candidate graph thread, that is, a single candidate graph data partition is divided into multiple groups, each group corresponds to a candidate graph thread, and a single candidate graph data partition corresponds to multiple candidate graph threads.

[0110] The technical solution provided by the embodiment of the present invention can well ensure the integrity and consistency of data during data concurrency, and at the same time has good scalability for increasing data volume.

[0111] Exemplarily, the working principle of the candidate graph thread may include data query. For example, taking the data of the point vid1 that is continuously written, there are the following situations: if there is only the candidate point time queue data1, the reference point data whose data generation time is (00 seconds, 02 seconds] will be stored in data1. If you need to query vid1 at this time, you will not query the file on the disk, and will directly return the latest point data from data1; when the system time exceeds 03 seconds, the state at this time is that there are candidate point time queues data1 and candidate point time queues data2. At this time, the reference point data whose data generation time is (00 seconds, 02 seconds] will still be stored in data1, but the reference point data whose data generation time is (02 seconds, 04 seconds] It will be stored in data2. When querying vid1 at this time, it will be found that data1 and data2 exist in the memory, but the time of data2 is later, so the latest point data will be directly returned from data2; when the system time reaches 03 seconds, the data of data1 will be written to the disk and destroyed in the memory. At this time, if the reference point data with the data generation time of (00 seconds, 02 seconds] is received again, the reference point data will be directly treated as erroneous data, because the data after 02 seconds has been received at this time, so the reference point data in the time range of (00 seconds, 02 seconds] is no longer the latest. At this time, if you need to query vid1, the latest point data will be directly returned from data2.

[0112] The technical solution provided by the embodiment of the present invention can send the same point / edge to the specified partition through the client sending the candidate data partition algorithm, so that the data is naturally ordered. At the same time, the partition and the thread are designed to correspond one to one, so that the data is related and isolated. At the same time, the overwriting logic is customized to avoid data consistency problems caused by concurrent operations. At the same time, the logic of time-based window storage is implemented, and the data is stored on the disk at the end of the window, and the offset is submitted at the same time. The integrity and consistency of the data can be guaranteed. At the same time, the window processing ensures that the data of a single window will not affect the next window, ensuring the accuracy of the query during concurrent operations. At the same time, because there is no need to wait for the disk to be stored, the data in the memory can be directly queried, which also ensures the real-time nature of the data.

[0113] An embodiment of the present invention also provides a graph data processing system, which has the characteristics of unique storage structure, low resource consumption, low deployment cost, easy maintenance, simplicity and high throughput, and real-time query guarantee.

[0114] Embodiment 4

[0115] Figure 4 This is a schematic diagram of the structure of a graph data processing device provided in Embodiment 4 of the present invention. This embodiment is applicable to the case of processing graph data, and the method can be executed by a graph data processing device, which can be implemented in software and / or hardware, and can be configured in an electronic device that carries the graph data processing function.

[0116] like Figure 4 As shown, the device includes: a candidate graph data acquisition module 310, a partition storage data module 320, a queue storage data module 330 and a data update module 340. Among them,

[0117] The candidate graph data acquisition module 310 is used to acquire the graph data to be processed and classify the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data;

[0118] A partition storage data module 320 is used to determine the reference image data partition of the candidate image data from the candidate image data partition according to a preset candidate data partition algorithm, and store the candidate image data in the corresponding reference image data partition;

[0119] The queue storage data module 330 is used to process the candidate graph data in the reference graph data partition according to the candidate graph thread corresponding to the reference graph data partition to obtain reference graph data, and store the reference graph data in the corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread;

[0120] The data updating module 340 is used to update the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue.

[0121] An embodiment of the present invention provides a graph data processing scheme, which obtains graph data to be processed and classifies the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; according to a preset candidate data partitioning algorithm, the reference graph data partition of the candidate graph data is determined from the candidate graph data partition, and the candidate graph data is stored in the corresponding reference graph data partition; according to the candidate graph thread corresponding to the reference graph data partition, the candidate graph data in the reference graph data partition is processed to obtain reference graph data, and the reference graph data is stored in the corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; according to the reference graph data in the reference time queue, the corresponding original graph data in the preset graph database is updated. The above scheme, by introducing candidate graph threads, and one candidate data partition corresponds to one candidate graph thread, realizes processing of graph data in the corresponding data partition by dedicated threads, improves the consistency of processing data in the data partition, improves the efficiency and accuracy of graph data processing, and thus improves the real-time performance of graph data processing.

[0122] Optionally, the queue storage data module 330 includes:

[0123] a reference time queue determination unit, configured to determine time association data of the reference graph data, and determine a reference time queue from candidate time queues according to the time association data;

[0124] The queue storage data unit is used to store the reference graph data in the reference time queue based on a preset graph data deduplication algorithm.

[0125] Optionally, if the time-related data includes data generation time and data collection time, then after the reference time queue determination unit, the device further includes:

[0126] A delay value determining unit, configured to use the difference between the data generation time and the data collection time as a data delay value;

[0127] A storage permission determination unit is used to determine whether the reference image data has permission to be stored in the reference time queue according to the data delay value and a preset data delay threshold.

[0128] Optionally, the storage permission determination unit is specifically used to:

[0129] If the data delay value is less than or equal to the preset data delay threshold, determining that the reference image data has the authority to be stored in the reference time queue;

[0130] If the data delay value is greater than the preset data delay threshold, it is determined that the reference image data does not have the authority to be stored in the reference time queue.

[0131] Optionally, the queue stores data units, specifically for:

[0132] Based on a preset graph data deduplication algorithm, determining whether the reference graph data is updated graph data;

[0133] If yes, updating the to-be-updated graph data in the reference time queue to the reference graph data;

[0134] If not, the reference image data is directly stored in the reference time queue.

[0135] Optionally, the partition storage data module 320 includes:

[0136] A reference partition algorithm determination unit, configured to select a corresponding reference data partition algorithm from preset candidate data partition algorithms according to the data category of the candidate graph data;

[0137] A reference partition determination unit is used to determine a reference graph data partition of the candidate graph data from the candidate graph data partitions according to the reference data partition algorithm.

[0138] The graph data processing device provided in the embodiment of the present invention can execute the graph data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each graph data processing method.

[0139] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of the graph data to be processed and the candidate data partitioning algorithm involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] Embodiment 5

[0141] Figure 5 It is a structural schematic diagram of an electronic device for implementing a graph data processing method provided by Embodiment 5 of the present invention. The electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Figure 5 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0143] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a graph data processing method.

[0145] In some embodiments, the graph data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the graph data processing method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the graph data processing method in any other appropriate manner (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / edgemodel> < / vertexmodel>

Claims

1. A graph data processing method, characterized in that: include: Acquire graph data to be processed, and classify the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; According to a preset candidate data partitioning algorithm, determining a reference graph data partition of the candidate graph data from the candidate graph data partitions, and storing the candidate graph data in the corresponding reference graph data partitions; According to the candidate graph thread corresponding to the reference graph data partition, the candidate graph data in the reference graph data partition is processed to obtain reference graph data, and the reference graph data is stored in a corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; According to the reference graph data in the reference time queue, the corresponding original graph data in the preset graph database is updated.

2. The method according to claim 1, characterized in that The storing the reference image data in a corresponding reference time queue comprises: Determining time association data of the reference graph data, and determining a reference time queue from candidate time queues based on the time association data; Based on a preset graph data deduplication algorithm, the reference graph data is stored in the reference time queue.

3. The method according to claim 2, characterized in that If the time-related data includes data generation time and data collection time, after determining the reference time queue from the candidate time queues, the method further includes: Taking the difference between the data generation time and the data collection time as the data delay value; According to the data delay value and a preset data delay threshold, it is determined whether the reference image data has the authority to be stored in the reference time queue.

4. The method according to claim 3, characterized in that The determining, according to the data delay value and the preset data delay threshold, whether the reference image data has the authority to be stored in the reference time queue comprises: If the data delay value is less than or equal to the preset data delay threshold, determining that the reference image data has the authority to be stored in the reference time queue; If the data delay value is greater than the preset data delay threshold, it is determined that the reference image data does not have the authority to be stored in the reference time queue.

5. The method according to claim 2, characterized in that: Based on a preset graph data deduplication algorithm, storing the reference graph data in a corresponding reference time queue includes: Based on a preset graph data deduplication algorithm, determining whether the reference graph data is updated graph data; If yes, updating the to-be-updated graph data in the reference time queue to the reference graph data; If not, the reference image data is directly stored in the reference time queue.

6. The method according to claim 1, characterized in that The step of determining the reference graph data partition of the candidate graph data from the candidate graph data partitions according to a preset candidate data partition algorithm comprises: According to the data category of the candidate graph data, a corresponding reference data partitioning algorithm is selected from preset candidate data partitioning algorithms; According to the reference data partitioning algorithm, a reference graph data partition of the candidate graph data is determined from the candidate graph data partitions.

7. A graph data processing device, characterized in that: include: A candidate graph data acquisition module is used to acquire graph data to be processed, and classify the graph data to be processed to obtain candidate graph data; wherein the candidate graph data is candidate point data or candidate edge data; A partition storage data module, used to determine the reference graph data partition of the candidate graph data from the candidate graph data partition according to a preset candidate data partition algorithm, and store the candidate graph data in the corresponding reference graph data partition; A queue storage data module, used for processing the candidate graph data in the reference graph data partition according to the candidate graph thread corresponding to the reference graph data partition, obtaining reference graph data, and storing the reference graph data in a corresponding reference time queue; wherein one candidate graph data partition corresponds to one candidate graph thread; The data updating module is used to update the corresponding original graph data in the preset graph database according to the reference graph data in the reference time queue.

8. The device according to claim 7, characterized in that The queue storage data module includes: a reference time queue determination unit, configured to determine time association data of the reference graph data, and determine a reference time queue from candidate time queues according to the time association data; The queue storage data unit is used to store the reference graph data in the reference time queue based on a preset graph data deduplication algorithm.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a graph data processing method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a graph data processing method as described in any one of claims 1 to 6 is implemented.

Images