Graph data extraction method and device, storage medium and computer device

Abstract

Embodiments of the present application disclose a kind of graph data extraction method, device, storage medium and computer equipment, the method comprises: based on the elastic distributed dataset in Spark, the first graph data stored in JanusGraph is read, then the deserialization module coupled in the data processing procedure of JanusGraph is called to carry out the first graph data deserialization processing and obtain the second graph data, the second graph data is saved to the elastic distributed dataset in Spark. Using the embodiments of the present application, the graph data stored in JanusGraph underlying can be directly extracted in the graph data extraction based on Spark+JanusGraph architecture, the extraction steps are simplified and some unnecessary processing logic is saved, and the data extraction efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and particularly relates to a graph data extraction method and device, a storage medium and computer equipment. BACKGROUND

[0002] With the continuous development of computer technology and the continuous improvement of informatization level, the amount of data is rapidly increasing, and the data structure is gradually complex. Therefore, various non-relational databases are born.

[0003] The graph database is one of the non-relational databases, and is used for storing various relationship network data. Among many graph databases, JanusGraph is a distributed graph database, which has high scalability. The upper limit of graph storage can be linearly improved by expanding the cluster size, and can support the storage and retrieval of super large graphs.

[0004] If JanusGraph uses Spark for graph algorithm, when graph data extraction is performed, a Gremlin query statement needs to be input through the interface of JanusGraph. TinkerPop parses the Gremlin statement to generate a step set. Different types of steps have different processing logic. Finally, data is obtained by API interaction with HBase. The link is long, and there are many traversal interface calls. In addition, due to the limitation of the design of various middleware APIs, the concurrent effect of data extraction is also poor, which leads to low graph data extraction efficiency. SUMMARY

[0005] The embodiments of the present application provide a graph data extraction method, device, storage medium and computer equipment, which can improve the graph data extraction efficiency by simplifying the data extraction logic.

[0006] The technical solution is as follows:

[0007] In a first aspect, the embodiments of the present application provide a graph data extraction method, which comprises:

[0008] reading first graph data stored in the JanusGraph based on a resilient distributed dataset in Spark;

[0009] calling a deserialization module coupled in a data processing flow of the JanusGraph to perform deserialization processing on the first graph data, to obtain second graph data;

[0010] saving the second graph data to the resilient distributed dataset in the Spark.

[0011] In a second aspect, the embodiments of the present application provide a graph data extraction device, the graph data extraction device comprises:

[0012] a graph data reading module, configured to read first graph data stored in the JanusGraph based on a resilient distributed dataset in the Spark;

[0013] a deserialization module, configured to call a deserialization module coupled in a data processing flow of the JanusGraph to perform deserialization processing on the first graph data to obtain second graph data;

[0014] a graph data saving module, configured to save the second graph data to the resilient distributed dataset in the Spark.

[0015] In a third aspect, the embodiments of the present application provide a storage medium, the storage medium stores a plurality of instructions, the instructions are suitable for being loaded by a processor and performing the method steps described above.

[0016] In a fourth aspect, the embodiments of the present application provide a computer device, which can include a processor and a memory; wherein the memory stores a computer program, the computer program is suitable for being loaded by the processor and performing the method steps described above.

[0017] The technical solutions provided by some embodiments of the present application have at least the following beneficial effects:

[0018] In the embodiments of the present application, the first graph data stored in the JanusGraph is read based on the resilient distributed dataset in the Spark, then the deserialization module coupled in the data processing flow of the JanusGraph is called to perform deserialization processing on the first graph data to obtain the second graph data, and the second graph data is saved to the resilient distributed dataset in the Spark. By using the embodiments of the present application, the graph data stored in the JanusGraph can be directly extracted in the graph data extraction based on the Spark+JanusGraph architecture, the extraction steps are simplified and some unnecessary processing logic is saved, and the data extraction efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0020] Figure 1An example schematic diagram of a graph database architecture of Spark+JanusGraph is provided for the embodiments of the present application.

[0021] Figure 2 An example schematic diagram of a graph data extraction method is provided for the embodiments of the present application.

[0022] Figure 3 An example schematic diagram of graph data extraction is provided for the embodiments of the present application.

[0023] Figure 4 An example schematic diagram of a graph data extraction method is provided for the embodiments of the present application.

[0024] Figure 5 An example schematic diagram of a graph data extraction apparatus is provided for the embodiments of the present application.

[0025] Figure 6 An example schematic diagram of a graph data reading module 11 is provided for the embodiments of the present application.

[0026] Figure 7 An example schematic diagram of a computer device is provided for the embodiments of the present application.

[0027] Figure 8 An example schematic diagram of an operating system and user space is provided for the embodiments of the present application.

[0028] Figure 9 is Figure 7 An architecture diagram of an Android operating system in the embodiments of the present application.

[0029] Figure 10 is Figure 7 An architecture diagram of an IOS operating system in the embodiments of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0031] In the description of the present application, it is understood that the terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance. In the description of the present application, it is necessary to explain that, unless otherwise explicitly specified and limited, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, the process, method, system, product or equipment including a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment. For those skilled in the art, the specific meaning of the above terms in the present application can be understood in specific cases. In addition, in the description of the present application, unless otherwise specified, "multiple" means two or more. The association relationship of the associated objects is described, which means that there can be three relationships, for example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are a "or" relationship.

[0032] In order to more clearly describe the technical scheme of the embodiments of the present application, before the description, some concepts in the present application are described in detail in order to better understand the present scheme. With the rapid development of social, e-commerce, finance, retail, Internet of Things and other industries, a huge and complex relationship network has been woven in the real society, and traditional databases are difficult to handle relationship operations. The relationship between the data that the big data industry needs to process increases by geometric progression with the amount of data, and a database supporting mass complex data relationship operation is urgently needed, and the graph database emerges as the times require. The graph database is not a database for storing pictures, but a data structure for storage and data query. The graph database is an online database management system, which has the creation, reading, updating and deletion (CRUD) operation of processing graph data model. Unlike other databases, relationships are given priority in graph databases, and compared with other relational databases or NoSQL databases, the data model of graph database is also simpler and more expressive.

[0033] According to the storage and graph processing engine, there are some differences in the market graph data. Some graph databases use native graph storage, which is optimized and designed specifically for storing and managing graphs, not all graph databases use native graph storage, and some graph databases serialize graph data and save it to a relational database or object-oriented database, or other general-purpose data storage. For example, Neo4J is a native graph database that uses a backend storage specifically designed and optimized for Neo4J graph databases, which can theoretically better leverage the performance of graph databases. JanusGraph is not a native graph database, but stores data in other systems such as Hbase. Native graph processing is the most efficient way to process graph data, as connected nodes are physically pointed to each other in the database.

[0034] See Figure 1 An example diagram of a Spark+JanusGraph graph database architecture is provided for embodiments of the present application.

[0035] As Figure 1 shown, Spark GraphX is a distributed graph processing framework that provides a simple and easy-to-use interface for graph computing and graph mining based on the Spark platform. JanusGraph is a distributed graph database, and Hbase is the underlying storage for JanusGraph. Currently, graph data extraction based on Spark GraphX for JanusGraph underlying storage requires passing in a Gremlin query statement through the JanusGraph interface. The TinkerPop parses the Gremlin statement to generate a step collection, different types of steps have different processing logic, and finally the underlying graph data is obtained through API interaction with HBase. The link is long, and there are many traversal interface calls, which is inefficient. At the same time, due to the design of the middleware API in the graph, the concurrency effect is also poor, such as the API provided by HBase is not friendly to batch data operations, and the batch operation will eventually be converted into multiple single data operations at the bottom, which is not suitable for OLAP scenarios. In addition, each step of operation in the graph will first load all data, and then perform the next logical processing, with loose coupling and large resource consumption.

[0036] The resilient distributed dataset (RDD) is an abstract concept of distributed memory, and the RDD provides a highly restricted shared memory model, that is, the RDD is a read-only record partition set, and can only be created by performing a deterministic transformation operation (such as map, join and group by) on other RDDs, and the restrictions make the fault tolerance overhead very low. The Gremlin is a graph traversal language under the ThinkerPop framework, and the Gremlin is a functional data flow language, which can enable a user to express complex property graph traversals or queries in a concise manner. The ThinkerPop is an open source graph computing framework, and is a data system for graph data, some of which are in-memory transactional data systems, and some of which are distributed data systems. As long as these graph database systems are "TinkerPop-enabled", the systems can be switched conveniently, and the overall goal of TinkerPop is to enable different data systems for graph data to be integrated conveniently. If a graph database system uses graph data as an underlying data model and supports querying by using the Gremlin graph traversal language, the system is referred to as "TinkerPop-enabled". Figure 1 As shown in the embodiment of the present application,

[0037] In the embodiment of the present application, in order to solve the problem of low efficiency of graph data extraction in the graph database architecture of Spark+JanusGraph, the data extraction channel is restructured, the processing logic in the data extraction process is simplified, and a high-efficiency and low-consumption graph data extraction method is proposed.

[0038] Please refer to Figure 2 , a flowchart of a graph data extraction method is provided in the embodiment of the present application. As shown in Figure 2 , the graph data extraction method can include the following steps S101-S103.

[0039] S101, reading first graph data stored in JanusGraph based on a resilient distributed dataset in Spark;

[0040] Specifically, the resilient distributed dataset is constructed, the first graph data stored in the JanusGraph is read based on the constructed resilient distributed dataset, the JanusGraph can be an HBase database, and the first graph data refers to graph data in byte format stored in the HBase database.

[0041] The first graph data read from the JanusGraph bottom storage based on the self-defined elastic distributed dataset can be a process of finding and reading the first graph data in the disk according to a file name or a file category.

[0042] It is not difficult to understand that the graph data needs to be serialized into a byte structure before being stored in the HBase database of the bottom storage, and serialization is to convert an object into a byte sequence and permanently store it in the disk.

[0043] In S102, a deserialization module coupled in the data processing flow of the JanusGraph is called to perform deserialization processing on the first graph data to obtain second graph data.

[0044] Specifically, the first graph data is stored in the JanusGraph in the form of a byte sequence, and the deserialization module is stripped in the JanusGraph data processing flow through a self-defined function program, and the deserialization module is called to deserialize the read first graph data into the second graph data with a graph structure.

[0045] The deserialization refers to a process of converting a binary string generated in the serialization process into a data structure or an object. In the embodiment of the present application, the deserialization refers to the deserialization of the first graph data with a byte structure into the second graph data with a graph structure.

[0046] In S103, the second graph data is saved to the elastic distributed dataset in the Spark.

[0047] Specifically, the second graph data is the graph data with a graph structure obtained by deserializing the first graph data with a byte structure stored in the JanusGraph bottom storage, and the second graph data with a graph structure is sent to the elastic distributed dataset of the Spark for graph calculation.

[0048] The elastic distributed dataset is the most basic data abstraction in the Spark, which represents an immutable, partitionable, and parallel-computable collection.

[0049] Steps S101-S103 are described in combination with Figure 3 An example diagram of graph data extraction is provided for the embodiment of the present application.

[0050] As Figure 3As shown, Spark GraphX does not need to go through a series of complex data processing procedures of JanusGraph, directly reads the first graph data stored in the bottom storage HBase database in JanusGraph based on the custom-built elastic distributed dataset, and strips the deserialization module coupled in the data processing procedure of JanusGraph, uses the deserialization module to deserialize the first graph data into second graph data, and sends the second graph data into the customized elastic distributed dataset. The customized RDD is the custom-built elastic distributed dataset.

[0051] In the embodiment of the present application, by directly reading the first graph data of the JanusGraph bottom layer based on the elastic distributed dataset in Spark, then calling the deserialization module coupled in the data processing procedure of JanusGraph to deserialize the first graph data into second graph data, and finally sending the second graph data into the elastic distributed dataset for graph calculation, the processing logic in the graph data extraction process is simplified, the data extraction efficiency is improved, and the resource consumption is reduced.

[0052] Please refer to Figure 4 A flowchart of a graph data extraction method is provided for the embodiment of the present application. As shown in Figure 4 The graph data extraction method can include the following steps S201-S205.

[0053] S201, divide the elastic distributed dataset in Spark into at least one partition;

[0054] Specifically, the elastic distributed dataset is divided into a preset number of partitions, and the elastic distributed dataset can be specified for partitioning, wherein one partition corresponds to one processing task, and the number of partitions determines the number of tasks. The more tasks executed in parallel, the higher the parallelism.

[0055] In an implementable manner, the partitions of the elastic distributed dataset can also be obtained by obtaining the file directory corresponding to the first graph data stored in the JanusGraph, and then dividing the elastic distributed dataset into partitions corresponding to each file directory.

[0056] S202, iteratively read the first graph data in the first format in batches based on each partition, parse the first graph data in the first format, and obtain the first graph data in the second format;

[0057] The first graph data in the first format refers to the first graph data file in the storage format adapted to the file system stored in the JanusGraph bottom layer.

[0058] Specifically, the first graph data in the first format is obtained in an iterative manner according to the number of processing tasks corresponding to the number of partitions, and the first graph data in the first format is parsed into the first graph data in the second format which is convenient for processing by the deserialization module. The first graph data in the second format can be in the form of a string.

[0059] The iterative manner refers to reading and parsing according to the storage order of the first graph data in the first format in the directory. For example, one partition can correspond to one file directory of the first graph data in the first format, and the partition is used to process all the first graph data in the first format corresponding to the file directory, and the processing order is according to the sorting of the first graph data in the first format in the directory.

[0060] S203, calling the deserialization module coupled in the data processing process of JanusGraph to perform deserialization processing on the first graph data in the second format to obtain second graph data;

[0061] Specifically, the deserialization module is coupled in the data processing process of JanusGraph, which is separated from the original data processing process and integrated into the newly constructed data processing process, and is used to deserialize the first graph data in the second format into the second graph data.

[0062] S204, based on the correspondence between each partition and the file directory, saving the second graph data to the partition corresponding to the file directory;

[0063] As can be seen from step S201, the elastic distributed dataset can increase the parallelism of the processing task by dividing the partition. The partition can be self-defined division, or can be based on the read file division. If the partition belongs to self-defined division, the processing task is allocated to the partition, that is, the file to be processed can be allocated to the partition in units of file directory, and one partition can correspond to one file directory or multiple file directories.

[0064] Specifically, based on the file directory of the second graph data before deserialization, the second graph data is sent to the corresponding partition according to the correspondence between each partition and the file directory.

[0065] S205, performing graph calculation on the second graph data according to the self-defined graph calculation logic in the elastic distributed dataset;

[0066] Specifically, the user can self-define the graph calculation logic in the elastic distributed dataset according to the need, and perform graph calculation using the second graph data.

[0067] In the embodiment of the present application, by partitioning the resilient distributed datasets in Spark, then iteratively batch reading the first graph data in the first format based on each partition, and parsing the first graph data in the first format to obtain the first graph data in the second format, the parallelism of data extraction can be improved, the data extraction efficiency can be improved, then the deserialization module coupled in the data processing process of JanusGraph is called to deserialize the first graph data in the second format to obtain the second graph data, and then the second graph data is sent into the corresponding partition for graph calculation based on the custom graph calculation logic, the entire data extraction process simplifies the processing logic in the graph data extraction process, improves the data extraction efficiency and reduces resource consumption.

[0068] Please refer to Figure 5 , a structural diagram of a graph data extraction device is provided for the embodiment of the present application. As Figure 5 shown, the graph data extraction device 1 can be realized by software, hardware or a combination of both to become all or part of a terminal device. According to some embodiments, the graph data extraction device 1 includes a graph data reading module 11, a deserialization module 12, a graph data saving module 13, a graph data calculation module 14, and specifically includes:

[0069] The graph data reading module 11 is configured to read the first graph data stored in JanusGraph based on the resilient distributed datasets in Spark.

[0070] The deserialization module 12 is configured to call the deserialization module coupled in the data processing process of JanusGraph to deserialize the first graph data to obtain the second graph data.

[0071] The graph data saving module 13 is configured to save the second graph data to the resilient distributed datasets in Spark.

[0072] The graph data calculation module 14 is configured to perform graph calculation on the third graph data according to the custom graph calculation logic in the resilient distributed datasets.

[0073] Optionally, as Figure 6 shown, a structural diagram of a graph data reading module 11 is provided for the embodiment of the present application.

[0074] The graph data reading module 11 includes:

[0075] The partition division unit 111 is configured to divide the resilient distributed datasets into at least one partition.

[0076] The graph data analysis unit 112 is configured to batch-read first graph data in a first format in an iterative manner based on each of the partitions, analyze the first graph data in the first format, and obtain first graph data in a second format.

[0077] Optionally, the partition division unit 111 is specifically configured to:

[0078] divide the elastic distributed dataset into a preset number of partitions; or

[0079] obtain a file directory corresponding to the first graph data stored in the JanusGraph, and divide the elastic distributed dataset into partitions corresponding to each of the file directories.

[0080] Optionally, the graph data saving module 13 is specifically configured to:

[0081] save the second graph data to a partition corresponding to the file directory based on the correspondence between each partition and the file directory.

[0082] It should be noted that the graph data extraction apparatus provided in the above embodiments is only used as an example for the division of the above functional modules when the graph data extraction method is executed. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the graph data extraction apparatus and the graph data extraction method provided in the above embodiments belong to the same concept, and the implementation process is described in detail in the method embodiments. Therefore, it will not be described here.

[0083] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0084] In the embodiments of the present application, the elastic distributed dataset in Spark is partitioned, then the first graph data in a first format is batch-read in an iterative manner based on each of the partitions, and the first graph data in the first format is analyzed to obtain first graph data in a second format. This can improve the parallelism of data extraction and improve the data extraction efficiency. Then, the first graph data in the second format is deserialized by calling the deserialization module coupled in the data processing process of JanusGraph to obtain second graph data. The second graph data is sent to the corresponding partition for graph calculation based on the custom graph calculation logic. The entire data extraction process simplifies the processing logic in the graph data extraction process, improves the data extraction efficiency, and reduces resource consumption.

[0085] The embodiments of the present application also provide a computer storage medium, which can store a plurality of instructions. The instructions are suitable for being loaded and executed by a processor to perform the aboveFigures 1-4 The graph data extraction method of the embodiment shown, the specific execution process can be seen from Figures 1-4 The specific description of the embodiment shown, here will not be repeated.

[0086] The present application also provides a computer program product, the computer program product stores at least one instruction, the at least one instruction is loaded and executed by the processor as described above Figures 1-4 The graph data extraction method of the embodiment shown, the specific execution process can be seen from Figures 1-4 The specific description of the embodiment shown, here will not be repeated.

[0087] Please refer to Figure 7 It shows the structure block diagram of the computer equipment provided by an exemplary embodiment of the present application. The computer equipment in the present application can include one or more of the following components: processor 110, memory 120, input device 130, output device 140 and bus 150. The processor 110, memory 120, input device 130 and output device 140 can be connected through the bus 150.

[0088] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the entire computer equipment by various interfaces and lines, and performs various functions and processes data of the computer equipment 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be realized in at least one of the hardware forms of digital signal processing (digital signal processing, DSP), field programmable gate array (field-programmable gate array, FPGA) and programmable logic array (programmable logic array, PLA). The processor 110 can integrate a combination of one or more of central processing unit (central processing unit, CPU), graphics processing unit (graphics processing unit, GPU) and modem. Among them, the CPU mainly processes operating system, user interface and application program etc.; the GPU is responsible for rendering and drawing display content; the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but be realized by a separate communication chip.

[0089] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 120 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing each of the methods described below, etc., and the operating system can be an Android system, an IOS system developed by Apple Inc., a system developed based on the Android system or the IOS system, or other systems. The data storage area can also store data created by the computer device during use, such as a phone book, audio and video data, chat record data, etc.

[0090] Referring to Figure 8 As shown, the memory 120 can be divided into an operating system space and a user space, and the operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good running effects, the operating system allocates corresponding system resources to different third-party applications. However, there are also differences in the demand for system resources in different application scenarios in the same third-party application, for example, in the local resource loading scenario, the third-party application has a higher requirement for the disk reading speed, and in the animation rendering scenario, the third-party application has a higher requirement for the GPU performance. However, the operating system and the third-party application are independent of each other, and the operating system often cannot timely perceive the current application scenario of the third-party application, resulting in that the operating system cannot perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0091] In order to enable the operating system to distinguish the specific application scenario of the third-party application, it is necessary to open up the data communication between the third-party application and the operating system, so that the operating system can obtain the current scenario information of the third-party application at any time, and then perform targeted system resource adaptation based on the current scenario.

[0092] Taking the operating system as an Android system for example, the programs and data stored in the memory 120 are as follows Figure 9As shown, the memory 120 can store a Linux kernel layer 320, a system runtime library layer 340, an application framework layer 360, and an application layer 380, wherein the Linux kernel layer 320, the system runtime library layer 340, and the application framework layer 360 belong to an operating system space, and the application layer 380 belongs to a user space. The Linux kernel layer 320 provides underlying drivers for various hardware of a computer device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, power management, and the like. The system runtime library layer 340 provides main feature support for an Android system through some C / C++ libraries. For example, an SQLite library provides database support, an OpenGL / ES library provides 3D drawing support, a Webkit library provides browser kernel support, and the like. An Android runtime is also provided in the system runtime library layer 340, which mainly provides some core libraries to allow developers to use Java language to write Android applications. The application framework layer 360 provides various APIs that can be used when building an application, and developers can also build their own applications by using these APIs, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application is running in the application layer 380, which can be native applications provided by the operating system, such as a contact program, a message program, a clock program, a camera application, and the like, or third-party applications developed by third-party developers, such as game applications, instant messaging programs, photo beautification programs, figure data extraction programs, and the like.

[0093] For example, taking an IOS system as the operating system, the programs and data stored in the memory 120 can include an IOS kernel 320, an IOS runtime library 340, an application framework 360, and an application 380. Figure 10As shown, the IOS system includes: a core operating system layer 420, a core service layer 440, a media layer 460, and a Cocoa Touch layer 480. The core operating system layer 420 includes an operating system kernel, drivers, and low-level hardware abstractions that provide more hardware-specific functionality to program frameworks in the core service layer 440. The core service layer 440 provides system services and / or program frameworks that applications need, such as a Foundation framework, an account framework, an advertisement framework, a data storage framework, a network connection framework, a geographic location framework, a motion framework, and the like. The media layer 460 provides interfaces for applications related to audio and video, such as interfaces related to graphics images, interfaces related to audio technology, interfaces related to video technology, an AirPlay interface for wireless audio and video transmission technology, and the like. The Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development, and is responsible for user touch interaction operations on the computer device. For example, a local notification service, a remote push service, an advertisement framework, a game tool framework, a message user interface (UI) framework, a user interface UIKit framework, a map framework, and the like.

[0094] In Figure 10 In the framework shown, the frameworks related to most applications include, but are not limited to, the Foundation framework in the core service layer 440 and the UIKit framework in the Cocoa Touch layer 480. The Foundation framework provides many basic object classes and data types, and provides the most basic system services for all applications, and is UI-independent. The UIKit framework provides basic UI class libraries for creating touch-based user interfaces, and iOS applications can provide UIs based on the UIKit framework, so it provides the basic framework of the application for building user interfaces, drawing, processing and user interaction events, responding to gestures, and the like.

[0095] In the IOS system, the manner and principle of implementing data communication between a third-party application and an operating system can refer to the Android system, and will not be described herein.

[0096] The input device 130 is configured to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is configured to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In an example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are a touch display screen configured to receive a touch operation of a user using a finger, a touch pen, or any suitable object on or near the touch display screen, and display a user interface of each application. The touch display screen is usually arranged on a front panel of the computer device. The touch display screen can be designed as a full screen, a curved screen, or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, a combination of a special-shaped screen and a curved screen, and the embodiments of the present application do not limit the combination.

[0097] In addition, those skilled in the art can understand that the structure of the computer device shown in the above-described drawings does not constitute a limitation on the computer device, and the computer device can include more or fewer components than those shown in the drawings, or combine certain components, or different component arrangements. For example, the computer device further includes radio frequency circuitry, an input unit, a sensor, audio circuitry, a wireless fidelity (WiFi) module, a power supply, a Bluetooth module, and the like, which are not described herein.

[0098] In the embodiments of the present application, the execution subject of each step can be the computer device introduced above. Alternatively, the execution subject of each step is an operating system of the computer device. The operating system can be an Android system, an IOS system, or other operating systems, and the embodiments of the present application do not limit the operating system.

[0099] The computer device of the embodiments of the present application can also have a display device installed thereon, which can be various devices capable of realizing display functions, for example, a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), a plasma display panel (PDP), and the like. A user can use the display device on the computer device 101 to view displayed text, images, video information, and the like. The computer device can be a smart phone, a tablet computer, a game device, an AR (Augmented Reality) device, a car, a data storage device, an audio playback device, a video playback device, a notebook computer, a desktop computing device, a wearable device such as an electronic watch, electronic glasses, an electronic helmet, an electronic bracelet, an electronic necklace, an electronic clothing, and the like.

[0100] In Figure 7 In the computer device shown, the processor 110 can be configured to invoke the graph data extraction program stored in the memory 120 and specifically perform the following operations:

[0101] read the first graph data stored in the JanusGraph based on the resilient distributed dataset in Spark;

[0102] invoke the deserialization module coupled in the data processing flow of the JanusGraph to perform deserialization processing on the first graph data, to obtain second graph data;

[0103] save the second graph data to the resilient distributed dataset in the Spark.

[0104] In one embodiment, when the processor 110 performs the reading of the first graph data stored in the JanusGraph based on the resilient distributed dataset, the processor 110 specifically performs the following operations:

[0105] divide the resilient distributed dataset into at least one partition;

[0106] read the first graph data in a first format in batches in an iterative manner based on each partition, parse the first graph data in the first format, and obtain first graph data in a second format.

[0107] In one embodiment, when the processor 110 performs the saving of the second graph data to the resilient distributed dataset, the processor 110 specifically performs the following operations:

[0108] performing graph computation on the second graph data according to the graph computation processing logic defined in the elastic distributed dataset.

[0109] In one embodiment, the processor 110, after performing the saving of the second graph data to the elastic distributed dataset, further performs the following operations:

[0110] saving the second graph data to the partition corresponding to the file directory based on the correspondence between each partition and file directory.

[0111] In the embodiments of the present application, by partitioning the elastic distributed dataset in Spark, then iteratively batch reading the first graph data in the first format based on each partition, and parsing the first graph data in the first format to obtain the first graph data in the second format, the parallelism of data extraction can be improved, and the data extraction efficiency can be improved. Then, the deserialization module coupled in the data processing process of JanusGraph is called to perform deserialization processing on the first graph data in the second format to obtain the second graph data, and then the second graph data is sent to the corresponding partition for graph computation based on the defined graph computation processing logic. The entire data extraction process simplifies the processing logic in the graph data extraction process, improves the data extraction efficiency, and reduces resource consumption.

[0112] Those skilled in the art can clearly understand that the technical solutions of the present application can be implemented by means of software and / or hardware. The "unit" and "module" in the specification refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0113] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action order, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0114] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0115] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented in other manners. For example, the division of the apparatus embodiments described above is merely illustrative, and the division of the units can be changed according to actual needs. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0116] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0117] In addition, each functional unit in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0118] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that makes a contribution to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0119] A person of ordinary skill in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0120] The above descriptions are merely some example embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the description and practicing the present disclosure. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not described in the present disclosure. The scope of the present disclosure is defined by the claims, and the spirit and principles of the present disclosure are not limited by the descriptions and examples.

Claims

1. A graph data extraction method, characterized by, The method is applied to an architecture combining a Spark of an open-source computing framework including a resilient distributed dataset and a JanusGraph of a distributed graph database, and the method comprises the following steps: directly reading first graph data stored in the JanusGraph based on the resilient distributed dataset in the Spark; the step of directly reading the first graph data stored in the JanusGraph based on the resilient distributed dataset in the Spark comprises the following steps: dividing the resilient distributed dataset into at least one partition; obtaining a number of tasks based on a number of the at least one partition, wherein the number of the at least one partition is the same as the number of the tasks, and the number of the tasks is positively correlated with parallelism of graph data extraction; batch reading the first graph data in a first format in an iterative manner according to the number of the tasks, and obtaining the first graph data in a second format by analyzing the first graph data in the first format; calling a deserialization module coupled in a JanusGraph data processing procedure of the JanusGraph to perform deserialization processing on the first graph data in the second format, and obtaining second graph data; saving the second graph data to the resilient distributed dataset in the Spark; wherein the step of dividing the resilient distributed dataset into at least one partition comprises the following steps:

2. The method of claim 1, wherein, obtaining file directories corresponding to the first graph data stored in the JanusGraph, and dividing the resilient distributed dataset into partitions corresponding to the file directories one by one. after the step of saving the second graph data to the resilient distributed dataset in the Spark, the method further comprises the following step:

3. The method of claim 1, wherein, performing graph computation on the second graph data according to graph computation processing logic defined in the resilient distributed dataset. the step of saving the second graph data to the resilient distributed dataset in the Spark comprises the following step:

4. A graph data extraction device characterized by comprising: based on a corresponding relationship between each partition and a file directory, saving the second graph data to a partition corresponding to the file directory. The configuration is applied to an architecture combining a Spark of an open-source computing framework including a resilient distributed dataset and a JanusGraph of a distributed graph database, and the configuration comprises the following steps: The graph data reading module is configured to directly read first graph data stored in the JanusGraph based on the resilient distributed dataset in the Spark. The direct reading of the first graph data stored in the JanusGraph based on the resilient distributed dataset in the Spark includes: dividing the resilient distributed dataset into at least one partition; obtaining a number of tasks based on a number of the at least one partition, wherein the number of the at least one partition is the same as the number of the tasks, and the number of the tasks is positively correlated with parallelism of graph data extraction; and batch reading the first graph data in a first format in an iterative manner according to the number of the tasks, and parsing the first graph data in the first format to obtain the first graph data in a second format. The division of the resilient distributed dataset into the at least one partition includes: a partition division unit included in the graph data reading module is configured to obtain file directories corresponding to the first graph data stored in the JanusGraph, and divide the resilient distributed dataset into partitions corresponding to the file directories one by one. The deserialization module is configured to call a deserialization module coupled in a data processing flow of the JanusGraph to perform deserialization processing on the first graph data in the second format to obtain second graph data. The graph data saving module is configured to save the second graph data to the resilient distributed dataset in the Spark.

5. The graph data extraction apparatus according to claim 4, characterized by, Further comprising: The graph data computing module is configured to perform graph computation on the second graph data according to graph computation processing logic defined in the resilient distributed dataset.

6. The graph data extraction apparatus according to claim 4, wherein The graph data reading module includes: The graph data parsing unit is configured to batch read the first graph data in the first format in an iterative manner based on each of the partitions, parse the first graph data in the first format to obtain the first graph data in the second format.

7. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1-3.

8. A computer device, comprising: Including: A processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to implement the steps of the method of any one of claims 1-3.

Images