An implementation method and system for online data loading based on a graph database
By using memory mapping technology and concurrent processing of start and end point information, the problem of long loading time for massive amounts of data in graph databases is solved, enabling the creation of point and edge information quickly and in one read, thus optimizing data loading efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU DAMENG DATABASE CO LTD
- Filing Date
- 2023-06-15
- Publication Date
- 2026-05-29
AI Technical Summary
Existing graph databases require reading data text twice when loading massive amounts of data, which increases time consumption, especially when processing complex relational data, resulting in low efficiency. In addition, although the existing preprocessing module reduces the repetitive reading of point information, it increases the complexity of the system.
The data is divided into multiple mapping blocks using memory mapping technology. Each block is given a unique ID. Start and end point information is parsed and created concurrently according to the ID order, ensuring that start and end points are created before edge information. Parallel processing of point and edge information is achieved through a single read.
It enables rapid data loading in graph database systems, shortens parsing time, reduces the time consumed by repeated readings, enables concurrent processing of complex relational data, and optimizes data loading efficiency.
Smart Images
Figure CN116881509B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of graph database technology, and specifically relates to an implementation method and system for online data loading based on graph database. Background Technology
[0002] In graph database systems or other data systems that maintain vertex-edge relationships, a small amount of vertex information can construct complex relational networks. For example, in a community of 1,000 people, each person knows several other people in the community. Building a network of interpersonal relationships within the community, starting with each person and ending with those acquaintances, will result in an exponentially expanding amount of acquaintance data. Rapidly loading massive amounts of online data is a primary challenge for graph databases and other vertex-edge relationship systems, especially when dealing with data containing complex relationships.
[0003] In loaded data with massive relationships, the number of edges often far exceeds the number of nodes, much like how everyone has more than one friend. Typically, the content of loaded data text can be simply divided into: {starting point information (column 1-m) | ending point information (column (m+1)-k) | edge relationship information (column (k+1)-n)}. Because the starting and ending point information often has significant overlap, the size of the data text mainly depends on the number of edges. Edge relationship information describes a certain relationship between the starting and ending points; therefore, before creating an edge, it is necessary to ensure that the starting and ending points already exist.
[0004] The common processing method requires reading the data text twice. The first time, the data text is read to load the start and end information. The second time, the data text is read to load the edge relationship information. This loading method has clear logic and is simple to implement. However, reading the same data text twice will increase the time consumption. In particular, when loading the point information in the first time, the repetition of points also leads to a lot of invalid data reading.
[0005] Some database systems add a data preprocessing module. This module divides the data into point information text and edge information text, importing the two texts sequentially. The advantage of data preprocessing is that the point data in the point information text is no longer repetitive; the point information extracted from the loaded data text forms a smaller text data, thus reducing the data reading time during loading. However, introducing a preprocessing module also increases system complexity. From the overall process perspective, data preprocessing also takes time, so the data reading time during loading is not reduced.
[0006] Ideally, the data text should be read only once, so that only one data parsing is needed to fully create the edge relationship information. At the same time, it is necessary to ensure that the start and end information of the edge is created before the edge is created. Summary of the Invention
[0007] Purpose of the invention: To address the problem of long loading times for existing online data, this invention proposes a method and system for loading online data based on graph databases.
[0008] Technical solution: A method for implementing online data loading based on a graph database, comprising the following steps:
[0009] Step 1: Divide the text data to be loaded into multiple mapping blocks in a continuous manner, and map each mapping block to the computer system; each mapping block obtains a unique ID representing itself in an incremental manner;
[0010] Step 2: Parse the mapping blocks one by one in ascending order of ID to obtain the data blocks that correspond one-to-one with the mapping blocks, and record the largest ID in the mapping block being parsed at the moment; the data block consists of start and end point information and edge information;
[0011] Step 3: Process the data blocks one by one according to the parsing completion order. The processing operations include: firstly creating the start and end points of the data blocks in the database system; after all the start and end points of the data blocks with IDs less than or equal to the maximum ID have been created in the database system, create the edge information of the data blocks in the database system.
[0012] Furthermore, in step 2, the step of parsing the mapping blocks one by one in ascending ID order to obtain the data blocks corresponding to the mapping blocks specifically includes:
[0013] By offsetting the mapped address to the beginning and end of the block, it is determined whether there is cross-block data; if there is cross-block data at the beginning of the block, the cross-block data at the beginning of the block is skipped; if there is cross-block data at the end of the block, part of the content of the beginning of the next mapped block is read.
[0014] According to the configuration file, a row of data is parsed into: start point information, end point information, and edge information; the start point information, end point information, and edge information form a data block that corresponds one-to-one with the mapping block.
[0015] Furthermore, if the start / end point information has already been successfully created or parsed, then the start / end point information will not be added to the data block.
[0016] Furthermore, the database system is a graph database system or a data system that maintains vertex-edge relationships.
[0017] Furthermore, in step 3, the start and end points are created concurrently.
[0018] This invention discloses an implementation system for online data loading based on a graph database, comprising:
[0019] The mapping module is used to divide the text data to be loaded into multiple mapping blocks in a continuous manner and map each mapping block to the computer system; each mapping block obtains a unique ID representing itself in an incrementing manner;
[0020] The parsing module is used to parse the mapping blocks one by one in ascending order of ID to obtain the data blocks that correspond one-to-one with the mapping blocks, and to record the largest ID in the mapping block currently being parsed; the data block consists of start and end point information and edge information;
[0021] The import module is used to process data blocks one by one according to the parsing completion order. The processing operations include: firstly creating the start and end points of the data blocks in the database system; after all the start and end points of the data blocks with IDs less than or equal to the maximum ID have been created in the database system, creating the edge information of the data blocks in the database system.
[0022] Furthermore, it also includes:
[0023] The point import module is used to handle the concurrent creation of start and end points in the database system.
[0024] Furthermore, in the parsing module, the presence of cross-block data is determined by offsetting the mapped address to the block head and block tail. If cross-block data exists at the block head, it is skipped. If cross-block data exists at the block tail, a portion of the contents of the next mapped block's block head is read. According to the configuration file, a line of data is simultaneously parsed into: start point information, end point information, and edge information. The start / end point information and edge information form a data block that corresponds one-to-one with the mapped block. If the start / end point information has already been successfully created or parsed, it is not added to the data block.
[0025] Furthermore, the database system is a graph database system or a data system that maintains vertex-edge relationships.
[0026] Furthermore, assuming that when the import module has completed creating the start and end points in data block i, and there are data blocks with IDs less than or equal to the maximum ID whose start and end points have not all been created, the import module can wait for it to finish processing or process the next data block in the order of parsing completion. When the start and end points in all data blocks with IDs less than or equal to the maximum ID have all been created in the database system, the import module imports and creates the edge information in data block i.
[0027] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0028] (1) The method / system of the present invention does not require additional preprocessing of the large text data loaded, and eliminates the time consumption caused by repeated reading. That is, it reads the text data once and parses the point information data and edge information data at the same time, realizing the rapid loading of online data in graph database systems and various point-edge data processing systems.
[0029] (2) The method / system of the present invention utilizes the memory mapping principle to concurrently parse data files, and processes individual texts in batches in the form of parsing blocks, which greatly shortens the data parsing time and enables fast online data loading of graph database systems and various point-edge data processing systems.
[0030] (3) The method / system of the present invention is good at processing complex relational data by creating point information and edge information in the database system at the same time; and by using the data block control algorithm to ensure the priority creation of the start and end points on the edge, it effectively shortens the loading time of point and edge relational data, and can quickly provide services to the outside world with the loaded graph data.
[0031] (4) The method / system of the present invention adds a point-edge loading order control algorithm and imports parsed data blocks concurrently, which effectively alleviates the dependency effect between data blocks. Attached Figure Description
[0032] Figure 1 System diagram for data loading implementation;
[0033] Figure 2 Partition the graph to represent the mapping blocks of the loaded text;
[0034] Figure 3 This is the state diagram of the parsing module at a certain moment;
[0035] Figure 4 This is a state diagram of the imported module at a certain moment. Detailed Implementation
[0036] Example 1:
[0037] This embodiment discloses a method for online data loading based on a graph database, enabling the reading and batch parsing of loaded data text, and the creation of vertex and edge data. The implementation method of this embodiment will now be described in detail with reference to the accompanying drawings, as follows: Figure 1 As shown, the main steps include the following:
[0038] Step 1: Select an appropriate memory mapping block size. Taking 64MB per block as an example, the mapping module divides the large text data to be loaded into multiple mapping blocks. The computer system directly establishes the relationship between virtual memory and each mapping block, so that multiple mapping blocks are mapped to the computer system. Each mapping block obtains a unique identifier ID in an incrementing manner. Figure 2As shown, the file is divided into n mapped blocks in a contiguous manner. The advantage of this step is that database systems typically load large amounts of data, and edge data with complex relationships often corresponds to one or more large text data sets. When using memory mapping, the computer system partitions virtual space and establishes a mapping relationship with the text data. Text data is read using memory mapping, thus eliminating the need for file I / O operations. Furthermore, typically only one mapping module is needed to quickly establish the mapping relationship between virtual memory and the entire file, and subsequent data processing will be performed in blocks.
[0039] Step 2: The parsing module parses the mapping blocks one by one in ascending order of ID. The parsing process includes: first, determining whether there is cross-block data by offsetting the mapping address to the beginning and end of the block (assuming one line of data corresponds to one relationship information). If there is cross-block data at the beginning of the block, it is skipped; if there is cross-block data at the end of the block, the beginning of the next mapping block needs to be read. Then, according to the configuration file, the information in the mapping block is processed into a data block composed of start and end point information and edge information. That is, one line of data is parsed into: start information, end information, and edge information simultaneously. Here, the data block and the mapping block are in one-to-one correspondence. If the start and end point information has already been successfully created or read by other parsing modules, it does not need to be added to the data block. Finally, the maximum ID (id-max) of the mapping block currently being parsed by all parsing modules is recorded. Figure 3 The diagram shows the state of the parsing module at a certain moment. Parsing module-1 has completed parsing mapping block-1 and is currently parsing mapping block-4. If mapping block-4 is parsed before mapping block-5 and mapping block-3, the data block-4 parsed from mapping block-4 will record the maximum ID number currently being parsed (id-max=5, mapping block-5).
[0040] Step 3: The import module processes data blocks one by one in the order they were parsed. If a data block contains point information that needs processing, the point information in that data block is created first. After the points are created, the creation of edge information in the data block waits until all points in all data blocks less than or equal to id-max have been created. The purpose of this operation is also to ensure that the start and end points of edges are created successfully first. Finally, all edge information for this data block is created in the database.
[0041] like Figure 4The diagram shows the state of the import modules at a certain moment. Import module-1 has completed the creation of all point information in data block-4, while import modules-2 and-3 are processing the point information of data blocks-5 and-3. From step 2, we know that the maximum ID obtained by data block-4 is id-max = 5. Therefore, the creation of edges in data block-4 must wait for the completion of point information creation in data blocks-5 and-3. At this time, import module-1 can prioritize processing other data blocks, and switch back to continue processing data block-4 once the condition is met. After the point creation of data blocks-5 and-3 is completed, the edge dependency relationship is broken, and import module-1 can continue processing other data blocks after completing the creation of data block-4. In other words, in the case of multiple import modules, each import module independently processes its own data blocks. Overall, point creation and edge creation are performed in parallel. To ensure that the start and end point information of each edge is created in the database before the edge itself, a control algorithm must be added to control the data block processing flow; therefore, the dependency relationship between data blocks is introduced. However, the amount of point data is much smaller than the amount of edge data, so waiting for the creation of points will also be a short process.
[0042] Step 4: This step is to mitigate the impact of the dependencies that occurred in Step 3, so it is not a necessary step. To shorten the time it takes for the import module to create point data, an additional point import module can be added. This module would specifically handle the concurrent creation of start and end points in the database system, quickly completing the creation of points in the database system. Because the points exist independently, the point import module can create point information in batches. Alternatively, the size of the mapping blocks can be reduced, decreasing the number of points in each data block, which will directly weaken the dependencies between data blocks.
[0043] The implementation method of this embodiment can handle massive amounts of complex relationship information by reading and parsing simultaneously, loading edge information along with the start and end points of each edge. This effectively shortens data loading time and offline time for loading graphs in the database system, allowing the loaded data to be quickly provided to external users.
[0044] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0046] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for implementing online data loading based on a graph database, characterized in that: Includes the following steps: Step 1: Divide the text data to be loaded into multiple mapping blocks in a continuous manner, and map each mapping block to the computer system; each mapping block obtains a unique ID representing itself in an incremental manner; Step 2: Parse the mapping blocks one by one in ascending order of ID to obtain the data blocks that correspond one-to-one with the mapping blocks, and record the largest ID in the mapping block being parsed at the moment; The data block consists of start and end point information and edge information; Step 3: Process data blocks one by one according to the parsing completion order. The processing operations include: firstly creating the start and end points of the data blocks in the database system; after all the start and end points of the data blocks with IDs less than or equal to the maximum ID have been created in the database system, creating the edge information of the data blocks in the database system; in Step 2, parsing the mapping blocks one by one according to the ascending ID order to obtain the data blocks corresponding to the mapping blocks specifically includes: By offsetting the mapped address to the beginning and end of the block, it is determined whether there is cross-block data; if there is cross-block data at the beginning of the block, the cross-block data at the beginning of the block is skipped; if there is cross-block data at the end of the block, part of the content of the beginning of the next mapped block is read. According to the configuration file, a row of data is parsed into: start point information, end point information, and edge information; the start point information, end point information, and edge information form a data block that corresponds one-to-one with the mapping block.
2. The method for implementing online data loading based on a graph database according to claim 1, characterized in that: If the start / end point information has already been successfully created or parsed, then the start / end point information will not be added to the data block.
3. The method for implementing online data loading based on a graph database according to claim 1, characterized in that: The database system is a graph database system or a data system that maintains vertex-edge relationships.
4. The method for implementing online data loading based on a graph database according to claim 1, characterized in that: In step 3, the start and end points are created concurrently.
5. A system for implementing online data loading based on a graph database, characterized in that: include: The mapping module is used to divide the text data to be loaded into multiple mapping blocks in a continuous manner, and map each mapping block to the computer system; each mapping block obtains a unique ID representing itself in an incremental manner; The parsing module is used to parse the mapping blocks one by one in ascending order of ID, obtain the data blocks that correspond one-to-one with the mapping blocks, and record the largest ID in the mapping block currently being parsed; The data block consists of start and end point information and edge information; The import module is used to process data blocks one by one according to the parsing completion order. The processing operations include: firstly creating the start and end points of the data blocks in the database system; after all the start and end points of the data blocks with IDs less than or equal to the maximum ID have been created in the database system, creating the edge information of the data blocks in the database system. In the parsing module, the offset mapping address is used to determine whether there is cross-block data. If there is cross-block data at the beginning of the block, it is skipped. If there is cross-block data at the end of the block, part of the content of the beginning of the next mapped block is read. According to the configuration file, a line of data is parsed into: start point information, end point information, and edge information. The start / end point information and edge information form a data block that corresponds one-to-one with the mapped block. If the start / end point information has been successfully created or parsed, it is not added to the data block.
6. The implementation system for online data loading based on a graph database according to claim 5, characterized in that: Also includes: The point import module is used to handle the concurrent creation of start and end points in the database system.
7. The implementation system for online data loading based on a graph database according to claim 5, characterized in that: The database system is a graph database system or a data system that maintains vertex-edge relationships.
8. The implementation system for online data loading based on a graph database according to claim 5, characterized in that: Assuming the import module has access to data blocks i If the start and end points of a data block are created, but the start and end points of some data blocks with IDs less than or equal to the maximum ID are not yet fully created, the import module will wait for them to finish processing or process the next data block in the order of parsing completion. Once the start and end points of all data blocks with IDs less than or equal to the maximum ID have been fully created in the database system, the import module will process the data blocks... i Import and create the edge information.