System and method for full-table scanning of distributed graph database

Abstract

The invention relates to the field of databases, in particular to a distributed graph database full-table scanning system and method. Comprising a client, a calculation engine and a storage server, and specifically comprises the following steps: the client is used for sending a query request to the calculation engine and obtaining a record loaded into a memory by the calculation engine; the calculation engine is used for loading a corresponding record into a local memory of the client according to the query request; wherein the number of records loaded into the memory each time is the maximum number of records allowed to be cached in the memory; and the storage server is used for storing all records of the graph database so that the calculation engine can read the records needing to be queried. According to the method, the records needing to be queried can be loaded into the memory in batches, the highest utilization rate is achieved under the limited memory, the most efficient data acquisition mode is selected according to the speeds of different reading modes, and the data query efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of databases, and in particular to a system and method for full table scanning of a distributed graph database. Background Art

[0002] With the popularization of the Internet and the development of IoT technology, the amount of data is growing exponentially. Large-scale graph databases need to process billions or even tens of billions of nodes and edges, which poses a huge challenge to data query. In order to effectively deal with these problems, graph databases introduce distributed storage and batch data acquisition mechanisms. Under this mechanism, graph databases use two types of key messages to coordinate the interaction between the client and the server, namely msgrun and msgpull.

[0003] Although the msgrun and msgpull models can effectively improve system performance and stability in most scenarios, they are still insufficient in some extreme cases. For example, the client needs to send msgpull messages frequently to gradually obtain complete query results, which increases the overhead of message communication, especially in a high-latency network environment, which will significantly affect performance. In addition, query results need to be continuously returned from the server memory. When the amount of data is extremely large, even if a batch return mechanism is adopted, the memory pressure on the server may still be too high. For example, when it is necessary to scan the entire table of 20,000,000 data, assuming that the client requests 1,000 data at a time, then the communication between the client and the server will take 20,000 times to complete. If each communication takes 30ms, then it will take 600,000ms to scan the entire table. If in a high-latency network environment, the time required to scan the entire table will be significantly increased, and the overall scanning performance will be greatly reduced.

[0004] In view of this, how to overcome the defects of the prior art and solve the problem of excessive communication times between the client and the server and excessive memory pressure in the prior query method is a problem to be solved in this technical field. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention solves the problem of excessive communication times between the client and the server and excessive memory pressure in the existing query method.

[0006] The embodiment of the present invention adopts the following technical solution:

[0007] In the first aspect, the present invention provides a system for full table scanning of a distributed graph database, specifically comprising: a client, a computing engine and a storage service end, specifically comprising: the client is used to send a query request to the computing engine, and obtain the records that the computing engine has loaded into the memory; the computing engine is used to load the corresponding records into the local memory of the client according to the query request; wherein the number of records loaded into the memory each time is the maximum number of records allowed to be cached in the memory; the storage service end is used to save all records of the graph database for the computing engine to read the records to be queried.

[0008] Preferably, the computing engine includes a memory iterator and a log manager, specifically including: the memory iterator is used to obtain the records to be queried from the storage service client through network reading, or to obtain the records to be queried from the local log file through disk reading, and load the obtained records into the memory; the log manager is used to obtain the records to be queried from the storage service client through network reading, and cache the obtained records in the local log file.

[0009] In the second aspect, the present invention provides a method for full table scanning of a distributed graph database, specifically: using the distributed graph database full table scanning system provided in the first aspect to perform graph database query, specifically including: obtaining the total number of records to be queried and the size of each record in the query request, calculating the available memory capacity according to the current memory usage, and calculating the number of cache records that can currently be loaded in the memory according to the size of each record and the available memory capacity; according to the calculated number of cache records, the records to be queried are loaded into the memory at one time, and the records in the memory are used to respond to the query request.

[0010] Preferably, obtaining the total number of records to be queried and the size of each record in the query request specifically includes: parsing the parameters of the query request, and calculating the total number of records to be queried based on the parameters; obtaining the total data size and the total number of records in the graph database based on the metadata and statistical information of the graph database, calculating the average size of each record based on the total data size and the total number of records, and using the average size as the size of each record.

[0011] Preferably, the calculating of available memory capacity based on current memory usage specifically includes: calculating remaining memory capacity based on total memory and currently used memory capacity, calculating reserved memory capacity based on remaining memory capacity and a specified reserved ratio, and taking the difference between remaining memory capacity and reserved memory capacity as available memory capacity.

[0012] Preferably, the calculating the number of cache records that can be loaded into the memory according to the size of each record and the available memory capacity specifically includes: calculating the maximum number of records that can be loaded in the available memory capacity according to the size of each record, and using the maximum number of records as the number of cache records.

[0013] Preferably, the method of calculating the number of cache records that can be loaded into the memory based on the size of each record and the available memory capacity also includes: using machine learning to predict the number of cache records based on historical query requests, historical available memory capacity and historical number of cache records, and using the predicted result as the number of cache records for the next load.

[0014] Preferably, the records that need to be queried are loaded into the memory at one time, specifically including: when the total number of records is less than or equal to the specified batch processing number, all records that need to be queried are obtained, and the obtained records are loaded into the memory at one time; when the total number of records is greater than the specified batch processing number, a corresponding number of records are obtained according to the number of cached records, and the obtained records are loaded into the memory at one time.

[0015] Preferably, the process of loading the acquired records into the memory as the current batch records specifically includes: acquiring the network read speed and the disk read speed when loading the records; when the network read speed is greater than the disk read speed, loading the records into the memory from the storage service end via the network read method; when the network read speed is less than or equal to the disk read speed, loading the records into the memory from the local log file via the disk read method.

[0016] Preferably, after loading the current batch of records from the local log file into the memory by disk reading, the method further includes: obtaining the number of records saved in the local log file, and when the number of records saved in the local log file is less than the number of cached records, obtaining a specified multiple of the number of cached records from the distributed storage service client, and writing the obtained records into the local log file.

[0017] Compared with the prior art, the beneficial effects of the present invention are: providing a system for full table scanning of a distributed graph database, wherein the computing engine loads the records to be queried into the memory in batches according to the query request, achieving the highest utilization rate under limited memory, avoiding excessive memory pressure and significantly improving the performance of scanning the full table. In the preferred embodiment, a memory iterator and a log manager are set in the computing engine, and the data to be used is pre-stored locally through the log manager, so that when loading records, it is possible to choose to load them through the storage service end in a network reading mode, or through the log manager in a disk reading mode.

[0018] On the other hand, the present invention provides a method for full table scanning of a distributed graph database. For each query request, the maximum number of cache records that the computing engine can load into the memory each time is calculated according to the current available memory capacity, and the records to be queried are loaded into the memory at one time according to the calculated maximum number of cache records, thereby reducing the number of communications between the client and the storage service end and reducing the time-consuming impact of communication. In the preferred embodiment, the network reading method or the disk reading method is selected according to the speed of different reading methods to reduce network delays and improve the efficiency of processing large amounts of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of the system structure of a distributed graph database full table scan provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic diagram of recording loading method in the prior art;

[0022] Figure 3 A schematic diagram of a record loading method in a system provided in an embodiment of the present invention;

[0023] Figure 4 A flow chart of a method for full table scanning of a distributed graph database provided by an embodiment of the present invention;

[0024] Figure 5 A flow chart of another method for full table scanning of a distributed graph database provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0027] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0028] A graph database is a database management system designed for storing and querying graph data structures. In a graph database, data is organized in the form of a graph, where the data includes nodes (representing entities) and edges (representing the relationships between entities). This data structure is very suitable for representing complex relational data such as social networks, knowledge graphs, recommendation systems, and logistics networks.

[0029] Traditional relational databases require strict definition of the structure of data tables, including column names, data types, and relationships. This structure has clear specifications when storing and querying data. Nodes and edges can have different types and quantities of attributes, which can dynamically adapt to the changes and complexity of data without strict planning in advance. This flexibility is particularly important when facing complex data in the real world, because the nature of data is often difficult to define with a fixed structure. By allowing diversity in attributes, graph databases give data greater expressiveness and can better capture and reflect the multidimensionality and diversity of data.

[0030] In summary, the flexible model of graph database is an evolution of modern data management. It goes beyond the limitations of traditional databases, can process complex data more freely and powerfully, and maximize the innovative potential of data models. This feature has not only triggered major changes in the field of data science, but also brought unprecedented opportunities for business intelligence and decision support.

[0031] However, although graph databases perform well in processing complex relational data, there are still challenges and requirements in certain aspects. With the popularization of the Internet and the development of IoT technology, the amount of data is growing exponentially. Large-scale graph databases need to process billions or even tens of billions of nodes and edges, which poses a huge challenge to data query.

[0032] As the amount of data grows dramatically, modern graph databases need to maintain efficient performance and resource utilization when processing large-scale complex queries. Database systems usually face two major challenges when responding to query requests: one is how to efficiently execute queries and generate results, and the other is how to avoid excessive burden on memory and network when returning large amounts of data.

[0033] Database systems usually rely on a combination of memory and logs (data_log) to store data. Logs are usually saved in the form of log files on local disks. The reading speed of memory is much faster than that of data_log on disks. Therefore, many modern database systems use memory to cache query results or part of the data to improve query efficiency. However, in the prior art, it is usually necessary to load all the data to be queried in the memory, and then return all the data to be queried to the client at one time. When the data set to be stored exceeds the loadable range of the memory, the memory bottleneck will significantly affect the system performance, and even cause query timeouts or system crashes.

[0034] In order to effectively deal with these problems, graph databases have introduced a mechanism for distributed storage and batch data acquisition. Under this mechanism, graph databases use two types of key messages to coordinate the interaction between the client and the server, namely msgrun and msgpull.

[0035] msgrun is a message used to carry execution statements. The client uses it to send query requests (such as Cypher statements) to the server to execute complex graph database queries. After receiving the msgrun message, the server will start parsing and executing the query, but at this time the server will not return all the results to the client at once. Instead, the server will obtain query data batch by batch according to the specified batch processing quantity set in the graph database, and gradually prepare and cache each batch of query results obtained in the memory, waiting for further requests from the client.

[0036] The msgpull message is used to obtain query results in batches to avoid resource problems caused by returning a large amount of data at one time. When the client sends the msgpull message, it specifies the number of query results to be obtained, which is usually the specified batch processing number set in the graph database (for example, 1,000 data). The server returns the corresponding number of data in batches according to the request. The client can send msgpull messages as many times as needed until all query results are obtained.

[0037] In a typical query execution process, the client first sends a msgrun message to execute the statement, and the server executes the query and puts some results into memory. Then, the client sends a msgpull message to request the query results in batches. Each msgpull message requests the server to return a specific number of results, and the server returns a corresponding batch of data based on the request. When all batches of data are pulled, the entire query is considered complete.

[0038] Although the msgrun and msgpull models can effectively improve system performance and stability in most scenarios, they are still insufficient in some extreme cases. For example, the client needs to send msgpull messages frequently to gradually obtain complete query results, which increases the overhead of message communication, especially in a high-latency network environment, which will significantly affect performance. In addition, query results need to be continuously returned from the server memory. When the amount of data is extremely large, even if a batch return mechanism is adopted, the memory pressure on the server may still be too high. For example, when scanning the entire table of 20,000,000 data, assuming that the client requests 1,000 data at a time, then the communication between the client and the server will take 20,000 times to complete. If each communication takes 30ms, then it will take 600,000ms to scan the entire table. If in a high-latency network environment, the time required to scan the entire table will be significantly increased, and the overall scanning performance will be greatly reduced.

[0039] Embodiment 1:

[0040] In order to solve the above technical problems, this embodiment provides an efficient, scalable and flexible graph database data query system to meet the growing data processing needs and provide important support and assistance for research and applications in the fields of data analysis, knowledge graph construction, social network analysis, etc.

[0041] like Figure 1 As shown, the system provided by this embodiment includes: a client, a computing engine and a storage service end.

[0042] The client is used to send query requests to the computing engine and obtain the records that the computing engine has loaded into the memory. In actual implementation, the client can include various functional modules related to graph database operations. Figure 1 In the client, there is a graph platform for building a graph database, a driver for connecting to the graph database, and a Cypher-shell for operating the graph database through Cypher statements. Through the graph platform, the client can send the msgrun command to the computing engine as a query command; after the computing engine loads the query results into the memory according to the query command, the client can send the msgpull command to the computing engine to obtain the records loaded into the memory. In general scenarios, the number of records obtained each time is the specified batch processing number set by the graph database.

[0043] The computing engine is used to load the corresponding records into the local memory of the client according to the query request. In this embodiment, a computing engine is added between the existing client and the storage service end, and the acquisition and caching of records are managed by the computing engine. The computing engine caches the records that the client needs to use into the local memory of the client. The client can quickly obtain the records directly from the memory without reading from the log file or distributed storage node. Since the memory reading speed is much higher than the disk reading speed and the network transmission speed, the data query efficiency is significantly improved.

[0044] On the other hand, in order to improve memory utilization and reduce the number of times data is loaded into the memory, the number of records loaded into the memory each time is the maximum number of records allowed to be cached in the memory. In specific implementation, you can refer to the following process to calculate the number of records loaded into the memory each time according to the memory usage and the size of each record.

[0045] In actual implementation, the msgrun message can be used as a query request. When the computing engine receives the msgrun message sent by the client, it first checks the total number of records in the target data table to be queried as a basis for subsequent data processing. Specifically, the computing engine can obtain the total number of records in the target data table through database metadata or other storage indexes.

[0046] In the system provided by this embodiment, when the msgrun message responds successfully, the records to be queried will be preloaded into the memory through the computing engine so that the results can be requested at the same time as the query is started. In actual implementation, the number of records loaded into the memory each time can be determined based on the total number of records to be queried to optimize the response speed and memory resources. If the total number of records is less than or equal to the specified batch processing number set by the graph database (such as the default 1000), the computing engine directly returns all records to the client at one time, reducing additional processing and improving the query response speed. If the total number of records is greater than the specified batch processing number, the computing engine iteratively loads as many records as possible based on the current available memory capacity, and then returns the records to be queried to the client in batches based on the client's msgpull request to control memory usage and avoid the impact of loading a large number of records at one time on system performance.

[0047] In actual implementation, due to memory capacity limitations, the memory may not contain all the records requested by the client. Therefore, when the client obtains records, the calculation engine determines whether the obtained records have been loaded into the memory; if the records have been loaded into the memory, the records in the memory are directly returned to the client; if the records have not been loaded into the memory, the required records are directly loaded into the memory according to the current memory capacity, or the required records are loaded into the memory after eliminating some data according to the specified data elimination algorithm, and then the newly loaded data is returned to the client. By iteratively loading records, as many records as possible are cached in the memory, thereby making full use of the memory capacity.

[0048] In the prior art, when using the msgpull message to obtain data, the amount of data obtained each time is usually the number of batch requests that the graph database can process at one time. When the total number of records to be queried is greater than the number of batch requests, multiple communications with the storage service are required to obtain all the records. In actual implementation, the memory capacity of the hardware device that carries the graph database is usually large, and the amount of data that can be loaded into the memory each time will be much larger than the specified batch processing number set by the graph database. Therefore, using the method in this embodiment, it is possible to load dozens or hundreds of times the number of batch requests into the memory through a single communication without the need for multiple communications according to the prior art, thereby reducing the number of interactions between the client and the storage service and improving query efficiency.

[0049] In the system provided by this embodiment, the computing engine will calculate the maximum amount of data that can be loaded into the memory at one time based on the current memory resource situation to reduce the number of times data is requested from the storage service end. For example, when the available memory capacity of the computing engine is 2GB, this capacity is used as the benchmark for the maximum number of cache records that can be loaded. In actual implementation, the computing engine can obtain the current actual remaining memory capacity through dynamic memory monitoring to calculate the appropriate number of cache records for each load, thereby ensuring memory utilization efficiency and system stability during data processing.

[0050] The computing engine obtains the total data size and total number of records in the target data table through the metadata and statistical information of the graph database, thereby calculating the average size of each record, and using the average size of each record as the size of each record when calculating the number of cache records that can be loaded. If the size of each record is 1KB, the computing engine can accommodate approximately 2,000,000 records (2GB / 1KB) in the remaining memory capacity of 2GB, and this number of records can be used as the theoretical number of cache records that can be loaded at present. Calculating the average data size through metadata query and obtaining the current actual remaining memory capacity through dynamic memory monitoring can improve the accuracy of batch settings and reduce memory pressure caused by inaccurate estimates. Furthermore, in actual implementation, there are usually multiple tasks using memory at the same time. Therefore, the computing engine also needs to reserve a certain proportion of memory space to prevent memory overflow caused by batch record loading. For example, 80% of the remaining memory capacity is used as the upper limit of data loading, and 20% of additional memory space is reserved to meet the needs of other tasks in the system. When the remaining memory capacity is 2GB, after setting a specified reserved ratio of 20%, the number of cache records that can be loaded is currently 1,600,000 records. By reserving memory space, the risk of memory overflow can be effectively reduced, ensuring the stable operation of the computing engine in large-scale data loading scenarios.

[0051] In actual implementation, in addition to dynamically monitoring the remaining memory capacity when the client initiates a query or the computing engine obtains data, the actual remaining memory capacity can be dynamically monitored according to a specified time period, or the actual remaining memory capacity can be dynamically monitored each time the client obtains data. If the current remaining memory capacity has changed relative to the last monitoring, the number of cached records is recalculated based on the current remaining memory capacity. When the remaining memory capacity increases, the number of records that can be cached by the increased remaining memory capacity is calculated, and the computing engine loads the corresponding number of records from the storage service end into the memory to fully utilize the memory capacity through iterative loading; when the remaining memory capacity decreases, the number of records that can be cached by the reduced remaining memory capacity is calculated, and the corresponding number of records in the memory are cleaned up according to the specified data elimination strategy to avoid errors such as memory overflow caused by insufficient reserved memory space.

[0052] To further optimize system performance, a machine learning mechanism can also be introduced into the computing engine to dynamically adjust the batch size based on historical query requests, historical available memory capacity and historical cache record numbers, current system load, and memory status. Through the machine learning mechanism, the computing engine can predict the amount of data required for future queries and future memory usage through historical data, and adaptively adjust the number of cache records loaded each time based on historical data, and increase the number of cache records loaded each time as much as possible without memory overflow, so as to maximize data transmission efficiency and reduce resource consumption, and balance memory usage and data transmission times. The storage service end is used to save all records of the graph database for the computing engine to read the records to be queried. Distributed graph databases store and process graph data in a distributed environment. Graph data is distributed on multiple storage nodes, and has the characteristics of data sharding, data replication, data consistency, and scalability, so as to provide high performance and scalability through distributed computing and storage. In actual implementation, the storage service end will uniformly manage all storage nodes and interact with the computing engine through network interfaces for requests and data.

[0053] In the above query process, although the memory resources have been used as much as possible, it is still necessary to load records into the memory multiple times in the usual implementation scenario, and network communication with the storage service end is still required each time the records are loaded. In order to maintain high-performance query in a limited memory environment, in this embodiment, the computing engine includes a memory iterator and a log manager.

[0054] The memory iterator is used to obtain the records to be queried from the storage service client through network reading, or to obtain the records to be queried from the local log file through disk reading, and load the obtained records into the memory. When the client needs to obtain more queried records, it requests the next batch of data from the computing engine through the msgpull message. The computing engine reads the records loaded into the memory from the memory iterator and returns them to the client. In actual implementation, the records read by the memory iterator will be cached in the memory until they are cleaned up or replaced, so as to ensure a fast response when the client queries the same data later, reduce repeated read requests to the storage service client, and effectively reduce the system I / O consumption and optimize the overall response performance. However, in the process of large-scale data query, it is also necessary to regularly clear the records that the memory iterator has loaded into the memory to ensure that the memory space is released in time, prevent the accumulation of useless data in the memory from causing system resource tension, and improve the speed of subsequent data reading.

[0055] In actual scenarios, the same business usually only needs to use the same set of data, and the data that the client needs to query is usually repetitive and continuous. Therefore, the computing engine can use the log manager to further cache the records that have been loaded into the memory in the local log file, or obtain the next batch of records that may be loaded from the storage service and write them into the local log file. The log manager is used to obtain the records to be queried from the storage service through network reading, and cache the obtained records locally.

[0056] Logs are usually saved in the form of log files on the client's local disk, and the client can directly read the records in the log by disk reading. Figure 2 As shown, although log files are used to save records, the records are not cached in the client's local memory. Instead, the client directly obtains records from the log file through disk reading. When the required records do not exist in the log file, the client obtains the records from the server through network reading and writes them into the log file. However, both disk reading and network reading are much slower than directly obtaining records from memory. In this embodiment, the memory iterator caches the records obtained from the log file or storage service server into the client's local memory, and the client directly obtains the required records from the memory, thereby improving the client's data acquisition efficiency. Figure 3 As shown, according to the hardware resources of the computing engine and the storage service, the memory iterator can choose to obtain records from the storage service through network reading, or obtain records from the log through disk reading. If the network reading speed of the storage service is faster, data is read from the storage node of the storage service first; if the local disk reading speed of the computing engine is faster, the records cached in the log of the local disk are read first. In specific implementations, the process of the memory iterator obtaining records from the log file or storage service is independent of the process of the client obtaining records from the memory. The memory iterator can cache records in the memory when the system is idle to avoid resource occupation.

[0057] On the other hand, to reduce the frequency of log file writing operations, the computing engine can adopt a batch storage strategy. When the query result of the msgrun message is generated, if the number of record lines in the log is less than or equal to the number of cache records that can be currently loaded by the memory iterator, the computing engine needs to return twice the number of cache records from the storage service end through the network and write them into the log, thereby reducing the single write amount, improving storage I / O performance, and providing sufficient data support for subsequent batch reading of data.

[0058] The distributed graph database full table scan system provided in this embodiment implements an asynchronous query method through a computing engine, optimizes the performance of large-scale data query in the graph database, and is suitable for scenarios that require scanning a large amount of data. The system covers a wide range of data query operations in the graph database and can achieve efficient, concurrent and accurate large-scale data retrieval processes.

[0059] Embodiment 2:

[0060] Based on the system for full table scanning of a distributed graph database provided in the above-mentioned Example 1, the present invention also provides a method for full table scanning of a distributed graph database that can be used to implement the above-mentioned method. The method can be applied to the fields of data query, memory compression and batch processing technology of graph data, and optimizes the performance of data query in large-scale and complex data structures.

[0061] like Figure 4 As shown, the specific steps of the method for full table scanning of a distributed graph database provided by an embodiment of the present invention are as follows:

[0062] Step 101: Obtain the total number of records to be queried in the query request and the size of each record, calculate the available memory capacity according to the current memory usage, and calculate the number of cache records that can be currently loaded in the memory according to the size of each record and the available memory capacity.

[0063] In the method provided in this embodiment, when a query request sent by a client via a msgrun message is received, the total number of records in the target data table to be queried is first determined. In a specific implementation, the parameter of the msgrun message includes the name or range of the target data table to be queried, and the total number of records to be queried can be calculated based on the parameters by parsing the query request parameters.

[0064] When the total number of records is less than or equal to the specified batch processing number, all records that need to be queried are obtained, and the obtained records are loaded into the memory at one time. When the total number of records is greater than the specified batch processing number, the corresponding number of records is obtained according to the number of cached records, and the obtained records are loaded into the memory at one time. In specific implementations, the number of batch requests that the graph database can process at one time is usually used as the specified batch processing number, so that the graph database can process data in batch form. In a specific scenario, the default number of batch requests that the graph database can process for the first time is 1000 records; if the total number of records is less than or equal to 1000, all records are returned directly; if the total number of records is greater than 1000, they cannot be loaded at one time, but the maximum number of cached records that can be loaded into the memory is obtained in a subsequent manner, and the data is loaded into the memory at one time according to the maximum number of cached records, and based on the client's usage.

[0065] In actual implementation, memory usage will change dynamically at any time. In order to avoid memory overflow while maximizing the use of memory resources, it is necessary to monitor memory usage and calculate the available memory capacity for caching records based on the current memory usage. Specifically, the remaining memory capacity is calculated based on the total memory and the currently used memory capacity, and the reserved memory capacity is calculated based on the remaining memory capacity and the specified reserved ratio, and the difference between the remaining memory capacity and the reserved memory capacity is used as the available memory capacity.

[0066] In order to use the available memory capacity as much as possible, after obtaining the available memory capacity, the maximum number of records that the available memory capacity can carry is used as the number of cache records that need to be loaded into the memory. Specifically, the total data size and total number of records in the graph database are obtained based on the metadata and statistical information of the graph database, and the average size of each record is calculated based on the total data size and total number of records, and the average size is used as the size of each record. Based on the size of each record, the maximum number of records that can be loaded in the available memory capacity is calculated, and the maximum number of records is used as the number of cache records.

[0067] Furthermore, machine learning can be used to predict the number of cache records based on historical query requests, historical available memory capacity, and historical number of cache records, and the predicted result can be used as the number of cache records for the next load, thereby further avoiding the reduction of remaining memory capacity due to sudden memory usage, which causes the calculated number of cache records to be inconsistent with the actual available memory capacity at the current moment, and obtaining the most appropriate number of cache records, thereby effectively reducing memory pressure and avoiding system instability caused by excessive memory usage during large-scale data queries.

[0068] In a specific scenario, the remaining memory space is 2GB, and the specified reserved ratio is 20%. At this time, 80% of the remaining memory capacity can be used as the available memory capacity. In this scenario, the calculated size of each record is about 1KB, so the number of cache records currently loaded can be 1,600,000.

[0069] Step 102: according to the calculated number of cache records, the records to be queried are loaded into the memory at one time, and the records in the memory are used to respond to the query request.

[0070] After obtaining the number of cache records that can be loaded, the computing engine sends a data request to the storage service through the msgpull message, obtains the data corresponding to the number of cache records, and the client obtains the query results from the memory. Since the amount of data that can be loaded into the memory each time is much larger than the specified batch processing quantity set by the graph database, the number of records loaded into the memory each time is equivalent to the number of records loaded multiple times in the prior art, which can reduce the number of interactions between the client and the storage service; at the same time, since the data reading speed of the memory is much faster than the disk reading speed or the network reading speed, it can improve the data reading efficiency of the client.

[0071] In actual implementation, the client may reuse the same set of records. In this case, the records loaded into the memory will remain in the memory without having to obtain the data from the storage server again, further reducing the number of data interactions between the client and the storage server.

[0072] When the client needs to use records that are not loaded into the memory, it is necessary to calculate the number of cache records that can be loaded next time again according to step 101, obtain the corresponding records from the storage service again according to the number of cache records, and update the records loaded in the memory so that the client can obtain the new data it needs to use.

[0073] After steps 101 and 102 provided in this embodiment, the query of large batches of data can be completed in the graph database, achieving efficient query speed, low resource usage and scalability.

[0074] In the above process, due to the limitation of memory hardware resources, it is impossible to load all the records to be queried into the memory at once in all scenarios. Figure 5 As shown in the figure, each time the computing engine needs to load new data into the memory, it can obtain the corresponding number of records from the storage service end in the following manner and load the obtained records into the memory at one time. In actual implementation, the following data loading process is repeated by the computing engine when it needs to load data into the memory until all query requests from the client are completed.

[0075] Step 201: Obtain the network reading speed and disk reading speed when loading records.

[0076] To ensure high-performance query in a limited memory environment, the system of Example 1 introduces logs stored on local disks. When the client needs to obtain records that are not cached in the memory, based on the hardware characteristics of the storage service end, the computing engine can choose to prioritize obtaining them from the distributed storage nodes through network reading, or to prioritize reading from the logs on the local disk through disk reading. In actual implementation, the network status and disk usage status will change in real time. Therefore, it is necessary to obtain the network reading speed and disk reading speed when loading records in real time, and select the method of loading records according to the real-time speed.

[0077] Step 202: When the network reading speed is greater than the disk reading speed, the record is loaded into the memory from the storage service end through network reading.

[0078] In a certain scenario, when loading records, the network read speed of the storage service client to load data into the memory is 800Mbps, while the computing engine disk read speed is 500Mbps. Therefore, choose to obtain records from the storage service client through network reading first. To facilitate the subsequent use of data, you can also write the records obtained by network reading into the log when the system is idle, so that they can be used the next time you choose disk reading.

[0079] Furthermore, if the disk read speed of the computing engine is higher than the network bandwidth speed of the storage service, the computing engine can also adopt a batch storage strategy to avoid frequent writing of records to the log during the query process. Get the number of records saved in the log. When the number of records saved in the log is less than the number of cached records, get the records of the specified multiple of the number of cached records from the distributed storage service, and write the obtained records to the log. In a specific scenario, after the query results are initially generated, check the number of record rows saved in the log. If the number of rows is less than or equal to the number of cached records, 1,600,000, the computing engine requests the storage service to return twice the number of cached records, 3,200,000, and records the returned data and writes them to the log, thereby effectively reducing the I / O operations of the log and improving query performance.

[0080] Step 203: When the network reading speed is less than or equal to the disk reading speed, the record is loaded from the log into the memory by disk reading.

[0081] In a certain scenario, when loading records, the network read speed of the storage service client to load data into the memory is 500Mbps, while the disk read speed of the computing engine is 800Mbps. Therefore, it is chosen to obtain records from the logs on the local disk of the computing engine first to ensure the data reading speed. After the data in the log is read, a new batch of data can be pulled from the storage service client through the network and written into the log.

[0082] In actual implementation, when each batch of records is loaded, the above steps 201 to 203 can be repeated, and different methods can be selected to complete the loading of the records in the memory according to the real-time status to obtain the highest query efficiency.

[0083] The method provided in this embodiment can reduce memory usage, improve system performance, and achieve efficient query speed, low resource usage, and scalability through reasonable memory management, batch calculation, and real-time decision-making of multi-source data allocation.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed graph database full table scan system, characterized in that: include: Client, computing engine and storage service end, including: The client is used to send query requests to the computing engine and obtain the records that the computing engine has loaded into the memory; The computing engine is used to load corresponding records into the local memory of the client according to the query request; wherein the number of records loaded into the memory by the computing engine each time is the maximum number of records allowed to be cached in the memory; The storage service is used to save all records of the graph database so that the computing engine can read the records to be queried.

2. The system for full table scanning of a distributed graph database according to claim 1, characterized in that: The computing engine includes a memory iterator and a log manager, specifically including: The memory iterator is used to obtain the records to be queried from the storage service client through network reading, or obtain the records to be queried from the local log file through disk reading, and load the obtained records into the memory; The log manager is used to obtain the records to be queried from the storage service client through network reading, and cache the obtained records in the local log file.

3. A method for full table scanning of a distributed graph database, characterized in that: Using the distributed graph database full table scanning system described in claim 1 or 2 to query the graph database specifically includes: Get the total number of records to be queried and the size of each record in the query request, calculate the available memory capacity based on the current memory usage, and calculate the number of cache records that can be currently loaded in the memory based on the size of each record and the available memory capacity; According to the calculated number of cache records, the records to be queried are loaded into the memory at one time, and the records in the memory are used to respond to the query request.

4. The method for full table scanning of a distributed graph database according to claim 3, characterized in that: The total number of records to be queried and the size of each record in the query request specifically include: Parse the query request parameters and calculate the total number of records to be queried based on the parameters; The total data size and total number of records in the graph database are obtained based on the metadata and statistical information of the graph database. The average size of each record is calculated based on the total data size and the total number of records, and the average size is used as the size of each record.

5. The method for full table scanning of a distributed graph database according to claim 3, characterized in that: The calculation of available memory capacity according to current memory usage specifically includes: The remaining memory capacity is calculated based on the total memory and the currently used memory capacity. The reserved memory capacity is calculated based on the remaining memory capacity and the specified reserved ratio. The difference between the remaining memory capacity and the reserved memory capacity is taken as the available memory capacity.

6. The method for full table scanning of a distributed graph database according to claim 3, characterized in that: The calculation of the number of cache records that can be currently loaded into the memory based on the size of each record and the available memory capacity specifically includes: Based on the size of each record, calculate the maximum number of records that can be loaded within the available memory capacity, and use the maximum number of records as the number of cached records.

7. The method for full table scanning of a distributed graph database according to claim 3, characterized in that: The calculating the number of cache records that can be currently loaded in the memory according to the size of each record and the available memory capacity also includes: Based on historical query requests, historical available memory capacity, and historical number of cache records, machine learning is used to predict the number of cache records, and the prediction result is used as the number of cache records for the next load.

8. The method for full table scanning of a distributed graph database according to claim 3, characterized in that: The process of loading the records to be queried into the memory at one time specifically includes: When the total number of records is less than or equal to the specified batch processing quantity, all records that need to be queried are obtained and loaded into the memory at one time; When the total number of records is greater than the specified batch processing quantity, the corresponding number of records is obtained according to the number of cached records, and the obtained records are loaded into the memory at one time.

9. The method for full table scanning of a distributed graph database according to claim 8, characterized in that: The step of loading the acquired records into the memory at one time specifically includes: Get the network read speed and disk read speed when loading records; When the network read speed is greater than the disk read speed, the records are loaded from the storage service client into the memory through network reading; When the network read speed is less than or equal to the disk read speed, the records are loaded into the memory from the local log file by disk reading.

10. The method for full table scanning of a distributed graph database according to claim 9, characterized in that: After loading the records from the local log file into the memory by disk reading, the method further includes: Get the number of records saved in the local log file. When the number of records saved in the local log file is less than the number of cached records, get the records of the specified multiple of the number of cached records from the distributed storage service client and write the obtained records to the local log file.

Images