A distributed computing engine based on unified search and big data processing

Abstract

The application relates to a distributed computing engine based on unified search and big data processing, which comprises a unified interface module, a distributed execution module and a storage management module; the unified interface module is used for receiving a computing task and performing task analysis, so that a corresponding computing engine is started according to a task type obtained through task analysis; the distributed execution module comprises a search computing engine, a graph computing engine and a stream computing engine, which are respectively used for reading and executing corresponding computing tasks; and the storage management module is used for uniformly performing data table and storage management on data corresponding to the search computing engine, the stream computing engine and the graph computing engine. The application solves the problem that the data computing engine in the prior art cannot provide parallel and efficient real-time computing for multiple types of data processing tasks to perform real-time and rapid response when facing multiple complex computing scenes.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a distributed computing engine based on unified search and big data processing. Background Technology

[0002] With the rapid development of internet technology, the heterogeneous data generated by intelligence, surveillance, and reconnaissance systems is growing in size and number, and the types of data being designed are also increasing, such as image data, electronic reconnaissance data, visible spectral data, and intelligence data. As the diversity and complexity of data services in various application scenarios increase, there is a growing demand for more efficient data storage, computation, and low-latency real-time data processing to support decision-making; graph computing of massive amounts of data is needed to support relational reasoning and intelligence analysis; and there is also an increasing demand for rapid retrieval of massive amounts of data for efficient data acquisition. Therefore, technically, a unified computing engine is needed to handle the diverse data processing needs, including search computing, graph computing, and stream computing. Summary of the Invention

[0003] Based on the above analysis, the present invention aims to disclose a distributed computing engine based on unified search and big data processing, which solves the problem that existing data computing engines cannot provide parallel and efficient real-time computing for multiple types of data processing tasks in the face of various complex computing scenarios, so as to enable real-time and rapid response.

[0004] The objective of this invention is mainly achieved through the following technical solutions:

[0005] This invention discloses a distributed computing engine based on unified search and big data processing, comprising: a unified interface module, a distributed execution module, and a storage management module;

[0006] The unified interface module is used to receive computing tasks and parse the tasks to start the corresponding computing engine according to the task type obtained from the task parsing.

[0007] The distributed execution module includes a search computing engine, a graph computing engine, and a stream computing engine, which are used to read and execute corresponding computing tasks respectively. The search computing engine calculates the number of physical shards based on the performance of each physical node in the search cluster and the amount of indexed data, so as to store the document data accordingly.

[0008] The storage management module is used to perform unified data table and storage management for the data corresponding to the search computing engine, stream computing engine and graph computing engine.

[0009] Furthermore, the graph computing engine adopts a multi-Master-multi-Worker approach to build a distributed graph database system; and integrates various distributed graph computing algorithms and deep learning graph algorithms to construct a graph mining model, providing users with real-time graph query and offline graph analysis services;

[0010] The stream computing engine is used to schedule tasks based on the task information of each node in the directed acyclic graph and the resource information of the corresponding physical computing nodes, so as to achieve low-latency processing of streaming data.

[0011] Furthermore, the search computing engine includes an index storage management module and a search computing module, wherein:

[0012] The index storage management module is used to create an index based on document data, and divide the index into multiple index shards and store them on multiple physical shards. Each physical shard also stores the document data corresponding to that index shard. The number of physical shards is calculated based on the performance of each physical node in the search cluster and the amount of data in the index.

[0013] The search calculation module is used to perform keyword matching and matching degree calculation in each index shard based on the search keywords input by the client, and obtain the document IDs of multiple matching documents through query operations; and obtain the matching documents from the corresponding physical shards through value retrieval operations based on the document IDs and routing formulas.

[0014] Furthermore, the index created based on document data is an inverted index, which includes a word dictionary and an inverted list;

[0015] The word dictionary includes all words that have appeared in the document data and pointers to the corresponding words in the inverted list;

[0016] The inverted list includes the document ID of all documents in which each word appears in the word dictionary, its position in the document, and its frequency of occurrence.

[0017] Furthermore, the query operation includes: selecting an index shard as a coordinating node through round-robin, and sending a search request; the coordinating node broadcasting the search request to each index shard; each index shard performing keyword matching and calculating the matching degree based on the corresponding inverted index data, obtaining a matching result and sending it to the coordinating node, the matching result including the document ID and matching degree of each matching document; and the coordinating node performing a global sort on all received matching results to obtain the overall sorting result for all matching documents.

[0018] The value retrieval operation includes: based on the sorting result and the preset number of search results, determining the document ID to be returned through the coordination node, sending a get request to the index shard containing the document data, the corresponding index shard obtaining the corresponding document data from the physical shard based on the routing formula and returning it to the coordination node, and then returning the searched document data to the client through the coordination node.

[0019] Furthermore, the number of physical fragments is obtained using the following method:

[0020] The initial number of physical shards SN1 is calculated based on the amount of data in the index. Where D represents the amount of data in the index, which is obtained by performing data statistics on the indexes in the cluster;

[0021] The performance of each physical node in the cluster is verified to obtain the number of physical nodes that pass the verification, SN2.

[0022] The weighted sum of SN1 and SN2 is used to obtain the number of shards, shardNum, which meets the requirements of index data volume and node performance, using the following formula:

[0023]

[0024] Where θ is the expansion coefficient, e and k are the weighting coefficients, t is the number of redundant fragments, and 1≤t≤4.

[0025] Furthermore, the performance verification of each physical node in the search cluster to obtain the number of physical nodes that pass the verification, SN2, includes:

[0026] Verify the disk utilization rate of each physical node and the number of physical shards already existing on each physical node. If the disk utilization rate Dsrii of the physical node does not exceed 90% and the number of physical shards already existing on the physical node does not exceed a preset threshold, then the physical node is determined to pass the verification.

[0027] If the physical node passes the verification, the corresponding verification result nodeArr will be sent. i Set the element to 1, otherwise set it to 0, and store the verification result in the array nodeArr[].

[0028] The number of physical nodes that passed the verification, SN2, can be obtained using the following formula.

[0029] Furthermore, if the number of physical shards (shardNum) is greater than the number of currently available physical nodes, the upper limit (x) of the number of physical shards on the same physical node is adjusted using the following formula:

[0030]

[0031] If shardNum does not exceed the number of currently available physical nodes, then the number of physical shards is set according to the number of currently available physical nodes, that is, one physical shard is set for each physical node.

[0032] Furthermore, the preset index sharding placement strategy includes:

[0033] The performance of cluster nodes is evaluated using a linear weighted method and the following formula:

[0034] Qi = a × LAI + s × SNi + b × DsRi;

[0035] Where Qi represents the performance value of node i; LAi represents the average load of node i; SNi represents the number of shards of node i; DsRi represents the disk utilization of node i; a represents the first weighting coefficient; s represents the second weighting coefficient; and b represents the third weighting coefficient.

[0036] Based on the number of index shards (shardNum), and in descending order of node performance values, the index shards are divided and placed.

[0037] Furthermore, the search computing engine writes data through the following steps:

[0038] The Java client obtains a list of physical nodes based on the configured search cluster information, encapsulates the user-input data and index information into a TCP request, selects an index shard as a coordinating node through round-robin, and sends a write request to the coordinating node.

[0039] The coordinating node loads the corresponding index metadata according to the index information, and the index metadata includes the index mapping structure information;

[0040] The coordinating node checks whether the incoming data specifies a doc Id. If not, it generates a doc Id for the data. It also checks whether a routing value is specified. If not, it uses the doc Id as the routing value for the document data and calculates the corresponding physical fragment number using the routing formula.

[0041] The coordinating node obtains the primary shard information corresponding to the index and the IP address information of the corresponding primary shard node, and sends a remote request to the physical node corresponding to the IP address;

[0042] After the physical node receives the data, it first performs a content routing write consistency check. If the check passes, it performs a write sharding operation.

[0043] After the write operation is complete, write requests are sent to the replica shards in a loop. Once the replica shard is successfully written, a write success message is returned to the client through the coordinating node.

[0044] This invention can achieve at least one of the following beneficial effects:

[0045] 1. This invention integrates multiple data computing engines and performs data task parsing based on a unified data interface module, realizing parallel, efficient, and real-time processing of search computing, graph computing, and streaming data computing.

[0046] 2. When creating an index, the search computing engine of the present invention calculates the number of physical shards based on the performance of each physical node in the search cluster and the amount of data in the index, and places the index data in shards. It can optimize the settings according to user needs and hardware resources, thereby improving the utilization rate of hardware resources and the performance of the search computing engine.

[0047] 3. Before performing index sharding, this invention first verifies the disk utilization rate of each physical node and the number of physical shards already existing on each physical node. If the disk utilization rate Dsrii of a physical node does not exceed 90% and the number of physical shards already existing on the physical node does not exceed a preset threshold, then the physical node is determined to pass the verification and can be used for index sharding calculation. This effectively avoids the problem of load balancing caused by unreasonable sharding leading to some nodes being relatively busy or relatively idle. Attached Figure Description

[0048] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0049] Figure 1 This is a block diagram of the distributed computing engine structure based on unified search and big data processing in an embodiment of the present invention;

[0050] Figure 2 This describes the data writing process for the search computing engine in this embodiment of the invention.

[0051] Figure 3 This describes the workflow of the stream computing engine in this embodiment of the invention. Detailed Implementation

[0052] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and, together with the embodiments of the present invention, serve to illustrate the principles of the present invention.

[0053] One specific embodiment of the present invention discloses a distributed computing engine based on unified search and big data processing, such as... Figure 1 As shown, the computing engine includes: a unified interface module, a distributed execution module, and a storage management module;

[0054] The unified interface module receives computation tasks and parses them. The computation task includes task-related information such as task data type identifier, task name, task ID, task status information, and task storage location. Searching computation tasks also includes search keywords. Task parsing refers to obtaining task-related information based on the parsed computation task, which is used for subsequent data computation and other operations, such as starting the corresponding computation engine based on the data type identifier to perform corresponding data processing. The task data types include search computation tasks, stream computation tasks, and graph computation tasks.

[0055] Specifically, the distributed computing engine in this embodiment is based on a unified SQL engine, with a Cyber ​​interface for connecting to the search computing engine, stream computing engine, and graph computing engine. It also provides APIs and other interfaces to offer computing services. The engine parses the data type identifiers carried in the received computing tasks, matches them against the preset identifier databases of each computing engine, and launches the corresponding computing engine based on the parsing and matching results. This distributed big data computing engine is based on a distributed architecture design and deployed in a domestically produced server cluster, supporting servers with domestically built CPU architectures such as Phytium, Shenwei, Loongson, Zhaoxin, and Hygon. On the hardware servers, a Docker+Kubernetes-based basic resource management service is first deployed to achieve unified resource management and scheduling of the underlying domestically produced heterogeneous hardware. Then, search computing, graph computing, and stream computing services are deployed to realize a distributed computing engine integrating multiple types of computing tasks.

[0056] Furthermore, the distributed execution module includes a search computing engine, a graph computing engine, and a stream computing engine, which are used to read and execute corresponding computing tasks, respectively; the search computing engine is used to search for matching documents in the document data based on the keywords in the search computing task; including calculating the number of physical shards based on the performance of each physical node in the search cluster and the amount of indexed data, so as to store the document data accordingly;

[0057] Specifically, the search computing engine includes an index storage management module and a search computing module, wherein:

[0058] The index storage management module is used to create indexes based on document data, and divides the indexes into multiple index shards and stores them on multiple physical shards. Each physical shard also stores the document data corresponding to that index shard. The number of physical shards is calculated based on the performance of each physical node in the search cluster and the amount of data in the index.

[0059] The search calculation module is used to perform keyword matching and matching degree calculation in each index shard based on search keywords through query operations to obtain the document IDs of multiple matching documents; and to retrieve the matching documents from the corresponding physical shards through value retrieval operations based on the document IDs and routing formulas.

[0060] Specifically, for search computing engines, a reasonable configuration of the number of index shards is a crucial prerequisite for ensuring index performance. An index shard is both a container for data and a unit of work for executing read and write requests. How index shards are placed in the cluster is key to affecting index and cluster performance. This embodiment adopts a performance-based index shard placement strategy, combining cluster node performance and the size of the index data to configure index shards and physical shards, wherein the number of index shards and physical shards are the same; preferably, the number of physical shards is obtained through the following method:

[0061] The initial number of physical shards SN1 is calculated based on the amount of data in the index. Where D represents the amount of index data, obtained by statistically analyzing the indexes in the cluster. Since the size of the index data is a crucial factor affecting index sharding performance, and the optimal data size for a single shard is 25GB, the initial number of physical shards can be obtained based on the amount of index business data using the above formula.

[0062] The performance of each physical node in the cluster is verified to obtain the number of physical nodes that pass the verification, SN2.

[0063] The weighted sum of SN1 and SN2 yields the number of physical shards, shardNum, which is suitable for the index data volume and node performance.

[0064] As a specific implementation, firstly, the performance of each physical node in the physical node set nodeList in the search cluster is verified, and the verification result is stored in the array nodeArr[]. If the verification passes, the corresponding nodeArr element is set to 1; otherwise, it is set to 0. The verification conditions in this embodiment are: the disk utilization rate Dsr of the verified node does not exceed 90%, and the number of existing physical shards Si of the physical node does not exceed a preset threshold. In this embodiment, the threshold is set to 20, which can be set according to the actual performance of the physical nodes. The number of physical nodes that passed the verification is obtained.

[0065] Then, based on the number of physical nodes that passed verification and the initial number of physical shards, the number of shards, shardNum, that meets the requirements of index data volume and node performance is obtained using the following formula:

[0066] shardNum=[(e*SN1+k*SN2)*(1+θ)];

[0067] Where θ is the expansion coefficient, and e and k are the weighting coefficients, which are set according to the actual application.

[0068] Specifically, due to the instability of physical nodes, which can affect the stability of the distributed computing system, at least one number of physical shards can be set as redundant physical shards to ensure the stability of the distributed computing system. Therefore, based on the number of physical nodes that have passed verification and the initial number of physical shards, the number of shards, shardNum, that meets the requirements of index data volume and node performance is obtained using the following formula:

[0069] shardNum=[(e*SN1+k*SN2)*(1+θ)]+t;

[0070] Where t is the number of redundant fragments, 1≤t≤4, which are used for replacement when the preceding node fails.

[0071] In this embodiment, SN1 and SN2 are weighted and summed according to certain weighting coefficients, and a certain degree of scalability is introduced through the expansion coefficient θ to obtain a shard number result shardNum that comprehensively considers the amount of index data and node performance.

[0072] Furthermore, to ensure maximum utilization of the physical nodes in the current search cluster, if the shardNum calculated by the above formula does not exceed the number of currently available nodes, then the number of shards is set based on the number of currently available nodes. That is, the number of shards, shardNum, is calculated using the following formula:

[0073]

[0074] Based on the sharding result shardNum calculated using the above method, create an index and configure the number of shards number_of_shards: shardNum. Simultaneously, adjust the value of the index parameter total_shards_per_node to x. This parameter represents the upper limit on the number of shards for this index placed on the same node, facilitating the sharding placement strategy. x is calculated using the following formula:

[0075]

[0076] This embodiment uses a preset index sharding placement strategy to place index shards obtained from the index storage management module. During the index sharding placement process, physical nodes with better performance are prioritized for placement. This process involves verifying the `total_shards_per node` limit and considering the comparison between `shardNum` and the number of candidate nodes. Specifically, the preset index sharding placement strategy includes:

[0077] The performance of each physical node in the search cluster is evaluated using a linear weighted method and the following formula:

[0078] Qi = a × LAI + s × SNi + b × DsRi;

[0079] Where Qi represents the performance value of physical node i; LAi represents the average load of physical node i; SNi represents the number of existing shards of physical node i; DsRi represents the disk utilization of physical node i; and a, s, and b are weighting coefficients.

[0080] Based on the number of index shards (shardNum), the index shards are divided and placed in descending order of the performance values ​​of each physical node.

[0081] In this embodiment, index shards are placed based on the number of index shards (shardNum) and the list of nodes (nodeList) that meet performance requirements. During the placement process, it is necessary to verify whether the maximum number of physical shards created for each physical node (x, i.e., total_shards_per node()) has been reached. If the limit is reached, physical nodes are selected again for index shard placement. In this embodiment, each physical shard is a complete search engine, and retrieval within a physical shard is handled independently by each physical shard. By using the index shard number calculation method and index shard placement strategy, the stability of the search engine and the utilization rate of search cluster hardware resources are improved.

[0082] Furthermore, in this embodiment, the index created based on document data is an inverted index, which includes a word dictionary and an inverted list; the word dictionary includes all words that have appeared in the document data and pointers to the corresponding words in the inverted list; the inverted list includes the document ID of all documents in which each word in the word dictionary has appeared, its position information in the document, and its frequency of occurrence.

[0083] Specifically, in this embodiment, each document data corresponds to a document ID. The inverted index segments each document data according to a specified syntax and then maintains a table listing all words appearing in all documents, along with their document IDs and frequencies. This is a specific storage format for implementing a "word-document matrix." The lexicon is a string collection consisting of all words appearing in the document set. Each index entry in the lexicon records some information about the word itself and a pointer to the inverted list. The posting list records a list of all documents containing a given word and the word's position within each document; each record is called an inverted entry. The inverted lists of all words are typically stored sequentially in a file on disk, called the inverted file, which is the physical file storing the inverted index.

[0084] Preferably, the index shards in this embodiment include primary shards and replica shards; the primary shard is used to build an inverted index for document data and store the data, and write data to the replica shards; the replica shards are used to back up the data sent by the primary shard; the primary shard and the corresponding replica shards are respectively set on different physical nodes. Both the primary shard and the replica shard can perform data retrieval, but data writing can only be handled by the primary shard; the collaborative work of the primary shard and the replica shards ensures the integrity and consistency of the data.

[0085] Furthermore, the search calculation module in this embodiment is used to perform distributed index calculation and provides functions such as keyword retrieval, full-text retrieval, fuzzy search, and precise search.

[0086] Specifically, the search computation process includes two stages: query operation (Query stage) and value retrieval operation (Fetch stage). The query request is broadcast to all relevant shards, their responses are integrated into a globally sorted result set, and then returned to the client to complete the search operation.

[0087] More specifically, during the query operation phase, a physical shard is selected as the coordinating node through round-robin, and a search request is sent. The coordinating node broadcasts the search request to each physical shard. Each index shard performs keyword matching and calculates the matching degree based on its corresponding inverted index data, obtains the matching result, and sends it to the coordinating node. The matching result includes the document ID and matching degree of each matching document. The coordinating node performs a global sort on all received matching results to obtain the overall sorting result for all matching documents.

[0088] During the value retrieval phase: Based on the sorting result and the preset number of search results, the document ID to be returned is determined by the coordinating node, and a get request is sent to the index shard containing the document data. The corresponding index shard retrieves the corresponding document data from the physical shard based on the routing formula and returns it to the coordinating node. The coordinating node then returns the searched document data to the client.

[0089] The search mode in this embodiment supports keyword retrieval, full-text search, fuzzy search, and exact search using SQL syntax. Based on the search mode set according to the search computation task, full-text search can be used as a filter condition in SQL, mainly including the following functions:

[0090] It supports simple statistical functions, such as calculating the number of hit results and grouping the retrieved results. Grouping statistics can be achieved by using `GROUP BY` on the grouping field in the `SELECT` statement and then using functions like `COUNT` and `SUM`.

[0091] It supports sorting by relevance, allowing you to sort the results according to the degree of relevance of the search content; it also supports sorting by a specific field, which can be achieved by using the ORDER BY syntax in the query statement.

[0092] It supports phrase search, meaning a phrase must be completely matched, and supports exact matching.

[0093] It supports weighted retrieval, meaning different keywords have different weights, and entries containing keywords with higher weights are returned first. Weighted filtering can be achieved by adding a `WHERE MICE` clause to the `SELECT` statement.

[0094] It supports fuzzy search, allowing for both initial and final fuzzy search, and doesn't require all keywords to be met; it returns results that match only some keywords. Conditional filtering can be achieved by adding a WHERE CONCESS clause to the SELECT statement in conjunction with the fuzzy matching wildcard % character.

[0095] It supports character distance limiting, matching only when each keyword appears within a certain distance in the text. This can be easily achieved by using the `substr` function in the query statement to extract a segment of text and then performing a `where matches` match.

[0096] It supports exporting search results to a file. It also supports integrating search results into the data statistics and analysis modules for further processing and analysis.

[0097] Furthermore, during the process of building the search computing engine, such as Figure 2As shown, data is written through the following steps:

[0098] A piece of data is written through the Java client of the search cluster. The Java client obtains the list of physical nodes according to the configured search cluster information, encapsulates the user input data and index information into a TCP request, selects an index shard as the coordinating node through round-robin, and sends a write request to the coordinating node.

[0099] The coordinating node loads the corresponding index metadata according to the index information, and the index metadata includes the index mapping structure information;

[0100] The coordinating node checks whether the incoming data specifies a doc Id. If not, it generates a doc Id for the data. It also checks whether a routing value is specified. If not, it uses the doc Id as the routing value for the document data and calculates the corresponding physical fragment number using the routing formula.

[0101] The coordinating node obtains the primary shard information corresponding to the index and the IP address information of the corresponding primary shard node, and sends a remote request to the physical node corresponding to the IP address;

[0102] After the physical node receives the data, it first performs a content routing write consistency check. If the check passes, it performs a write sharding operation.

[0103] After the write operation is complete, write requests are sent to the replica shards in a loop. Once the replica shard is successfully written, a write success message is returned to the client through the coordinating node.

[0104] More specifically, the index building process of the search computing engine involves four distinct objects: the client, the coordinating node, the primary shard node, and the replica node. The client can be a browser, a Java client, etc. The coordinating node is any node in the search cluster, primarily responsible for data processing and request forwarding. The primary shard node is mainly responsible for building inverted indexes, data storage, and writing data to replica shards. The replica shards primarily serve as backups of the data sent from the primary shard, providing data reliability. Simultaneously, the replica shards also alleviate the data reading pressure on the primary shard node through their data retrieval capabilities.

[0105] Furthermore, to improve the performance of the search computing engine, this embodiment also includes a memory computing management module, a spatiotemporal query module, and an operator pushdown module; among which,

[0106] The memory computing management module is designed for distributed memory computing. It uses off-heap memory services to improve system robustness, avoids the impact of JVM garbage collection mechanism on the system, and thus improves the stability of massive data retrieval and single-machine data storage capacity.

[0107] Specifically, after making a search request, the search engine needs to load index data into memory for search computation. Current methods default to storing the data loaded into memory in the JVM's heap storage space, thus avoiding the performance overhead of Java object serialization or outputting the JVM heap. However, when the amount of search computation data is large, the heap storage occupied by search computation increases. The JVM's garbage collection (GC) mechanism, scanning all Java objects in the JVM heap storage and frequently reclaiming space, can significantly impact the overall search performance of the system. The main reason is that before Java removes old objects to make room for new objects, it needs to scan all Java objects to find those that are no longer in use. This can result in scanning and releasing a large number of memory dataset objects, even tens of gigabytes of storage space. This GC problem is exacerbated by the interference of multiple tasks running simultaneously. This lengthy GC process is difficult for application developers to control and resolve, which is detrimental to using the search engine for large-scale data computation and significantly reduces the engine's performance.

[0108] The memory computing management module in this embodiment is based on a distributed memory computing design and employs off-heap memory optimization technology to effectively avoid the aforementioned problems. By allocating data in memory outside the Java Virtual Machine's heap, it is directly managed by the operating system (rather than the virtual machine), thus reducing the impact of garbage collection on the application to a certain extent. This embodiment uses off-heap memory technology to reduce garbage collection, speed up copying, and improve single-machine data storage capacity.

[0109] Furthermore, to expand the application scope of the search computing engine in this embodiment, a spatiotemporal query module is also included. This module introduces an R-tree as a spatial index and integrates various analysis and computing algorithms to provide functions such as query analysis based on spatiotemporal data, trajectory clustering, and similarity analysis. It enables searches for geographic coordinates, polygon-based streets, and buildings, and can be used in typical map search domains. By introducing an R-tree as a spatial index, the inaccurate graph relationship query determination problem caused by Geohash / Quadtree is avoided, while also improving the writing performance of complex graphs.

[0110] Furthermore, to reduce I / O operations and thus improve search engine performance, this embodiment also includes an operator pushdown module for local data preprocessing through each physical shard, thereby reducing data read / write operations involved in engine-level computation. The operator pushdown module includes predicate logic optimization information pushdown, aggregation optimization information pushdown design, and other optimization information pushdown designs, enabling the computing engine to fully utilize its storage characteristics and significantly improve retrieval efficiency.

[0111] Furthermore, the graph computing engine, based on the Cypher compiler, performs syntactic and semantic interpretation of OpenCypher operation commands in graph computing tasks. It then compiles the interpreted OpenCypher operation commands into a distributed logical execution plan and generates a physical execution plan for execution in a distributed environment, enabling real-time graph querying and offline graph analysis. The graph computing engine employs a multi-Master-multi-Worker approach to construct a distributed graph database system and integrates various distributed graph computing algorithms and deep learning graph algorithms to build a graph mining model, providing users with real-time graph querying and offline graph analysis services. The graph algorithms supported in this embodiment include: StarNet, PageRank, Strong Connected Component, Label Propagation, K-core, Bow Tie, GraphCentrality, Fraud Rank, Heavy Edge Detector, and Motif Finding.

[0112] Specifically, in this embodiment, multiple Masters form a Master Group, responsible for metadata management, task scheduling, load balancing, and other functions. Workers, acting as the actual storage agents for the graph data, provide data processing operations, including reading, updating (including writing), and deleting graph data. The storage engine uses the Raft protocol to ensure data consistency and high availability.

[0113] To ensure the fault tolerance and high availability of a distributed system, additional design is required for both the Master and Workers. For the Master, since the data is reported by the Workers, a high availability group consisting of multiple Master processes is sufficient. However, for the Workers that provide data read / write services, process, disk, or server failures can prevent graph data from being read or written, leading to graph data unavailability.

[0114] In this embodiment, during data processing, the graph data is divided into multiple partitions, with each Worker's smallest logical storage unit being a partition. For each partition, several (three or more) Workers are selected as hosts; data consistency among multiple replicas of a partition is managed using the Raft protocol, thereby solving the aforementioned problem.

[0115] Furthermore, the stream computing engine in this embodiment is used to schedule tasks based on the task information of each node in the directed acyclic graph and the resource information of the corresponding physical computing nodes, so as to achieve low-latency processing of stream data.

[0116] Specifically, the stream computing engine includes a control node module, a compute node module, and a Zookeeper cluster module.

[0117] The compute node module includes multiple physical compute nodes, which are used to monitor and execute corresponding stream computing tasks;

[0118] The Zookeeper cluster module is deployed on multiple servers to store all status and task information of multiple physical computing nodes, so that the computing node module and control node module can perform real-time monitoring and calls.

[0119] The control node module is used to generate a directed acyclic graph based on the information of multiple tasks to be executed in the stream processing task and the resource information of each physical computing node; and to distribute the tasks to be executed to the corresponding physical computing nodes for processing according to the correspondence in the directed acyclic graph, and to schedule tasks based on the resource information of each physical computing node to achieve low-latency processing of stream data.

[0120] Specifically, the control node module is the central hub of the stream computing engine, configured with a master control node and slave control nodes. When the stream computing engine starts, both the master and slave control nodes are initialized. The control node that starts later sends registration information to the control node that started earlier and establishes a heartbeat keep-alive connection between the control nodes. Under normal working conditions, the master control node is in working mode and is used to execute control tasks. The slave control node is in standby mode as a backup node of the master control node. When the master control node fails, it takes over the control tasks of the master control node based on the heartbeat keep-alive connection to ensure the continuous execution of stream processing tasks and improve the reliability of the system.

[0121] In stream computing, tasks are the basic unit of scheduling. Different tasks are usually subject to priority constraints, with higher-priority tasks being executed before lower-priority tasks. The goal of task scheduling in this embodiment is to allocate and execute tasks under fixed hardware resources and task priority constraints, thereby reducing the total time required to complete tasks.

[0122] Preferably, the aforementioned task scheduling based on the resource information of each physical computing node includes:

[0123] The system obtains the resource utilization of each physical computing node, including CPU utilization, memory utilization, and disk I / O utilization. It calculates the proportion of each of these components in the total resource utilization. If the proportion of any one of these components exceeds a preset threshold, task scheduling for the tasks to be executed on that physical computing node is determined. Based on the resource information of each physical computing node, the system calculates the task scheduling priority for each physical computing node and schedules the tasks to be scheduled based on the physical computing node with the highest task scheduling priority.

[0124] The task scheduling priority of each physical computing node is calculated using the following method: based on the resource utilization rate of each physical computing node, the remaining rate of each resource of each physical computing node is obtained; based on the remaining rate of each resource of each physical computing node and the amount of resources required by the task to be scheduled, the task priority contribution of each resource is obtained; based on the task priority contribution of each resource of each physical computing node, the task scheduling priority of the task to be scheduled corresponding to each physical computing node is obtained.

[0125] Specifically, the resource requirements of the tasks to be scheduled are obtained using the following methods:

[0126] Run the stream computing task independently on the physical computing node, and record the CPU idle time t1 and running time t2 respectively; obtain the CPU resource amount taskcpu required for the stream computing task using the following formula:

[0127] taskcpu=1-P=1-t1 / (t2*Q);

[0128] Where P is the CPU idle rate when running a task alone, and Q is the number of CPUs;

[0129] The amount of memory resources (taskmem) and disk I / O resources (taskio) required for stream computing tasks were obtained by using memory and disk I / O statistics tools provided by the corresponding physical computing nodes.

[0130] This embodiment performs task scheduling based on the resource information of each physical computing node. Tasks from physical computing nodes with high loads that affect task processing speed are scheduled to relatively idle physical computing nodes for processing, which greatly improves the speed of streaming data processing and solves the problems of high latency and poor real-time performance caused by hardware resource limitations.

[0131] More specifically, since different tasks require different resources—compute-intensive tasks need more CPU resources, and I / O-intensive tasks need higher I / O efficiency—scheduling tasks based on the impact of different resources on scheduling can better solve the bottleneck problem in streaming data processing. In other words, a task's resource consumption is multifaceted; therefore, a mapping can be established between the load capacity of physical computing nodes and resource usage across different dimensions, thereby enabling task scheduling for each node.

[0132] Factors affecting node load generally include CPU utilization, memory utilization, and I / O efficiency, among which:

[0133] The CPU is a crucial resource for computer computing and has a significant impact on task scheduling. This embodiment uses the following method to calculate CPU utilization:

[0134] By examining both CPU runtime and idle time, and using the time difference between two consecutive CPU runs as a calculation cycle to calculate CPU runtime, and similarly calculating CPU idle time, the CPU utilization can be obtained based on both idle and runtime. cpu ;

[0135] Memory usage mem You can use the commands provided by the Linux system to check memory usage. Disk I / O utilization can also be monitored. IO Similar to memory usage, statistics are also collected using the system's I / O statistics tools.

[0136] In this embodiment, the control node module starts a thread specifically to monitor the running status of each task node. Under normal circumstances, the computing nodes are running stably, resource usage does not exceed thresholds, and data flow is stable, so periodic task scheduling is not required. Task scheduling is only necessary when some computing nodes fail or the load exceeds the threshold, resulting in significant fluctuations.

[0137] In distributed stream computing platforms, task allocation and scheduling are based on node load information. Therefore, analyzing node load is crucial to node operating efficiency. The relationship between the load and resources of physical computing nodes is as follows:

[0138] L = Use cpu +Use mem +Use IO ;

[0139] Among them, Use cpu Represents CPU utilization, Use mem Indicates memory usage, Use IOThis represents disk I / O utilization. In practical applications, a certain time interval can be set according to the actual situation to periodically collect information on various resources, obtain the load status of physical computing nodes, and calculate the proportion of each resource's occupancy, as shown in the following formula:

[0140] C cpu =ΔUsecpu / ΔL;

[0141] C mem =ΔUsemem / ΔL;

[0142] C IO =ΔUseIO / ΔL;

[0143] Wherein, ΔUsecpu, ΔUsemem, and ΔUseIO are the differences in CPU utilization, memory occupancy, and disk I / O utilization between two adjacent data collection cycles, respectively, and ΔL is the difference in total load between two adjacent data collection cycles.

[0144] Through C cpu C mem C IO This allows us to identify the factors that have the greatest impact on the load, and also reveals which factors are likely to become the bottleneck of a node. Extensive experiments have shown that when C... cpu C mem C IO When the maximum fluctuation in C exceeds 30%, it indicates that large fluctuations may occur. Therefore, this embodiment sets the threshold to 30%, that is, when C... cpu C mem C IO If any of the following exceeds the 30% threshold, task scheduling will be performed on the task corresponding to that physical computing node. Similarly, when C... cpu C mem C IO A smaller value indicates that the resources in this area are relatively idle. If such resources are needed, the corresponding tasks should be scheduled to this node first.

[0145] More specifically, based on the difference between the remaining resource rates of CPU, memory, and disk I / O interfaces and the amount of resources required by the task, the task priority contribution of CPU, memory, and disk I / O is obtained using the following formula:

[0146]

[0147]

[0148]

[0149] Where fcpu, fmem, and fio represent the task priority contributions of CPU, memory, and disk I / O, respectively; Δcpu, Δmem, and Δio represent the difference between the remaining resource rates of CPU, memory, and disk I / O and the amount of each resource required by the task; taskcpu represents the amount of CPU resources required by the task, taskm represents the amount of memory resources required by the task, taskio represents the amount of disk I / O resources required by the task, and qos represents the amount of qos. cpu qos mem and qos io These represent the remaining resource rates of CPU, memory, and disk I / O interfaces, respectively, with α, β, and γ as weighting factors.

[0150] Preferred, qos cpu qos cpu and qos cpu It is obtained through the following formula:

[0151] qos cpu =1–Use cpu ;

[0152] qos mem =1–Use mem ;

[0153] qos io =1–Use io ;

[0154] The differences between the CPU, memory, and disk I / O availability and the amount of CPU, memory, and disk I / O resources required by the task, Δcpu, Δmem, and Δio, are obtained using the following formulas:

[0155] Δcpu = qos cpu -task cpu ;

[0156] Δmem=qos mem -task mem ;

[0157] Δio=qos io -task io ;

[0158] Finally, the task scheduling priority corresponding to the physical computing node is obtained using the following formula:

[0159] Rank(task)=ρ+R(task)*fcpu*fmem*fio*v;

[0160] Where Rank(task) is the task scheduling priority; fcpu, fmem, and fio are the task priority contributions of CPU, memory, and disk I / O, respectively; and ρ is the compensation factor. R(task) represents the distance between the task at this node and the final task in the scheduling sequence; v is the task processing speed influence factor, v = (1-s) / t; s is the ratio of the task output data to the input data; and t is the time the task has been processed.

[0161] The higher the task scheduling priority of a physical computing node, the more efficient it is in processing the corresponding scheduled task. Therefore, after calculating the task scheduling priority, the scheduled task is scheduled to the physical computing node with the highest priority for computation and processing, so as to improve the system's operating efficiency and reduce the latency of data processing.

[0162] It should be noted that, taking fcpu as an example, when Δcpu>0, it means that the remaining CPU of the physical computing node is more than the CPU required by the task. fcpu has a positive correlation with the task scheduling priority Rank(task), and the positive effect is related to the weight factor α. The values ​​of α, β, and γ are all between 0 and 1 and can be set according to the actual application. When Δcpu<0, it means that the CPU demand of the task is greater than the current node's CPU remaining rate. According to the formula for fcpu, when Δcpu<0, fcpu has a negative correlation with the task scheduling priority Rank(task). For a node, the more input data a task processes per unit time, the less memory it consumes. This embodiment introduces a task processing speed influence factor v, defined as v = (1-s) / t, which takes into account the influence of task processing speed and improves the effectiveness of the task scheduling algorithm. In addition, this embodiment introduces a compensation factor, ρ, defined as: Compensation is provided for running tasks, increasing their scheduling priority, ensuring the continuity of task execution and the correctness and effectiveness of the scheduling algorithm, and reducing system latency.

[0163] Furthermore, such as Figure 3 As shown, the working process of the stream computing engine includes:

[0164] The master control node and the secondary control node are initialized separately. The initialization of the master control node and the secondary control node is sequential, and the control node that starts later sends registration information to the control node that starts earlier and establishes a heartbeat keep-alive connection between the control nodes.

[0165] The Tracker monitoring process for each physical computing node in the computing node module initializes upon startup.

[0166] The compute node monitoring process obtains metadata such as hardware resource information of the physical computer through the Zookeeper cluster module, registers with the master control node, and reports the hardware resource information.

[0167] Users send a start task command to the master control node through the Portal process in the management interface.

[0168] The master control node parses StreamSQL to obtain the static topology graph (static directed acyclic graph, StaticDAG) of the task; then, based on cluster resources and the status of running tasks, it schedules the task to obtain the dynamic topology graph (dynamic directed acyclic graph, DynamicDAG) of the task.

[0169] The master control node sends the execution instructions to the corresponding physical computing nodes according to the correspondence in the dynamic topology diagram;

[0170] The compute node monitoring process Tracker creates worker processes based on the instructions from the control node, which then run the specified stream processing tasks.

[0171] The compute node monitoring process Tracker replies to the master control node regarding the creation status of worker processes;

[0172] After the master node waits for all the computing units in the task topology graph to complete the creation of their working processes, it sends instructions to the computing node monitoring process to connect to the downstream computing units according to the task topology graph relationship.

[0173] The Tracker process of the compute node forwards the instructions to connect to the downstream to the Worker process, and the Worker process establishes a network connection with the Worker process of the downstream compute node.

[0174] The Tracker process, which monitors the compute nodes, sends the execution results of the downstream connections of the Worker processes back to the Master control node.

[0175] Once the topology of the waiting tasks is fully established, the master node will send a response to the management interface regarding the task startup and execution results.

[0176] Furthermore, the storage management module manages data storage for the distributed computing engine through data table management, data synchronization management, data source adapters, and unified storage management operations.

[0177] Data table management enables connections between different data storage systems, such as TEXT tables, ORC tables, transaction tables, and row-based tables. Data synchronization management defines data synchronization strategies and provides corresponding implementations to ensure data is synchronized across different data storage systems according to the defined strategies. Data source adapters prioritize data insertion based on the most frequent data table usage scenario during data writing, followed by synchronization. During data reading, they analyze the SQL scenario to determine the appropriate data source for data retrieval. This embodiment improves system stability and resource utilization through unified data storage management.

[0178] In summary, the distributed computing engine of this invention, based on unified search and big data processing, integrates graph computing, stream computing, and search computing engines. When creating an index, the search computing engine calculates the number of physical shards based on the performance of each physical node in the search cluster and the amount of data in the index, and then places the index data in shards. This allows for optimized settings based on user needs and hardware resources, improving hardware resource utilization and the performance of the search computing engine. Furthermore, the stream computing engine calculates the task scheduling priority for each physical computing node based on its hardware resource information, and schedules tasks based on the physical computing node with the highest task scheduling priority. This enables the allocation and execution of stream data processing tasks under fixed hardware resource and task priority constraints, achieving low-latency processing of stream processing tasks and improving system real-time performance. The integrated graph computing engine of this invention uses a multi-Master-multi-Worker approach to construct a distributed graph database system and integrates various distributed graph computing algorithms and deep learning graph algorithms to build a graph mining model, providing users with real-time graph query and offline graph analysis services. By integrating multiple computing engines and making full use of distributed cluster hardware resources, efficient graph computing and streaming data processing are achieved, improving the retrieval efficiency of massive amounts of data.

[0179] Those skilled in the art will understand that all or part of the methods in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0180] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed computing engine based on unified search and big data processing, characterized in that, include: Unified interface module, distributed execution module, and storage management module; The unified interface module is used to receive computing tasks and parse the tasks to start the corresponding computing engine according to the task type obtained from the task parsing. The distributed execution module includes a search computing engine, a graph computing engine, and a stream computing engine, which are used to read and execute corresponding computing tasks, respectively. The search engine calculates the number of physical shards based on the performance of each physical node in the search cluster and the amount of indexed data, and stores the document data accordingly. The number of physical shards is obtained by calculating a preliminary number of physical shards based on the amount of indexed data. SN1 ; ,in, D The amount of indexed data was determined by statistical analysis of the indexes within the cluster; performance verification was performed on each physical node in the cluster to obtain the number of physical nodes that passed the verification. SN2 ;right SN1 and SN2 By performing a weighted summation, the number of shards that meets the requirements of index data volume and node performance can be obtained using the following formula. : ； in, θ For expansion coefficient, e , k These are the weighting coefficients. t For the number of redundant fragments, ; The performance of each physical node in the search cluster is verified to obtain the number of physical nodes that pass the verification. SN2 This includes: verifying the disk utilization rate of each physical node and the number of existing physical shards on each physical node; if the disk utilization rate Dsri of the physical node... If the physical node's physical shard count does not exceed 90% and the number of existing physical shards does not exceed a preset threshold, then the physical node is deemed to have passed the verification. If the physical node passes the verification, the corresponding verification result will be... nodeArr i Set the element to 1 if it is not valid, otherwise set it to 0, and store the verification result in the array nodeArr[]. The number of physical nodes that pass the verification is obtained by the following formula. SN2 , ; If the number of physical fragments is obtained If the number of physical shards exceeds the currently available number of physical nodes, the upper limit x for the number of physical shards on the same physical node is adjusted using the following formula: ； like If the number of physical nodes does not exceed the number of currently available physical nodes, then the number of physical shards is set according to the number of currently available physical nodes, that is, each physical node is set to one physical shard; The storage management module is used to perform unified data table and storage management for the data corresponding to the search computing engine, stream computing engine and graph computing engine.

2. The distributed computing engine based on unified search and big data processing according to claim 1, characterized in that, The graph computing engine adopts a multi-Master-multi-Worker approach to build a distributed graph database system; and integrates various distributed graph computing algorithms and deep learning graph algorithms to construct a graph mining model, providing users with real-time graph query and offline graph analysis services. The stream computing engine is used to schedule tasks based on the task information of each node in the directed acyclic graph and the resource information of the corresponding physical computing nodes, so as to achieve low-latency processing of streaming data.

3. The distributed computing engine based on unified search and big data processing according to claim 1, characterized in that, The search computing engine includes an index storage management module and a search computing module, wherein: The index storage management module is used to create an index based on document data, and divide the index into multiple index shards and store them on multiple physical shards. This includes placing the index shards obtained from the index storage management module according to a preset index shard placement strategy. Each physical shard also stores the document data corresponding to that index shard. The number of physical shards is calculated based on the performance of each physical node in the search cluster and the amount of data in the index. The search calculation module is used to perform keyword matching and matching degree calculation in each index shard based on the search keywords input by the client, and obtain the document IDs of multiple matching documents through query operations; and obtain the matching documents from the corresponding physical shards through value retrieval operations based on the document IDs and routing formulas.

4. The distributed computing engine based on unified search and big data processing according to claim 1, characterized in that, The index created based on document data is an inverted index, which includes a word dictionary and an inverted list. The word dictionary includes all words that have appeared in the document data and pointers to the corresponding words in the inverted list; The inverted list includes the document ID of all documents in which each word appears in the word dictionary, its position in the document, and its frequency of occurrence.

5. The distributed computing engine based on unified search and big data processing according to claim 3, characterized in that, The query operation includes: selecting an index shard as a coordinating node through round-robin, and sending a search request; the coordinating node broadcasting the search request to each index shard; each index shard performing keyword matching and calculating the matching degree based on its corresponding inverted index data, obtaining a matching result and sending it to the coordinating node, the matching result including the document ID and matching degree of each matching document; and the coordinating node performing a global sort on all received matching results to obtain the overall sorting result for all matching documents. The value retrieval operation includes: based on the sorting result and the preset number of search results, determining the document ID to be returned through the coordination node, sending a get request to the index shard containing the document data, the corresponding index shard obtaining the corresponding document data from the physical shard based on the routing formula and returning it to the coordination node, and then returning the searched document data to the client through the coordination node.

6. The distributed computing engine based on unified search and big data processing according to claim 3, characterized in that, The preset index sharding placement strategy includes: The performance of cluster nodes is evaluated using a linear weighted method and the following formula: ； in, Qi The value represents the performance value of node i; LAi SNi represents the average load of node i; SNi represents the number of shards on node i; DsRi represents the disk utilization of node i; a represents the first weighting coefficient; s represents the second weighting coefficient; b represents the third weighting coefficient. Based on the number of index shards The index shards are placed in descending order of the performance values ​​of each physical node.

7. The distributed computing engine based on unified search and big data processing according to claim 1, characterized in that, The search calculation engine writes data through the following steps: The Java client obtains a list of physical nodes based on the configured search cluster information, encapsulates the user-input data and index information into a TCP request, selects an index shard as a coordinating node through round-robin, and sends a write request to the coordinating node. The coordinating node loads the corresponding index metadata according to the index information, and the index metadata includes the index mapping structure information; The coordinating node checks whether the incoming data specifies a doc Id. If not, it generates a doc Id for the data. It also checks whether a routing value is specified. If not, it uses the doc Id as the routing value for the document data and calculates the corresponding physical fragment number using the routing formula. The coordinating node obtains the primary shard information corresponding to the index and the IP address information of the corresponding primary shard node, and sends a remote request to the physical node corresponding to the IP address; After the physical node receives the data, it first performs a content routing write consistency check. If the check passes, it performs a write sharding operation. After the write operation is complete, write requests are sent to the replica shards in a loop. Once the replica shard is successfully written, a write success message is returned to the client through the coordinating node.

Images