Full path lookup method, apparatus, device, and medium

Abstract

The application discloses a full-path searching method and device, equipment and medium, and relates to the technical field of graph computing. The method comprises the following steps: receiving a path searching request; the path searching request comprises a graph, a starting point and an ending point in the graph, and a total path step number; in response to the path searching request, the backtracking step number is determined according to the total path step number; the graph is backtracked from the ending point, the reachable node set information in the backtracking process of each level is recorded; the path is searched from the starting point in parallel, and the recorded reachable node set information is used for pruning in the searching process; and the processed path is saved to a result queue. In this way, the overall searching efficiency can be improved, the memory occupation can be effectively controlled, the resource overconsumption can be prevented, all paths between the specified starting point and ending point in the graph within the total path step number can be quickly and accurately found, and the processing speed and the rationality of resource utilization are considered while the result integrity is ensured.

Description

Technical Field

[0001] This invention relates to the field of graph computing technology, and in particular to a method, apparatus, device and medium for finding a complete path. Background Technology

[0002] Traditional full path search algorithms are mainly divided into two categories: one is the pure brute-force search method, which involves a lot of redundant calculations and is prone to problems such as memory overflow or computation timeout; the other is the bidirectional search strategy based on the start point and the destination, which generates a lot of duplicate paths in the process of path combination, resulting in excessive memory consumption, which in turn wastes computing resources and reduces search efficiency. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, device, and medium for full-path search, which can improve overall search efficiency, effectively control memory usage, prevent excessive resource consumption, and balance processing speed with the rationality of resource utilization.

[0004] To address the aforementioned technical problems, this invention provides a full-path search method, comprising:

[0005] Receive a path lookup request; the path lookup request includes a graph, a start point and an end point in the graph, and a set total number of path steps;

[0006] In response to the path lookup request, the number of backtracking steps is determined based on the set total number of path steps;

[0007] Based on the number of backtracking steps, perform graph backtracking preprocessing starting from the endpoint, and record the set of reachable nodes during the backtracking process at each level;

[0008] The search path is parallelized starting from the starting point, and pruning is performed using the recorded reachable node set information during the search process. The processed path is then saved to the result queue.

[0009] To address the aforementioned technical problems, the present invention also provides a full-path search device, comprising:

[0010] A request receiving module is used to receive path lookup requests; the path lookup request includes a graph, a start point and an end point in the graph, and a set total number of path steps;

[0011] The step count determination module is used to determine the backtracking steps based on the set total path steps in response to the path search request;

[0012] The information recording module is used to perform graph backtracking preprocessing starting from the endpoint based on the backtracking step number, and record the set of reachable nodes during the backtracking process at each level;

[0013] The path search module is used to search for paths in parallel from the starting point. During the search process, it uses the recorded reachable node set information to perform pruning and saves the processed path to the result queue.

[0014] To address the aforementioned technical problems, the present invention also provides an electronic device, comprising:

[0015] Memory, used to store computer programs;

[0016] A processor is used to implement the steps of the full path lookup method described above when executing the computer program.

[0017] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described full path lookup method.

[0018] As can be seen from the above technical solution, the present invention provides a full path search method, which includes: receiving a path search request; the path search request includes a graph, a starting point and an ending point in the graph, and a set total number of path steps; in response to the path search request, determining the number of backtracking steps according to the set total number of path steps; performing backtracking preprocessing of the graph starting from the ending point, and recording the reachable node set information of each level of backtracking process; parallelizing the path search starting from the starting point, and simultaneously performing pruning processing using the recorded reachable node set information during the search process, and saving the processed path to the result queue.

[0019] The beneficial effects of this invention are as follows: The full path search method provided by this invention, upon receiving and responding to a path search request, records the set of reachable nodes at each level through backtracking preprocessing starting from the endpoint, based on a determined number of backtracking steps. This provides a precise pruning basis for the search process, directly filtering out path extensions that cannot reach the endpoint, avoiding a large amount of invalid computation and redundant path permutations, and eliminating the need for additional deduplication operations. Simultaneously, the parallel search method starting from the starting point fully utilizes processor resources, improving overall search efficiency and avoiding the performance limitations of single-threaded processing. Throughout the process, targeted pruning of preprocessed information and ordered result queue storage effectively control memory usage and prevent excessive resource consumption. Ultimately, it can quickly and accurately find all paths between a specified starting point and endpoint within a set total number of path steps, ensuring result integrity while balancing processing speed and resource utilization.

[0020] In addition, the present invention also provides a corresponding full-path search device, electronic device and computer-readable storage medium for the full-path search method, which have the same or corresponding technical features as the full-path search method mentioned above, and have the same effect. Attached Figure Description

[0021] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of the full path search method provided in the embodiments of the present invention;

[0023] Figure 2 A schematic diagram illustrating the framework of the full-path search method provided in this embodiment of the invention;

[0024] Figure 3 This is a schematic diagram of the structure of the full path search device provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0026] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The specific application environment architecture or specific hardware architecture on which the execution of the full path lookup method depends is described here.

[0029] The embodiments of the present invention provide a full path search method, and the method is described in detail in conjunction with the execution flow of the full path search method. Figure 1 A flowchart of the full path search method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the method includes:

[0030] S101, Receive path lookup request; the path lookup request includes a graph, the starting point and the ending point in the graph, and the total number of path steps.

[0031] In implementation, receiving a pathfinding request is the trigger point for the entire process of this invention. The content of this request defines the specific scope and constraints of the pathfinding: the graph is the basic carrier of the path, providing the topological structure composed of nodes and edges for the search; the start and end points clearly define the starting and ending boundaries of the path, limiting the search target to the path connecting these two specific nodes; setting the total number of path steps constrains the path in terms of length, specifying the range of steps the searched path must meet, ensuring that the result conforms to the preset length requirements. These elements together constitute the prerequisites for pathfinding, providing a clear basis for subsequent preprocessing, searching, and other operations.

[0032] S102. In response to the path lookup request, determine the number of backtracking steps based on the set total number of path steps.

[0033] In implementation, upon receiving a path lookup request, this invention can respond to the request and enter the specific processing stage. The total number of path steps is a path length constraint set by the user, and the determination of the backtracking steps must be based on this. There is a corresponding relationship between the backtracking steps and the total number of path steps. For example, the backtracking steps are half of the total number of path steps; that is, when the total number of path steps is set to N, the backtracking steps B can be defined as B = N / 2. This correspondence defines the range for subsequent backtracking preprocessing starting from the endpoint, ensuring that the preprocessing process matches the path length constraint required by the user.

[0034] S103. Based on the number of backtracking steps, perform backtracking preprocessing of the graph starting from the endpoint, and record the set of reachable nodes during the backtracking process at each level.

[0035] In implementation, this invention uses the number of backtracking steps as a basis to perform backtracking preprocessing of the graph starting from the endpoint. This is a reverse traversal of the graph structure with the endpoint as the source. This reverse tracing differs from forward search, as it can pre-identify the range of nodes reachable from the endpoint at each level. Recording the set of reachable nodes during the backtracking process at each level is the core objective of backtracking preprocessing. By storing these node sets hierarchically, it provides a crucial reference for subsequent forward search: the set of reachable nodes at each level marks the valid nodes that can ultimately lead to the endpoint at that stage. This serves as the pruning basis for determining whether a path has continuation value during the search process, reducing the expansion of invalid paths from the source and improving overall search efficiency.

[0036] S104. Starting from the starting point, the search path is parallelized. At the same time, the recorded reachable node set information is used for pruning during the search process, and the processed path is saved to the result queue.

[0037] In implementation, this invention expands the path from the starting point and improves search efficiency through multi-threaded parallel processing, making full use of computing resources to accelerate the search. During the search process, pruning is performed using the recorded reachable node set information. This involves calling the reachable node set recorded hierarchically in the preprocessing stage to determine the validity of each path to be expanded: if a candidate node is not in the reachable set at the corresponding level, the path expansion is terminated, thus filtering out a large number of invalid paths that cannot reach the destination and avoiding redundant calculations. Saving the processed paths to the result queue completes the temporary storage of valid paths, ensuring that all complete paths meeting the conditions (such as step constraints and reachable destination) are collected in an orderly manner, providing a foundation for subsequent persistent storage. This ensures the integrity of the results and balances efficiency and accuracy through pruning and parallel mechanisms.

[0038] It should be noted that the full path search method involves the Depth-First Search (DFS) algorithm. Traditional DFS algorithms can be broadly categorized into: pure brute-force search methods, which involve a large amount of redundant computation and are prone to memory overflow or computation timeouts; and bidirectional search strategies based on start and end points, which generate a large number of duplicate paths during path combination, leading to excessive memory consumption, wasted computational resources, and decreased search efficiency. The full path search method provided in this invention can solve this technical problem.

[0039] In the full path search method provided by this invention, after receiving and responding to a path search request, based on a determined number of backtracking steps, the method records the set of reachable nodes at each level through backtracking preprocessing starting from the endpoint. This provides a precise pruning basis for the search process, directly filtering out path extensions that cannot reach the endpoint, avoiding a large amount of invalid computation and redundant path permutations and combinations, and eliminating the need for additional deduplication operations. Simultaneously, a parallel search method is adopted starting from the starting point, fully utilizing processor resources and improving overall search efficiency, avoiding the performance limitations of single-threaded processing. Throughout the process, targeted pruning of preprocessed information and ordered result queue storage effectively control memory usage and prevent excessive resource consumption. Ultimately, it can quickly and accurately find all paths between a specified starting point and endpoint within a set total number of path steps, ensuring result integrity while balancing processing speed and resource utilization.

[0040] Furthermore, in a specific implementation, in the full path search method provided in the embodiments of the present invention, step S103 performs graph backtracking preprocessing starting from the endpoint based on the backtracking step number to generate a record array. Specifically, this may include: constructing an array for recording the set of reachable nodes during the backtracking process, as the record array; the size of the record array is the backtracking step number plus 1, and each element of the record array is a bit set structure used to represent all reachable nodes at the current level; the bit set structure at all levels is written with the encoding of the endpoint; initializing the backtracking queue; the element type of the backtracking queue is an array structure; the first element of the array structure represents the current backtracking point, and the second element represents the backtracking level; processing the tasks in the backtracking queue in parallel, recording the information of the set of reachable nodes during the backtracking process in the record array, until the backtracking queue is cleared and all threads are in an idle state.

[0041] In implementation, assuming the backtracking steps are B, the first step is array initialization: an array named `recordArray` is constructed to record the set of reachable nodes during the backtracking process. This array has a size of B+1, including the starting level 0. Each element in the array is a bitset, representing all reachable nodes at the current level. During initialization, the endpoint's index is written into the bitsets of all levels, indicating that the endpoint is considered reachable in the initial state at all levels. For example, if the total number of nodes is 100, each bitset will have 100 bits, and initially, the bit corresponding to the endpoint in each bitset will be set to 1.

[0042] Next, the backtracking queue is initialized. The elements of the backtracking queue are arrays, where the first element represents the current backtracking point and the second element represents the backtracking level. During initialization, the first element is added to the backtracking queue, which is [end point, 0].

[0043] Next, a multi-threaded backtracking task is initiated, which processes the tasks in the queue in parallel by multiple threads, thereby accelerating the backtracking process. Each thread will repeatedly execute a series of operations until the entire backtracking queue is cleared and all threads are in an idle state.

[0044] Further, in specific implementation, in the above steps, tasks in the backtracking queue are processed in parallel, and relevant data of the reachable node set during the backtracking process is recorded in the record array until the backtracking queue is emptied and all threads are in an idle state. Specifically, it may include: starting multi-threaded parallel processing of tasks in the backtracking queue; each thread takes out an element from the backtracking queue, calculates the next level as the current level plus 1, obtains the previous neighbors of the current node, and writes the previous neighbors into all bit-set structures from the next level index to the backtracking step number in the record array; if the next level is less than the backtracking step number, add the previous node and the next level to the backtracking queue; when the backtracking queue is empty and the threads are in an idle state, the backtracking preprocessing is completed.

[0045] In implementation, first take out an element [current node (currentNode), current level (currentLevel)] from the queue. Then calculate the next level (nextLevel), that is, nextLevel = currentLevel + 1. After that, obtain the list of all previous neighbor nodes (prevNeighbors) of the current node (currentNode). For each previous node (prev): write the previous node into all bit-set structures from index nextLevel to B in the record array; if nextLevel < B, add [prev, nextLevel] to the queue to continue backward backtracking. When the queue is in an empty state and all threads are not in an execution state, it is considered that the backtracking preprocessing is completed.

[0046] Suppose the total number of steps N = 6, then according to B = N / 2, it can be known that B = 3; the end node number is 95, and the connectivity relationship between nodes in the graph is as follows: 92 → 93 → 94 → 95, 88 → 89 → 90 → 91 → 95.

[0047] The execution process may include: initializing recordArray[0...3], and setting 95 as reachable for all bit-set structures; initializing the backtracking queue as [[95,0]]. Subsequently, the thread takes out [95,0] from the queue, obtains its previous neighbors 94 and 91, writes 94 and 91 into recordArray[1...3], and adds [94,1] and [91,1] to the queue. Then, the thread continues to take out [94,1] from the queue, obtains its previous neighbor 93, writes 93 into recordArray[2...3], and adds [93,2] to the queue. Execute repeatedly in this way until the queue is emptied and no thread is running, and the backtracking preprocessing is completed.

[0048] Furthermore, in a specific implementation, in the full path search method provided in the embodiments of the present invention, step S104 parallelizes the path search starting from the starting point, and simultaneously performs pruning processing using the recorded reachable node set information during the search process, saving the processed path to the result queue. Specifically, this may include: creating a result queue for storing paths; the elements of the result queue are a set of points from the starting point to the ending point; initializing the calculation queue; adding paths containing only the starting point to the calculation queue as initial values; parallel processing of paths in the calculation queue; if the length of the current path exceeds the maximum step threshold, discarding the current path; if the length of the current path does not exceed the maximum step threshold, taking the last point of the current path and obtaining the subsequent neighbor nodes of the last point; for all obtained subsequent neighbor nodes, performing pruning processing using the recorded reachable node set information, and saving the processed path to the result queue.

[0049] In implementation, firstly, a thread-safe result queue (resultQueue) can be created, which is specifically used to store the finally successfully generated paths. The elements in the queue are a collection of points, which can be either an array or a linked list. Each element represents a complete path from the starting point to the ending point. For example, the path [10,12,15,20,95] is a specific example of such an element.

[0050] Next, initialize a compute queue (computeQueue). The elements of this queue are also a collection of points, used to represent the path currently being traversed. During initialization, an initial value needs to be added to this queue, that is, the path containing only the starting point, specifically computeQueue.push([startId]).

[0051] Next, the computational logic phase begins, employing a multi-threaded approach to acquire and execute tasks. Each thread retrieves a task, or path, from the computation queue and processes it as follows: The retrieved path set is denoted as a list; first, a termination condition is checked: if the length of the list exceeds the maximum number of steps (maxStep), the path is too long and must be discarded; if the termination condition is not met, processing continues, retrieving the last node of the current path and denoting it as cur, then obtaining all subsequent neighbor nodes of cur, denoted as nexts. Subsequently, pruning logic is applied to nexts, and the resulting path is saved to the result queue.

[0052] Furthermore, in specific implementation, in the above steps, for all acquired subsequent neighbor nodes, pruning is performed using the recorded reachable node set information, and the processed path is saved to the result queue. Specifically, this may include: based on each candidate successor node among all acquired subsequent neighbor nodes, combined with the current level and the recorded reachable node set information, determining whether the candidate successor node is backtracked to by the endpoint at the current level; if not, pruning is performed on the candidate successor node; if so, no pruning is performed on the candidate successor node, and the candidate successor node is appended to the original path to construct a new path; if the candidate successor node is the endpoint, the new path is added to the result queue; if the candidate successor node is not the endpoint, the candidate successor node is returned to the calculation queue to continue expansion; when the calculation queue is empty and the thread is in an idle or sleeping state, the search terminates.

[0053] In implementation, candidate successor nodes refer to all nodes adjacent to the current node that have not been visited. During the pruning judgment process, for each candidate successor node `outId` in `nexts`, a pruning judgment must be performed to ensure that the search process only takes place within the area "theoretically reachable from the destination". The pruning rule can be as follows: calculate the value of `level`, which is equal to the maximum step threshold (`maxStep`) minus the length of the current path `list`, i.e., `level = maxStep - list.length`; if the length of `list` is greater than or equal to `level`, and the result of detecting `outId` in the bit set structure corresponding to the `level` level in the `recordArray` is false, i.e., if `list.length >= level` and `recordArray[level].test(outId) == false`, then the path is pruned, i.e., this path is no longer expanded. The pruning basis here is the reachable node set information recorded in step S103 (i.e., the set of nodes that can be traced back from the destination). If a node cannot be traced back from the destination at this level, then that node can never appear on a valid path.

[0054] For an outId that has not been pruned, a new set of path nodes can be constructed. The new path new_next is constructed by appending the current node outId to the end of the existing path list, that is, new_next = list + [outId].

[0055] Next, determine the destination of the path. If outId equals the destination (endId), the path is complete, and new_next is added to the result queue (resultQueue). If outId is not the destination, the path is incomplete, and new_next is returned to the compute queue (computeQueue) to continue expansion.

[0056] Finally, the search terminates when both the compute queue (computeQueue) and all threads are in an idle or sleeping state (i.e., no tasks are being executed).

[0057] Assuming the starting point is 10 (startId=10), the ending point is 95 (endId=95), and the maximum step threshold is 5 (maxStep=5), the graph structure is 10→12→15→20→94→95, and 12 also points to 18, 15 also points to 22, 20 also points to 91, 22 also points to 90, and subsequent nodes also have their own pointing relationships.

[0058] The execution process may include: initially, the queue contains only [

[10] ]; the thread takes

[10] from the queue, its subsequent node is

[12] , generates a new path [10,12] and adds it to the calculation queue; then the thread takes [10,12], its subsequent node is [15,18], after pruning retains the valid node (e.g., 15), generates a new path [10,12,15]; if the subsequent path expands to [10,12,15,20,94,95], then the path is added to the result queue.

[0059] Furthermore, in a specific implementation, in the full path search method provided in the embodiments of the present invention, after saving the processed path to the result queue in step S104, it may further include: extracting all paths from the result queue and performing structured persistent storage on the extracted paths.

[0060] In practice, after the path search is completed (step S104), the present invention can also extract all valid paths from the result queue in memory and store them in a structured and persistent manner to support subsequent analysis, visualization or system calls.

[0061] Furthermore, in specific implementation, in the above steps, all paths are extracted from the result queue, and the extracted paths are stored in a structured and persistent manner. Specifically, this may include: creating a file or database interface for path persistence; extracting paths one by one from the result queue, and writing the extracted paths into the created file or database interface in a streaming manner with a set format; each path will be a valid path from the start point to the end point, in the format of an ordered set of point numbers.

[0062] In implementation, after completing the path search, the first step in the result saving stage can be initializing the storage mechanism. This step requires creating a file or database interface for path persistence, through which path data storage operations are implemented. Simultaneously, to avoid excessive memory consumption, a streaming write approach can be adopted, such as writing line by line to a text file or inserting data into the database in batches, thus balancing data writing efficiency and memory resource consumption. Various storage types are supported, each suitable for different scenarios: text files (such as .txt or .csv formats) are suitable for simple path storage because their content is intuitive and easy to view and analyze later; binary files offer high-performance read / write capabilities, making them more suitable for handling large-scale path result sets; databases (such as SQLite, PostgreSQL, etc.) support structured queries and retrieval optimization, suitable for scenarios requiring frequent complex queries on path data.

[0063] Next, elements are retrieved from the result queue. From the result queue, which stores the final valid paths, complete paths are extracted one by one through a loop. Specifically, while the result queue is not empty, paths (denoted as `path`) are continuously popped from the queue and saved to the target storage. Here, `path` is a valid path from the starting point to the ending point, formatted as an ordered set of point numbers, ensuring the completeness and validity of each path.

[0064] The structure for writing data is then designed, and the storage format can be flexibly customized according to the actual scenario. The storage format can include the following: The first is CSV text format, which organizes data using two fields: "path_id" and "node_sequence". For example, 1, "10-12-15-20-94-95"; 2, "10-11-16-22-91-95", ..., where path_id is the path identifier, and node_sequence records the node sequence in a hyphenated format. The second is JSON format, which contains multiple objects as a list. Each object contains id (path identifier) ​​and path (an array of node sequences), such as {"id":1,"path":[10,12,15,20,94,95]}; {"id":2,"path":[10,11,16,22,91,95]}. The third is a database table structure, where a table named "path_result" can be created, containing an auto-incrementing id primary key and a pathTEXT field (used to store JSON strings or hyphenated node sequences) to achieve structured storage of path data.

[0065] Finally, result verification and disk write confirmation are performed. To ensure storage integrity, relevant statistical information can be recorded after data writing is complete, such as the total number of paths written and the number of nodes in each path. This information can be used to verify the integrity and accuracy of the stored data later, ensuring that all valid paths have been correctly and completely saved to the target storage.

[0066] It should be noted that, in order to improve processing capabilities and meet the computing needs of complex graph structures or resource-constrained environments, this invention provides an implementation method that supports distributed deployment and expansion. This approach, by introducing a multi-node collaborative computing mechanism, can linearly scale the system's concurrency capabilities while maintaining efficient task distribution and path result processing.

[0067] Figure 2 This is a schematic diagram illustrating the framework of the full-path search method provided in an embodiment of the present invention. Figure 2 As shown, the entire distributed system consists of a master node and multiple worker nodes, which collaborate with each other through network communication and message queue middleware.

[0068] Step one is distributed backtracking preprocessing, executed uniformly by the master node. This step functions identically to the single-machine mode, i.e., backtracking the graph backwards from the endpoint to generate a record array (recordArray) for pruning. Specifically, the master node is responsible for performing all backtracking calculations in this stage. After generating the recordArray, the master node broadcasts it to all worker nodes to support subsequent distributed pruning decisions. For example, after completing a backtracking to depth 3 and generating recordArray[0...3], the master node will send this array to all worker nodes.

[0069] Step two involves distributed parallel path search, scheduled via message queues. The logic is largely the same as the single-machine mode, but with some differences. For queues, message queue middleware such as Kafka, RabbitMQ, and Redis can be used. Each worker node subscribes to tasks (i.e., the current path list) from the message queue. The termination condition is updated to a distributed state determination, requiring all computation queues to be empty and all worker nodes to be in an idle state (no active tasks). To achieve this determination, a coordination mechanism is introduced: the master node maintains the global scheduling state, and each node periodically reports its own running status. When the master node detects that the global state is idle and the queues are empty, it triggers a termination signal. For example, if a worker node completes a path expansion but still has unpruned nodes, it will push the newly generated path (list + outId) into the Kafka queue for other nodes to consume and process.

[0070] Step three involves result storage, which can employ various deployment strategies, including local, centralized, or distributed processing. Specifically, three strategies are available: 1) Local writing, where each worker node writes the results to its local disk; 2) Master node aggregation, where worker nodes send results to the master node for unified storage; and 3) Distributed storage writing, utilizing HDFS, S3, MinIO, or similar technologies for parallel storage. For example, worker nodes can directly write the path [10,12,15,20,94,95] to their local file `results_worker1.csv`, or upload it to object storage at `s3: / / search-results / part-1.json`.

[0071] It should be added that, in order to further improve the controllability and performance of the system, this invention can introduce several practical mechanisms, as follows:

[0072] The first mechanism controls the growth of the computation queue to limit memory usage. The core of this mechanism lies in preventing the computation queue from expanding indefinitely by setting clear boundaries, thus avoiding excessive memory consumption. Specifically, a maximum length can be set for the message queue, and an upper limit can be set for the local cache of each node. When the queue length gradually approaches the set upper limit, threads automatically switch execution modes, no longer pushing tasks into the queue to wait, but directly expanding the computation path—that is, "skipping the enqueue and directly computing." This approach effectively avoids a surge in memory pressure caused by queue congestion. For example, when the Kafka topic backlog exceeds the limit, worker nodes will use their local stacks to directly expand the path instead of pushing tasks back into the Kafka queue, thereby ensuring stable system memory usage.

[0073] Secondly, there is the adaptive mechanism for graph structure loading strategy. This mechanism can flexibly select the most suitable graph structure loading method based on the actual available memory of the system, thereby achieving a balance between memory utilization and computational efficiency. When the system detects sufficient available memory, it will load the entire graph structure into memory, which can greatly reduce disk I / O operations during data reading, significantly improve computation speed, and allow operations such as path calculation to be performed more efficiently. When available memory is scarce, the system will automatically switch to a caching mechanism. By caching frequently accessed graph data, it ensures normal computation while strictly controlling memory usage, avoiding the impact of insufficient memory on the overall operation of the system.

[0074] Finally, there's the mechanism for parallel execution of computation and result writing. The key to this mechanism is breaking the serial relationship between computation and result writing, allowing them to proceed simultaneously, thus eliminating mutual blocking and effectively improving system throughput. In practice, each node performs path computation while asynchronously writing the results. To achieve this, a mechanism combining a write thread pool and a buffer is typically used: once the path computation generates a result, it's immediately placed into a buffer for temporary storage, while a dedicated thread for writing retrieves the data from the buffer and performs the disk write operation. This clearly defined parallel processing method ensures that computational resources are not idle while waiting for I / O operations, and that I / O operations are not stalled due to incomplete computation, significantly improving the overall system efficiency.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0076] Embodiments of the present invention also provide a full path lookup device. Figure 3 This is a schematic diagram of the full-path search device provided in an embodiment of the present invention. This embodiment is based on functional modules, such as… Figure 3 As shown, the device may include:

[0077] The request receiving module 10 is used to receive path lookup requests; the path lookup request includes a graph, the start point and the end point in the graph, and a setting of the total number of path steps;

[0078] The step count determination module 11 is used to determine the backtracking step count based on the set total path steps in response to the path lookup request;

[0079] The information recording module 12 is used to perform backtracking preprocessing of the graph starting from the endpoint based on the number of backtracking steps, and to record the set of reachable nodes during the backtracking process at each level;

[0080] The path search module 13 is used to search for paths in parallel from the starting point. During the search process, it uses the recorded reachable node set information to perform pruning and saves the processed path to the result queue.

[0081] In the full path finding device provided in this embodiment of the invention, through the interaction of the four modules, after receiving and responding to a path finding request, the device performs graph backtracking preprocessing starting from the endpoint based on a determined number of backtracking steps. This records the set of reachable nodes at each level, providing precise pruning criteria for the search process. It can directly filter out path extensions that cannot reach the endpoint, avoiding a large amount of invalid computation and redundant path permutations, without requiring additional deduplication. Simultaneously, a parallel search method starting from the starting point fully utilizes processor resources, improving overall search efficiency and avoiding the performance limitations of single-threaded processing. Through targeted pruning of preprocessed information and ordered result queue storage, the device effectively controls memory usage and prevents excessive resource consumption. Ultimately, it can quickly and accurately find all paths between a specified starting point and endpoint within a set total number of path steps, ensuring result integrity while balancing processing speed and resource utilization.

[0082] Since the embodiments of the full path lookup device and the full path lookup method correspond to each other, the description of the features in the embodiment corresponding to the full path lookup device can be found in the relevant description of the embodiment corresponding to the full path lookup method, and will not be repeated here. Furthermore, it has the same beneficial effects as the full path lookup method mentioned above.

[0083] Furthermore, in a specific implementation, in the full-path search device provided in the embodiments of the present invention, the information recording module 12 may specifically include:

[0084] The record array construction unit is used to construct an array for recording the set of reachable nodes during the backtracking process, which serves as the record array. The size of the record array is the number of backtracking steps plus 1. Each element of the record array is a bit set structure used to represent all reachable nodes at the current level. The end point is written into the bit set structure of all levels.

[0085] The backtracking queue initialization unit is used to initialize the backtracking queue. The elements of the backtracking queue are array structures. The first element of the array structure represents the current backtracking point, and the second element represents the backtracking level.

[0086] The backtracking preprocessing unit is used to process tasks in the backtracking queue in parallel. It records the information of the set of reachable nodes during the backtracking process in a record array until the backtracking queue is cleared and all threads are in an idle state.

[0087] Specifically, the backtracking preprocessing unit can be used to start multi-threaded parallel processing of tasks in the backtracking queue. Each thread takes an element from the backtracking queue, calculates the next level as the current level plus 1, obtains the previous neighbor of the current node, and writes the previous neighbor into the record array into all bit sets from the next level index to the backtracking step number. If the next level is less than the backtracking step number, the previous node and the next level are added to the backtracking queue. When the backtracking queue is empty and the thread is in an idle state, the backtracking preprocessing is completed.

[0088] Furthermore, in a specific implementation, in the full path search device provided in the embodiments of the present invention, the path search module 13 may specifically include:

[0089] The result queue creation unit is used to create a result queue for storing paths; the elements of the result queue are a set of points from the start point to the end point.

[0090] The computation queue initialization unit is used to initialize the computation queue; the computation queue will add the path containing only the starting point as the initial value.

[0091] The path processing unit is used to process paths in the computation queue in parallel. If the length of the current path exceeds the maximum step threshold, the current path is discarded. If the length of the current path does not exceed the maximum step threshold, the last point of the current path is taken, and the next-order neighbor node of the last point is obtained.

[0092] The node pruning unit is used to prune all acquired subsequent neighbor nodes using the recorded reachable node set information, and save the processed path to the result queue.

[0093] Specifically, the node pruning unit can be used to determine whether each candidate successor node in the current level is backtracked to by the endpoint, based on the information of the current level and the set of reachable nodes recorded, for each candidate successor node among all the acquired subsequent neighbor nodes. If not, the candidate successor node is pruned; if so, no pruning is performed, and the candidate successor node is appended to the original path to construct a new path. If the candidate successor node is the endpoint, the new path is added to the result queue; if the candidate successor node is not the endpoint, the candidate successor node is returned to the calculation queue to continue expansion; the search terminates when the calculation queue is empty and the thread is in an idle or sleeping state.

[0094] Furthermore, in a specific implementation, the full path lookup device provided in the embodiments of the present invention may further include: a path storage module, used to extract all paths from the result queue and perform structured persistent storage of the extracted paths.

[0095] Furthermore, in a specific implementation, in the full path lookup device provided in the embodiments of the present invention, the path storage module can be specifically used to create a file or database interface for path persistence; extract paths one by one from the result queue, and write the extracted paths into the created file or database interface in a streaming manner with a set format; each path will be a valid path from the starting point to the ending point, and the format is an ordered set of point numbers.

[0096] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the full path lookup method.

[0097] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described full path lookup method embodiments when running.

[0098] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0099] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the full path lookup method.

[0100] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described full path lookup method embodiments.

[0101] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0102] The foregoing has provided a detailed description of the full-path tracing method, apparatus, device, and medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A full path search method, characterized in that, include: Receive a path lookup request; the path lookup request includes a graph, a start point and an end point in the graph, and a set total number of path steps; In response to the path lookup request, half of the set total path steps is used as the backtracking steps; Based on the number of backtracking steps, perform graph backtracking preprocessing starting from the endpoint, and record the set of reachable nodes during the backtracking process at each level; Starting from the origin, a parallel path search is performed, and pruning is carried out using the recorded reachable node set information during the search process. The processed path is saved to a result queue. Specifically, a result queue is created to store the paths; the elements of the result queue are the set of points from the origin to the destination. A computation queue is initialized; the computation queue is added with paths containing only the origin as initial values. Paths in the computation queue are processed in parallel. If the length of the current path exceeds the maximum step threshold, the current path is discarded; if the length of the current path does not exceed the maximum step threshold, the last point of the current path is taken, and its successor neighbors are obtained. The process involves several steps: First, for each candidate successor node among all acquired subsequent neighbor nodes, and considering the current level and the recorded reachable node set information, determining whether the candidate successor node is backtracked to by the endpoint at the current level. If not, pruning is performed on the candidate successor node. If so, no pruning is performed, and the candidate successor node is appended to the original path to construct a new path. If the candidate successor node is the endpoint, the new path is added to the result queue. If the candidate successor node is not the endpoint, it is returned to the computation queue for further expansion. The search terminates when the computation queue is empty and the thread is in an idle or sleeping state.

2. The full path search method according to claim 1, characterized in that, Based on the backtracking step count, perform graph backtracking preprocessing starting from the endpoint, recording the reachable node set information for each level of backtracking, including: Construct an array to record the set of reachable nodes during the backtracking process, as a record array; the size of the record array is the number of backtracking steps plus 1, and each element of the record array is a bit set structure to represent all reachable nodes at the current level; the encoding of the endpoint is written into the bit set structure of all levels; Initialize the backtracking queue; the elements of the backtracking queue are array structures; the first element of the array structure represents the current backtracking point, and the second element represents the backtracking level; The tasks in the backtracking queue are processed in parallel, and the information of the set of reachable nodes during the backtracking process is recorded in the record array until the backtracking queue is cleared and all threads are in an idle state.

3. The full path search method according to claim 2, characterized in that, Tasks in the backtracking queue are processed in parallel, and relevant data of the reachable node set during the backtracking process are recorded in the record array until the backtracking queue is cleared and all threads are in an idle state, including: Start multi-threaded parallel processing of tasks in the backtracking queue; each thread takes out an element from the backtracking queue, calculates the next level as the current level plus 1, obtains the previous neighbor of the current node, and writes the previous neighbor into the record array into all bit set structures from the next level index to the backtracking step number. If the next level is less than the number of backtracking steps, then add the previous node and the next level to the backtracking queue; Backtracking preprocessing is completed when the backtracking queue is empty and the thread is in an idle state.

4. The full path search method according to claim 1, characterized in that, After saving the processed paths to the results queue, the process also includes: Extract all paths from the result queue and perform structured persistent storage of the extracted paths.

5. The full path search method according to claim 4, characterized in that, Extract all paths from the result queue and perform structured persistent storage of the extracted paths, including: Create a file or database interface for path persistence; Extract paths one by one from the result queue, and write the extracted paths into a pre-created file or database interface in a streaming manner according to a set format; each path will be a valid path from the starting point to the ending point, and the format is an ordered set of point numbers.

6. A full-path search device, characterized in that, include: The request receiving module is used to receive path lookup requests; the path lookup request includes a graph, the starting point and the ending point in the graph, and a set total number of path steps; The step count determination module is used to respond to the path search request by taking half of the set total path steps as the backtracking step count; The information recording module is used to perform graph backtracking preprocessing starting from the endpoint based on the backtracking step number, and record the set of reachable nodes during the backtracking process at each level; A path search module is used to search for paths in parallel starting from the starting point. During the search, it performs pruning using the recorded reachable node set information and saves the processed paths to a result queue. Specifically, it creates a result queue to store paths; the elements of the result queue are a set of points from the starting point to the ending point; it initializes a computation queue, adding paths containing only the starting point as initial values; it processes the paths in the computation queue in parallel; if the length of the current path exceeds the maximum step threshold, the current path is discarded; if the length of the current path does not exceed the maximum step threshold, the last point of the current path is retrieved. Subsequent neighbor nodes; based on each candidate successor node among all acquired subsequent neighbor nodes, combined with the current level and the recorded reachable node set information, determine whether the candidate successor node is backtracked to by the endpoint at the current level; if not, prune the candidate successor node; if so, do not prune the candidate successor node, and append the candidate successor node after the original path to construct a new path; if the candidate successor node is the endpoint, put the new path into the result queue; if the candidate successor node is not the endpoint, put the candidate successor node back into the calculation queue to continue expansion; when the calculation queue is empty and the thread is in an idle or sleeping state, the search terminates.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the full path lookup method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the full path lookup method as described in any one of claims 1 to 5.

Images