A data search processing method, device and equipment

Abstract

The application discloses a data search processing method, device and equipment, and the method comprises the following steps: obtaining a first abstract resilient dataset of a parallel computing framework algorithm according to a dataset of a file system; performing conversion processing on the first abstract resilient dataset to obtain a second abstract resilient dataset; forming a third abstract resilient dataset according to a random projection seed of a random projection tree and the second abstract resilient dataset; performing calculation processing on the third abstract resilient dataset to obtain a fourth abstract resilient dataset; when judging that a leaf node of the random projection tree reaches a condition according to the fourth abstract resilient dataset, obtaining an optimized random projection tree; and searching for target data according to the optimized random projection tree. Through the above method, the application realizes fast retrieval of target data in a high-dimensional space.

Description

Technical Field

[0001] This invention relates to the field of data search and processing technology, and specifically to a data search and processing method, apparatus, and device. Background Technology

[0002] Current search techniques include random projection tree search algorithms, Spark search algorithms (parallel computing framework), and pseudo-random number search algorithms; existing techniques have the following drawbacks:

[0003] It is impossible to perform fast text search effectively in high-dimensional space, and when the data volume is large, the index data consumes a lot of storage resources on a single machine, which limits the expansion of large-scale data. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a data search and processing method, apparatus and device that overcomes or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a data search processing method is provided, the method comprising:

[0006] The first abstract elastic dataset of the parallel computing framework algorithm is obtained from the dataset of the file system;

[0007] The first abstract elastic dataset is transformed to obtain the second abstract elastic dataset;

[0008] Based on the random projection seed of the random projection tree and the second abstract elastic dataset, a third abstract elastic dataset is formed;

[0009] The third abstract elastic dataset is processed to obtain the fourth abstract elastic dataset;

[0010] Based on the fourth abstract elastic dataset, when the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained;

[0011] The optimized random projection tree is used to search the data to be searched to obtain the target data.

[0012] According to another aspect of the present invention, a data search processing apparatus is provided, the apparatus comprising:

[0013] The first acquisition module is used to obtain the first abstract elastic dataset of the parallel computing framework algorithm based on the dataset of the file system;

[0014] The second acquisition module is used to transform the first abstract elastic dataset to obtain the second abstract elastic dataset;

[0015] The third acquisition module is used to form a third abstract elastic dataset based on the random projection seed of the random projection tree and the second abstract elastic dataset.

[0016] The fourth obtaining module is used to perform calculations on the third abstract elastic dataset to obtain the fourth abstract elastic dataset;

[0017] The processing module is used to determine, based on the fourth abstract elastic dataset, that when the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained.

[0018] The search module is used to search the data to be searched based on the optimized random projection tree to obtain the target data.

[0019] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0020] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the data search processing method described above.

[0021] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a processor to perform an operation corresponding to the data search processing method described above.

[0022] According to the solution provided by the above embodiments of the present invention, target data can be retrieved quickly in a high-dimensional space, thereby solving the problem that the prior art cannot perform effective searches in a high-dimensional space; and achieving the beneficial effect of quickly retrieving target data in a high-dimensional space.

[0023] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific implementation methods of the embodiments of the present invention are described below. Attached Figure Description

[0024] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0025] Figure 1A flowchart of the data search and processing method provided in an embodiment of the present invention is shown;

[0026] Figure 2 A schematic diagram of the architecture of the Spark algorithm, a parallel computing framework, is shown in an embodiment of the present invention.

[0027] Figure 3 This diagram illustrates the relationship between RDDs and worker nodes in the Spark algorithm, a parallel computing framework according to an embodiment of the present invention.

[0028] Figure 4 A flowchart illustrating the data search and processing method according to an embodiment of the present invention is shown;

[0029] Figure 5 A schematic diagram of a misclassification in an embodiment of the present invention is shown;

[0030] Figure 6 Another misclassification diagram is shown in an embodiment of the present invention;

[0031] Figure 7 A schematic diagram of the structure of the data search and processing apparatus provided in an embodiment of the present invention is shown;

[0032] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0033] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0034] Figure 1 A flowchart of a data search and processing method provided by an embodiment of the present invention is shown. Figure 1 As shown, the method includes the following steps:

[0035] Step 11: Obtain the first abstract elastic dataset of the parallel computing framework algorithm based on the dataset of the file system; here, in specific implementation, it may include: after obtaining the dataset of the file system, performing initialization processing on the dataset according to the parallel computing framework algorithm Spark to obtain the first abstract elastic dataset, which can be denoted as RDD0.

[0036] Step 12: Transform the first abstract elastic dataset to obtain the second abstract elastic dataset; in specific implementation, the second abstract elastic dataset can be denoted as RDD1.

[0037] Step 13: Based on the random projection seed of the random projection tree and the second abstract elastic dataset, form the third abstract elastic dataset; in specific implementation, the third abstract elastic dataset here can be denoted as RDD2;

[0038] Step 14: Perform calculations on the third abstract elastic dataset to obtain the fourth abstract elastic dataset; in a specific implementation, the third abstract elastic dataset can be denoted as RDD3.

[0039] Step 15: Based on the fourth abstract elastic dataset, determine when the leaf nodes of the random projection tree meet the conditions, and obtain the optimized random projection tree;

[0040] Step 16: Search the data to be searched according to the optimized random projection tree.

[0041] This embodiment of the invention combines a parallel computing framework algorithm with a random projection tree to generate an optimized random projection tree. The optimized random projection tree is then used to search the data to be searched, thereby enabling rapid retrieval of target data in a high-dimensional space.

[0042] In the embodiments of this invention, Spark is built on a unified abstract Resilient Distributed Datasets (RDDs), which allows for seamless integration of Spark's various components and enables large-scale data processing within a single application. An RDD is a special collection of data with fault tolerance mechanisms, which can be distributed across nodes in a cluster and perform various parallel operations in a functional set manner. An RDD is a distributed collection of objects, essentially a read-only collection of partitioned records. Each RDD can be divided into multiple partitions, each partition being a fragment of the dataset. Different partitions of an RDD can be stored on different nodes in the cluster, allowing for parallel computation across these nodes.

[0043] like Figure 2 The diagram shows the Spark architecture. The Cluster Manager manages and schedules the resources of the entire cluster, while Worker nodes compute user-submitted tasks. The Executor executes the tasks; it is an application startup process on a Worker node.

[0044] When a client submits a task, the SparkContext requests resources from the cluster manager. The cluster manager allocates computing resources to the Executors based on the CPU core count and memory information specified by the user. The Driver divides the user program into different execution phases, each executed by a set of Tasks. After receiving a Task, the Executor prepares and configures the environment for the Task and reports the Task's execution status back to the Driver.

[0045] RDDs have fault tolerance mechanisms, are read-only and cannot be modified, and can be created by performing deterministic transformation operations. Specifically, RDDs have the following properties:

[0046] Read-only: Cannot be modified; new RDDs can only be generated through transformation operations.

[0047] Distributed: It can be distributed across multiple machines for parallel processing.

[0048] Elasticity: When memory is insufficient during the calculation process, it will exchange data with the disk.

[0049] Memory-based: It can be fully or partially cached in memory and reused across multiple computations.

[0050] RDD operations are lazy. When an RDD performs a transformation operation, the actual computation is not performed. Only when an RDD performs an action operation will the computation task be submitted, thereby executing the corresponding computation operation.

[0051] RDD construction operations: Computations in Spark are all performed by manipulating RDDs. From the perspective of data source, RDD construction methods can be divided into the following two categories:

[0052] Read data directly from memory.

[0053] Reading data from a file system involves various types of file systems, the most common being HDFS (Distributed File System) and local file systems.

[0054] The first approach involves constructing an RDD from memory, which requires using the makeRDD method.

[0055] The second method involves constructing an RDD through the file system.

[0056] RDD transformation operations are operations that return new RDDs. Functions used in RDD transformation operations include: map(), flatmap(), filter(), etc.; the meanings of each function are as follows:

[0057] The map() function applies a function to each element of an RDD and returns a new RDD.

[0058] flatmap() applies a function to each element of an RDD, splits the element data into iterators, and returns a new RDD.

[0059] The filter() function filters out elements that do not meet the conditions and returns a new RDD.

[0060] Action operations in RDDs are used to perform computations and output results in a specified manner. An action operation takes an RDD but returns a non-RDD, meaning it outputs a value or result. During RDD execution, the actual computation occurs within the action operation. Functions used in RDD action operations include: `collect()`, `count()`, and `reduce()`, with the following meanings:

[0061] collect() returns all elements of an RDD;

[0062] count() returns the number of elements in an RDD;

[0063] `reduce()` performs parallel integration of all RDD data, such as summation operations.

[0064] In an optional embodiment of the present invention, step 12 may include:

[0065] The first abstract elastic dataset is transformed according to the mapping operator or function of the parallel computing framework algorithm to obtain the second abstract elastic dataset.

[0066] In this embodiment, the cluster reads a dataset from HDFS (Distributed File System). The dataset can be text data. After forming an initial RDD0, RDD0 starts the map operator to execute the vector transformation function Vectors.dense(), which converts the original text data into processable vector data key-value pairs, forming RDD1.

[0067] In an optional embodiment of the present invention, step 13 may include:

[0068] Step 131, construct the random projection tree (RPtree);

[0069] Step 132: Generate a random seed for the random projection tree;

[0070] Step 133: Generate a random vector based on the random seed;

[0071] Step 134: Obtain the dot product of each data point in the second abstract elastic dataset with the random vector, wherein the random vector is perpendicular to the vector originating from the data point;

[0072] Step 135: Based on the dot product, form the third abstract elastic dataset.

[0073] Specifically, when the dot product is greater than a preset value, the key is the hash value of the first random vector, indicating that the data point belongs to the right subtree of the random projection tree;

[0074] When the data dot product is less than the preset value, the key is the hash value of the second random vector, indicating that the data point belongs to the left subtree of the random projection tree; the preset value here can preferably be 0.

[0075] The key-value pairs in the form of (key, data point coordinates) are used as the third abstract elastic dataset.

[0076] In this embodiment, the random generator of the random projection tree generates a random vector z based on a random seed. The second abstract elastic dataset RDD1 starts the map operator to calculate the dot product of each data point in the dataset with the vertical vector z. When the dot product is greater than 0, the key is the hash value of the random vector connected to 0 (adv75x1_0), indicating that it belongs to the right subtree; when the dot product is less than 0, the key is the hash value of the random vector connected to 1 (adv75x1_1), indicating that it belongs to the left subtree; and forms a key-value pair RDD2 in the form of (key, data point coordinates).

[0077] In an optional embodiment of the present invention, step 14 may include:

[0078] Step 141: According to the reduce operator of the parallel computing framework algorithm, perform calculation processing on the third abstract elastic dataset to obtain the number of leaf nodes count in the third abstract elastic dataset;

[0079] Step 142: Use key-value pairs of the form (key, count) as the fourth abstract elastic dataset.

[0080] In this embodiment, RDD2 starts the reduceByKey operator to calculate the data split point split, the number of leaf nodes count, and forms RDD3(key, count).

[0081] In an optional embodiment of the present invention, step 15 may include:

[0082] Based on the key-value pairs in the fourth abstract elastic dataset, when the number of leaf nodes count reaches the maximum cluster size c, the partitioning of the data space is stopped, and an optimized random projection tree is obtained.

[0083] In this embodiment, the number of leaf nodes (count) is determined. When the number of leaf nodes reaches the maximum cluster size (c), the data space partitioning is stopped and the process ends; otherwise, the process jumps to step 11 to continue.

[0084] In this embodiment of the invention, a pseudo-random seed is used to construct and store the index. In reality, the system does not need to store every random vector in the random vector pool; it only needs to construct a random seed for each RP tree. The process of partitioning the RP tree using the random seed in the above steps is as follows:

[0085] A) Select a random seed from the random vector pool;

[0086] B) Use the Spark sample operator to generate random vectors and new random seeds;

[0087] C) Broadcast a random vector to each Spark worker node;

[0088] D) Divide the data space using random vectors according to the partitioning rule;

[0089] E) Repeat step B until the number of leaf nodes reaches the maximum cluster size c, then stop partitioning.

[0090] The following flowchart illustrates the process of generating the optimized random projection tree:

[0091] like Figure 4 As shown, the process specifically includes:

[0092] Data preprocessing stage:

[0093] The cluster reads the dataset from HDFS (Distributed File System) to form the initial RDD0; RDD0 starts the map operator to execute Vectors.dense(), which transforms the original text data into processable vector data key-value pairs, forming RDD1;

[0094] Constructing a random projection tree (RP tree):

[0095] Step 1: The random generator produces a random vector z based on the random seed. RDD1 starts the map operator to calculate the dot product of each point in the dataset with the vertical vector z. When the dot product is greater than 0, the key is the hash value of the random vector connected to 0 (adv75x1_0), indicating that it belongs to the right subtree; when the dot product is less than 0, the key is the hash value of the random vector connected to 1 (adv75x1_1), indicating that it belongs to the left subtree; and forms a key-value pair RDD2 in the form of (key, data point coordinates).

[0096] Step 2: RDD2 starts the reduceByKey operator to calculate the data split point split, the number of leaf nodes count, and forms RDD3(key, count);

[0097] Step 3: Determine the value of count. If the number of leaf nodes count reaches the maximum cluster size c, stop partitioning and end the process; otherwise, go back to step 1.

[0098] In this embodiment of the invention, the improved RP trees algorithm solves the problem of quickly retrieving text in a high-dimensional space by constructing RP trees in parallel across multiple nodes on the Spark big data platform.

[0099] When constructing the RP tree in parallel across multiple nodes on the Spark big data platform, a random vector pool is used to reuse random vectors to partition the Euclidean vector space, which solves the problem of storing large-scale index data. By using pseudo-random seeds to construct and store the index, the significant network congestion problem when broadcasting the random vector pool to each Spark node is solved.

[0100] In an optional embodiment of the present invention, the data search processing method may further include:

[0101] The number of optimized random projection trees and the maximum cluster size c are adjusted to obtain the final optimized random projection tree pool.

[0102] like Figure 5 As shown, in a two-dimensional vector space, where the black dots and the four-pointed stars represent text vector data points, the process begins by projecting all points onto a random vector x. Then, based on the average value of all projected points on the random vector x, the entire space is divided into two parts to reduce the amount of data. For each new subspace, a new random vector is selected and used to divide it into two parts. This process continues recursively until the amount of data in each cluster reaches a pre-set threshold. During this process, a binary tree is constructed to facilitate the retrieval of data points in the vector space.

[0103] exist Figure 5 In the given scenario, points A, B, and C appear close in projection onto the random vector x, but in reality, point C is quite far from A and B. As more and more random projection vectors iteratively divide the vector space and cut corresponding blocks, the probability of this misclassification will become very low.

[0104] like Figure 6As shown, points B and D are assigned to different subspaces. This misclassification severely impacts the system's retrieval accuracy. To reduce the probability of such misclassification, embodiments of the present invention choose to construct RP trees to improve the overall retrieval accuracy of the system. Although the query time remains essentially unchanged, this requires a significant amount of memory to store additional index data. When a query point is retrieved, the randomness during tree construction can introduce additional data points into the retrieved nearest neighbor set. When retrieving nearest neighbors, the above processing can minimize the second type of misclassification. When the system uses more RP trees, the probability of a query point approaching the dividing hyperplane is greatly reduced, and therefore the probability of its nearest neighbors being assigned to different clusters is also very small. By reducing this misclassification, the retrieval accuracy is improved.

[0105] By covering data points with multiple vector space partitions, data points are repeatedly stored in the leaf nodes of multiple RP trees. Using more RP trees reduces two misclassifications: 1. Data points projected close distances onto a random vector x, but are actually far apart; 2. Similar data points are misclassified by being grouped into different subspaces. In embodiments of this invention, retrieval accuracy and retrieval time are obtained by adjusting the number of RP trees t and the maximum cluster size c.

[0106] The following analysis, based on specific experimental data, examines the retrieval accuracy and retrieval time obtained by adjusting the number of RP trees (t) and the maximum cluster size (c):

[0107] In this embodiment of the invention, a corpus of 2 million articles is used as the test database. The database retrieval time represents the time required to process a request and can also be used to calculate the database throughput. Throughput is arguably the most important indicator for measuring the scalability of database retrieval. This embodiment uses the English Wikipedia dataset for testing.

[0108] For KNN search, when the amount of web page data reaches millions, the retrieval time consumed by linear search and kd-tree under single-machine conditions can reach several seconds. This is unacceptable to users and applications.

[0109] In the evaluation, the system still accesses the original high-dimensional space, first identifying clusters associated with the query vector, and then performing a linear search within those clusters to find the nearest neighbor. Table 1 shows that the system improves scalability and retrieval time at the cost of reduced accuracy by using a projected space.

[0110]

[0111] Table 1. System retrieval time and retrieval accuracy

[0112] The index can be tuned by configuring two parameters: the maximum cluster size *c* and the number of random projection trees *t*. *c* determines how many random vectors the system will allocate to each leaf node of the retrieval tree, which further affects retrieval accuracy. The first row of Table 1 lists all configurations. Table 1 shows that grouped retrieval accuracy is higher with more random projection trees, but this also increases the index size. Since the system only stores the random seeds for all random vectors, the storage space occupied by these random seeds is negligible; therefore, the increase in index size is mainly due to storing the random projection tree structure. The time overhead of retrieval also increases with the number of random projection trees. Since retrieval in different random projection trees is independent and can be easily parallelized, performance can be further improved by using more worker nodes in the cluster.

[0113] With a fixed random projection tree, larger clusters lead to higher retrieval accuracy, but also increase retrieval time. One might expect retrieval accuracy to decrease, but experimental results show that larger clusters reduce the probability of misclassifying two near split points into different subspaces. Secondly, since the system ultimately performs a linear search within the cluster to find the nearest neighbor, larger clusters provide more points for searching, increasing the probability of finding the actual nearest neighbor. However, the increased retrieval time due to the need for linear searching within larger clusters also contributes to the overall retrieval time.

[0114] Given a fixed cluster size, data shows that this improves both retrieval performance and accuracy while using smaller clusters with more random projection trees. For example, two index configurations, (20, 64) and (80, 16), result in clusters of the same size by merging all leaf nodes of the random projection trees. Experimental data shows that (20, 64) performs better than (80, 16) for the following reasons: First, (20, 64) has higher accuracy than (80, 16). Second, random projection trees are easier to parallelize, so using more random projection trees does not reduce retrieval performance.

[0115] The above experimental data analysis shows that when constructing an RPtree in parallel across multiple nodes on the Spark big data platform, the probability of misclassification will be very low by iteratively dividing the vector space and cutting the corresponding blocks through random projection vectors, which is beneficial to improving the retrieval accuracy of the system.

[0116] Figure 7 A schematic diagram of the structure of the data search and processing apparatus provided in an embodiment of the present invention is shown. Figure 3 As shown, the device includes:

[0117] The first acquisition module 71 is used to obtain the first abstract elastic dataset of the parallel computing framework algorithm based on the dataset;

[0118] The second acquisition module 72 is used to transform the first abstract elastic dataset to obtain the second abstract elastic dataset;

[0119] The third obtaining module 73 is used to form a third abstract elastic dataset based on the random projection seed of the random projection tree and the second abstract elastic dataset.

[0120] The fourth obtaining module 74 is used to perform calculations on the third abstract elastic dataset to obtain the fourth abstract elastic dataset;

[0121] Processing module 75 is used to obtain an optimized random projection tree when the leaf nodes of the random projection tree meet the conditions based on the fourth abstract elastic dataset.

[0122] Search module 76 is used to search the data to be searched based on the optimized random projection tree to obtain the target data.

[0123] Optionally, the first abstract elastic dataset is transformed to obtain a second abstract elastic dataset, including:

[0124] The first abstract elastic dataset is transformed using the map operator of the parallel computing framework algorithm to obtain the second abstract elastic dataset.

[0125] Optionally, a third abstract elastic dataset is formed based on the random projection seed of the random projection tree and the second abstract elastic dataset, including:

[0126] Construct a random projection tree;

[0127] Generate a random seed for the random projection tree;

[0128] Generate a random vector based on the random seed;

[0129] Obtain the dot product of each data point in the second abstract elastic dataset with the random vector, wherein the random vector is perpendicular to the vector originating from the data point;

[0130] The third abstract elastic dataset is formed based on the dot product.

[0131] Optionally, the third abstract elastic dataset is formed based on the dot product, including:

[0132] When the dot product is greater than 0, the key is the hash value of a random vector connected to 0, indicating that the data point belongs to the right subtree of the random projection tree;

[0133] When the dot product of the data is less than 0, the key is the hash value of a random vector connected with 1, indicating that the data point belongs to the left subtree of the random projection tree;

[0134] The key-value pairs in the form of (key, data point coordinates) are used as the third abstract elastic dataset.

[0135] Optionally, the third abstract elastic dataset is processed to obtain a fourth abstract elastic dataset, including:

[0136] Based on the reduce operator of the parallel computing framework algorithm, the third abstract elastic dataset is processed to obtain the count of leaf nodes in the third abstract elastic dataset.

[0137] The key-value pairs in the form of (key, count) are used as the fourth abstract elastic dataset.

[0138] Optionally, based on the fourth abstract elastic dataset, when it is determined that the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained, including:

[0139] Based on the key-value pairs in the fourth abstract elastic dataset, when the number of leaf nodes count reaches the maximum cluster size c, the partitioning of the data space is stopped, and an optimized random projection tree is obtained.

[0140] Optionally, data search and processing methods also include:

[0141] The number of optimized random projection trees and the maximum cluster size c are adjusted to obtain the final optimized random projection tree pool.

[0142] It should be noted that this device is the same as the method described above. All implementations in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect.

[0143] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the data search processing method in any of the above method embodiments.

[0144] Figure 8 The diagram shows a structural schematic of a computing device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0145] like Figure 8 As shown, the computing device may include a processor, a communications interface, memory, and a communications bus.

[0146] The processor, communication interface, and memory communicate with each other via a communication bus. The communication interface is used to communicate with other network elements, such as clients or other servers. The processor executes programs, specifically the relevant steps described in the data search and processing method embodiment for computing devices.

[0147] Specifically, the program may include program code, which includes computer operation instructions.

[0148] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0149] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0150] Specifically, the program can be used to cause the processor to execute the data search processing method in any of the above method embodiments. The specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units in the above data search processing embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0151] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the embodiments of the present invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the embodiments of the present invention.

[0152] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0153] Similarly, it should be understood that, in order to streamline the embodiments of the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0154] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0155] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0156] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The embodiments of the present invention can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0157] It should be noted that the above embodiments are illustrative of the present invention and not restrictive of the invention, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of the present invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A data search and processing method, characterized in that, The method includes: The first abstract elastic dataset of the parallel computing framework algorithm is obtained from the dataset of the file system; The first abstract elastic dataset is transformed to obtain the second abstract elastic dataset; A third abstract elastic dataset is formed based on the random projection seed of the random projection tree and the second abstract elastic dataset. This involves: constructing a random projection tree; generating a random seed for the random projection tree; generating random vectors based on the random seed; obtaining the dot product of each data point in the second abstract elastic dataset with the random vectors, where the random vectors are perpendicular to the vectors originating from the data points; when the dot product is greater than a preset value, the key is a concatenation of the hash values ​​of the first random vectors, indicating that the data point belongs to the right subtree of the random projection tree; when the dot product is less than the preset value, the key is a concatenation of the hash values ​​of the second random vectors, indicating that the data point belongs to the left subtree of the random projection tree; and using key-value pairs in the form of (key, data point coordinates) as the third abstract elastic dataset. The third abstract elastic dataset is processed to obtain the fourth abstract elastic dataset; wherein, according to the reduce operator of the parallel computing framework algorithm, the third abstract elastic dataset is processed to obtain the number of leaf nodes count in the third abstract elastic dataset; and the key-value pairs in the form of (key, count) are used as the fourth abstract elastic dataset. Based on the fourth abstract elastic dataset, when the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained; The optimized random projection tree is used to search the data to be searched to obtain the target data.

2. The data search and processing method according to claim 1, characterized in that, The first abstract elastic dataset is transformed to obtain the second abstract elastic dataset, which includes: The first abstract elastic dataset is transformed using the map operator of the parallel computing framework algorithm to obtain the second abstract elastic dataset.

3. The data search and processing method according to claim 1, characterized in that, Based on the fourth abstract elastic dataset, when the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained, including: Based on the key-value pairs in the fourth abstract elastic dataset, when the number of leaf nodes count reaches the maximum cluster size c, the partitioning of the data space is stopped, and an optimized random projection tree is obtained.

4. The data search and processing method according to claim 3, characterized in that, Also includes: The number of optimized random projection trees and the maximum cluster size c are adjusted to obtain the final optimized random projection tree pool.

5. A data search and processing device, characterized in that, The device includes: The first acquisition module is used to obtain the first abstract elastic dataset of the parallel computing framework algorithm based on the dataset of the file system; The second acquisition module is used to transform the first abstract elastic dataset to obtain the second abstract elastic dataset; The third acquisition module is used to form a third abstract elastic dataset based on the random projection seed of the random projection tree and the second abstract elastic dataset; wherein, a random projection tree is constructed; a random seed of the random projection tree is generated; a random vector is generated based on the random seed; the dot product of each data point in the second abstract elastic dataset and the random vector is obtained, wherein the random vector is perpendicular to the vector originating from the data point; when the dot product is greater than a preset value, the key is a concatenation of the hash values ​​of the first random vector, indicating that the data point belongs to the right subtree of the random projection tree; when the dot product is less than the preset value, the key is a concatenation of the hash values ​​of the second random vector, indicating that the data point belongs to the left subtree of the random projection tree; and key-value pairs in the form of (key, data point coordinates) are used as the third abstract elastic dataset; The fourth obtaining module is used to perform computational processing on the third abstract elastic dataset to obtain a fourth abstract elastic dataset; wherein, according to the reduce operator of the parallel computing framework algorithm, the third abstract elastic dataset is processed to obtain the number of leaf nodes count in the third abstract elastic dataset; and the key-value pairs in the form of (key, count) are used as the fourth abstract elastic dataset. The processing module is used to determine, based on the fourth abstract elastic dataset, that when the leaf nodes of the random projection tree meet the conditions, an optimized random projection tree is obtained. The search module is used to search the data to be searched based on the optimized random projection tree to obtain the target data.

6. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the data search processing method as described in any one of claims 1-4.

7. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the data search processing method as described in any one of claims 1-4.

Images