Updatable spatial learning indexing method and device based on partitioning and dimensionality reduction

Abstract

The present application relates to an updateable spatial learning indexing method and device based on partitioning and dimensionality reduction. The method includes: dividing a two-dimensional data space into multiple non-overlapping spatial regions, and clustering into spatial regions; mapping the original data into the two-dimensional space, obtaining the position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using the Hilbert curve, obtaining the sorted array of the Hilbert values ​​of the original data in the divided two-dimensional space and the position array of the spatial region sequence value; constructing a multi-segment linear structure model on the coordinate axis according to the one-dimensional sorted data corresponding to the sorted array and the position array; inputting the search conditions to determine the data to be searched and the clustering labels of the data to be searched, and obtaining the data points and spatial regions of the data to be searched; indexing the data points according to the multi-segment linear structure model corresponding to the spatial region. This method can update the index of high-dimensional data and takes up little space.

Description

Technical Field

[0001] The present application relates to the technical field of data indexing, and in particular to an updateable spatial learning indexing method and device based on partitioning and dimensionality reduction. Background Art

[0002] In the era of big data, data retrieval is crucial. The types of data are varied, but in terms of dimensional changes, data can still be divided into one-dimensional and multi-dimensional types. There are many traditional indexes on different dimensions, which can speed up data queries and provide superior conditions for data retrieval. The existing index structures for retrieving high-dimensional data include R-tree, quadtree, KD-tree and other variants. R-tree is a data structure similar to B-tree, with MBR areas divided by rectangles as its child nodes. R+ tree is its variant. Quadtree is similar to R-tree, which divides the plane space into four equal parts and then iterates the division on each equal part.

[0003] These index structures provide convenient conditions for data retrieval and can speed up data transmission, thus providing a more convenient channel for data application. However, because these index structures organize data in a tree structure, they take up a lot of storage space. Studies have shown that in database applications, the volume of index structures is high.

[0004] With the development of machine learning technology, some scholars have proposed that data retrieval can be regarded as a mapping from key to address, that is, a recursive model is constructed through machine learning to predict the location of data, thereby linking it to its corresponding storage address. This method can reduce the time complexity of the query from O (logN) to O (1) in one dimension, and the storage occupation of the index is also much smaller. Figure 1 As shown, according to Kraska's idea, the values ​​in array S are sorted, and each key value has a position k in the sorted array A. When searching for key k, it is necessary to check whether A(posk)=k. The learning index structure first learns the distribution of input data (x, (posx)) through training, x∈S. When querying, input the data key value k, and then recursively select the next layer model "responsible" for the key until the bottom layer of the model. Its output is the position returned by the model, but this is only an approximate position. In order to find the final precise position, it is necessary to perform a binary search within the adjacent position range, where the range of the binary search depends on the maximum prediction error of the model during training.

[0005] Currently, there are some available methods and models for multi-dimensional space or index update. However, in traditional solutions, the learning index for one-dimensional data cannot be updated for high-dimensional data. Summary of the invention

[0006] Based on this, it is necessary to provide an updateable spatial learning indexing method and device based on partitioning and dimensionality reduction to address the above technical problems.

[0007] An updateable spatial learning indexing method based on partitioning and dimensionality reduction, the method comprising:

[0008] Dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions;

[0009] Mapping the original data into a two-dimensional space to obtain position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using a Hilbert curve to obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values;

[0010] According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold;

[0011] Enter the search conditions to determine the data to be found and the cluster labels of the data to be found, and obtain the data points and spatial areas of the data to be found;

[0012] The data points are indexed according to the multi-segment linear structure model corresponding to the spatial region.

[0013] In one of the embodiments, it further includes: randomly selecting k data points as initial cluster centers, and randomly extracting a data batch of size B from a data set D of the original data;

[0014] Calculate the distance from each data point x to all cluster centers in each data batch, and assign the data point x to the cluster center closest to it;

[0015] Use the temporary cluster set T to store the distribution results of data points and calculate the new center of each temporary cluster T[i];

[0016] After each iteration, check whether the cluster center has changed. If the cluster center change is less than the threshold or reaches the maximum number of iterations m, the final cluster is obtained;

[0017] The final cluster assignment is performed on the entire data set D, and each data point is assigned to the nearest cluster center to obtain the final cluster {D1, D2, D3, ..., D k}.

[0018] In one of the embodiments, the method further includes: calculating the position of each high-dimensional data point in the original data in the two-dimensional space, dividing the two-dimensional space into a number of small areas using a Hilbert curve, assigning each data point to a corresponding area, and returning its corresponding sequence value in the area as the Hilbert value of the point;

[0019] Sort the Hilbert values ​​in ascending order from small to large to form a Hilbert value sorting array x of the spatial region and a position array y corresponding to the spatial region.

[0020] In one of the embodiments, it further includes: calculating an average error of the linear segment according to the one-dimensional sorted data corresponding to the sorted array and the position array;

[0021] Calculate the error between the predicted value and the actual value of the multi-segment linear structural model;

[0022] If the average error is not greater than the overall error threshold and the error between the predicted value and the actual value is not greater than the range error threshold, the current linear segment is valid; otherwise, the linear segment parameters are readjusted until the range error threshold is met, thereby constructing a multi-segment linear structure model.

[0023] In one of the embodiments, the method further includes: inputting a search condition to determine the data to be found and the clustering label of the data to be found;

[0024] Determine the line segment where the data to be searched is located, and predict the position of the data to be searched based on the multi-segment linear structure model. If the Hilbert value corresponding to the position prediction is equal to the data to be searched, return the predicted position;

[0025] If not, a search is performed within the overall error threshold and the range error threshold of the multi-segment linear structure model until a predicted position whose Hilbert value is equal to the data to be searched is found.

[0026] In one of the embodiments, it further includes: finding the to-be-inserted position of the multi-segment linear structure model through an index, inserting the to-be-stored data into the to-be-inserted position, and returning a sorted array and a position array of the to-be-stored data;

[0027] According to the sorting array and position array of the data to be stored, check whether it exceeds the overall error threshold and the range error threshold in the multi-segment linear structure model. If so, determine the next position to be inserted for insertion.

[0028] An updateable spatial learning indexing device based on partitioning and dimensionality reduction, the device comprising:

[0029] A clustering module, used to divide the two-dimensional data space into a plurality of non-overlapping spatial regions, and cluster the original data into the spatial regions;

[0030] A dimensionality reduction module is used to map the original data into a two-dimensional space to obtain the position information of the original data in the two-dimensional space, and recursively divide the two-dimensional space using a Hilbert curve to obtain a sorted array of the Hilbert values ​​of the original data in the divided two-dimensional space and a position array of the spatial region sequence values;

[0031] A model building module, used to build a multi-segment linear structure model on a coordinate axis according to the one-dimensional sorted data corresponding to the sorted array and the position array; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold;

[0032] The index module is used to input search conditions to determine the data to be searched and the clustering labels of the data to be searched, obtain the data points and spatial regions of the data to be searched; and index the data points according to the multi-segment linear structure model corresponding to the spatial region.

[0033] In one of the embodiments, the dimensionality reduction module is also used to calculate the position of each high-dimensional data point in the original data in the two-dimensional space, divide the two-dimensional space into several small areas using the Hilbert curve, assign each data point to the corresponding area, and return its corresponding sequence value in the area as the Hilbert value of the point; sort the Hilbert values ​​in ascending order from small to large to form a Hilbert value sorting array x of the spatial area, and a position array y corresponding to the spatial area.

[0034] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions;

[0036] Mapping the original data into a two-dimensional space to obtain position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using a Hilbert curve to obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values;

[0037] According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold;

[0038] Enter the search conditions to determine the data to be found and the cluster labels of the data to be found, and obtain the data points and spatial areas of the data to be found;

[0039] The data points are indexed according to the multi-segment linear structure model corresponding to the spatial region.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions;

[0042] Mapping the original data into a two-dimensional space to obtain position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using a Hilbert curve to obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values;

[0043] According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold;

[0044] Enter the search conditions to determine the data to be found and the cluster labels of the data to be found, and obtain the data points and spatial areas of the data to be found;

[0045] The data points are indexed according to the multi-segment linear structure model corresponding to the spatial region.

[0046] The above-mentioned updateable spatial learning indexing method and device based on partitioning and dimensionality reduction divides large-scale two-dimensional spatial data through a clustering algorithm, and then reduces the dimensionality of the two-dimensional data through the Hilbert curve, maintains the relative integrity and uniqueness of the data sorting information, and sorts the Hilbert values ​​after dimensionality reduction in each partition to form a sorted array, so that it is more convenient to build a multi-segment linear model in the next step, and then sets the overall error threshold and the range error threshold. According to the size of these two thresholds, a complete piecewise linear model structure is formed progressively, which is convenient for subsequent predictions. The prediction error can be kept within a certain range, thereby more accurately locating the actual position of the data; compared with the traditional index R-tree that supports two-dimensional data, it can achieve better query and new data point insertion efficiency than the R-tree, and has a very low occupied area. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an updateable spatial learning indexing method based on partitioning and dimensionality reduction in one embodiment;

[0048] Figure 2 is a schematic diagram of a Hilbert curve in one embodiment;

[0049] Figure 3 is a schematic diagram of a multi-segment linear structure model in one embodiment;

[0050] Figure 4It is a structural block diagram of an updateable spatial learning indexing device based on partitioning and dimensionality reduction in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 As shown, an updateable spatial learning indexing method based on partitioning and dimensionality reduction is provided, comprising the following steps:

[0053] Step 102 , dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions.

[0054] The clustering method may be a mini-bitch k-means clustering algorithm.

[0055] Step 104, mapping the original data into a two-dimensional space, obtaining the position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using the Hilbert curve, obtaining a sorted array of the Hilbert values ​​of the original data in the divided two-dimensional space and a position array of the spatial region sequence values.

[0056] Step 106: construct a multi-segment linear structure model on the coordinate axis according to the one-dimensional sorted data corresponding to the sorted array and the position array.

[0057] The multi-segment linear structural model meets the overall error threshold and the range error threshold.

[0058] Step 108: input search conditions to determine the data to be found and the clustering labels of the data to be found, and obtain the data points and spatial regions of the data to be found.

[0059] Step 110, indexing the data points according to the multi-segment linear structure model corresponding to the spatial region.

[0060] In the above-mentioned updateable spatial learning indexing method based on partitioning and dimensionality reduction, large-scale two-dimensional spatial data is divided by a clustering algorithm, and then the dimension of the two-dimensional data is reduced by the Hilbert curve to maintain the relative integrity and uniqueness of the data sorting information, and the Hilbert values ​​after dimensionality reduction in each partition are sorted to form a sorted array, so that it is more convenient to build a multi-segment linear model in the next step, and then the overall error threshold and the range error threshold are set. According to the size of these two thresholds, a complete piecewise linear model structure is formed progressively, which is convenient for subsequent predictions to keep the prediction error within a certain range, thereby more accurately locating the actual position of the data; compared with the traditional index R-tree that supports two-dimensional data, it can achieve better query and new data point insertion efficiency than the R-tree, and has a very low occupied area.

[0061] In one embodiment, k data points are randomly selected as initial cluster centers, a data batch of size B is randomly extracted from a data set D of the original data, the distance from the data point x in each data batch to all cluster centers is calculated, and the data point x is assigned to the cluster center closest to it, a temporary cluster set T is used to store the data point assignment results, and a new center of each temporary cluster T[i] is calculated; after each iteration, it is checked whether the cluster center changes, and if the cluster center change is less than a threshold or reaches a maximum number of iterations m, a final cluster is obtained; the final cluster assignment is performed on the entire data set D, and each data point is assigned to the nearest cluster center to obtain the final cluster {D1, D2, D3, ..., D k It is worth noting that after clustering is completed, the cluster partition data and cluster ID are obtained for subsequent indexing.

[0062] In one embodiment, the position of each high-dimensional data point in the original data in the two-dimensional space is calculated, the two-dimensional space is divided into several small areas using the Hilbert curve, and each data point is assigned to the corresponding area, and the corresponding sequence value in the area is returned as the Hilbert value of the point; the Hilbert values ​​are sorted in ascending order from small to large to form a Hilbert value sorting array x of the spatial area, and a position array y corresponding to the spatial area. The Hilbert curve is as follows Figure 2 shown.

[0063] In one embodiment, if Figure 3 As shown, according to the one-dimensional sorted data corresponding to the sorted array and the position array, the average error of the linear segment is calculated; the error between the predicted value and the actual value of the multi-segment linear structure model is calculated; if the average error is not greater than the overall error threshold and the error between the predicted value and the actual value is not greater than the range error threshold, the current linear segment is valid, otherwise the linear segment parameters are readjusted until the range error threshold is met, thereby constructing a multi-segment linear structure model.

[0064] Specifically, an overall error threshold α and a range error threshold β are set. The overall error threshold α indicates that the average error of the generated linear segment shall not exceed α, and the range error threshold β indicates that the error between the model prediction value and the actual value shall not exceed β.

[0065] For each linear segment i in the multi-segment linear model, the overall error is defined as the average of the prediction errors of all data points in the segment. is the predicted value of the model on the input array x, y j is the actual value, the overall error ME i It is expressed as:

[0066] ;

[0067] Where: Ni is the number of data points in the i-th linear segment; is the model's predicted position for the jth data point; y j is the actual value of the jth data point, and the threshold requirement for the overall error is: ME i ≤α.

[0068] The range error describes whether the error between the model's predicted value and the actual value on the linear segment meets the preset range threshold. j For example, the range error is defined as:

[0069] ;

[0070] Where: Ej is the error range of the jth data point. The threshold requirement for the error is: Ej≤β. For all data points in the entire linear segment, ensure that the range error of each data point is within the threshold β.

[0071] In one of the embodiments, a search condition is input to determine the data to be found and the cluster label of the data to be found; the line segment where the data to be found is located is determined, and the position of the data to be found is predicted based on a multi-segment linear structure model. If the Hilbert value corresponding to the position prediction is equal to the data to be found, the predicted position is returned; if not, a search is performed within the overall error threshold and range error threshold of the multi-segment linear structure model until the predicted position whose Hilbert value is equal to the data to be found is found.

[0072] In another embodiment, the position to be inserted in the multi-segment linear structure model is found through the index, the data to be stored is inserted into the position to be inserted, and the sorted array and position array of the data to be stored are returned; according to the sorted array and position array of the data to be stored, it is checked whether the overall error threshold and the range error threshold are exceeded in the multi-segment linear structure model, and if so, the next position to be inserted is determined for insertion.

[0073] Specifically, for index operations:

[0074] S1: Input includes: original data, searched data points, cluster labels of data points, multi-segment linear model and allowed prediction threshold error.

[0075] S11: Cluster selection: Use the original data and cluster labels to determine the id of the cluster to which the data point belongs.

[0076] S12 Model selection: Get the multi-segment linear model of the corresponding cluster, use the multi-segment linear model to predict the data points and search for the data points and return their positions.

[0077] S121: Segment positioning: determine the segment where the data point is located.

[0078] S122: Use a multi-segment linear model to predict the data point and obtain the predicted position. If the Hilbert value corresponding to the predicted position is equal to the value of the data point entering the prediction, mark the predicted position as found and return it.

[0079] S123: Otherwise, perform an index search within the prediction threshold error, perform an index search in the range near the predicted value, check whether the prediction result of each search value is within the allowable threshold error, and if the predicted position meets the threshold error and the corresponding Hilbert value is equal to the value of the data point entering the prediction, return to that position.

[0080] S13: Returns the found location or returns a "not found" flag if not found.

[0081] For update operations:

[0082] S1: Use the point query algorithm to find the insertion position.

[0083] S11: If the insertion position returns a "not found" flag, search for a position smaller than the value through index search, and return the position after the value as the insertion position.

[0084] S12: insert the new data into the global array, update the overall x array and y array, and insert the new data into the current segment, update the x array and y array of the segment.

[0085] S13: Check whether the predicted position of the linear model of the newly inserted value in the segment differs from the actual position by more than the prediction threshold range. If it does not exceed the threshold, insert it on the spot and judge and insert the next insertion point. If it exceeds the threshold, retrain the segment.

[0086] S131: If the inserted value causes an error exceeding the error range, the inserted value is inserted into the segment, and the piecewise linear fitting is re-performed according to the multi-segment linear model construction algorithm to calculate a new model and segment.

[0087] S14: Returns the updated multi-segment linear model and a new sorted array.

[0088] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0089] In one embodiment, Figure 4 As shown, an updateable spatial learning indexing device based on partitioning and dimensionality reduction is provided, comprising: a clustering module 402, a dimensionality reduction module 404, a model building module 406 and an indexing module 408, wherein:

[0090] A clustering module 402, for dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions;

[0091] Dimensionality reduction module 404, used to map the original data into a two-dimensional space, obtain the position information of the original data in the two-dimensional space, recursively divide the two-dimensional space using a Hilbert curve, and obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values;

[0092] A model building module 406 is used to build a multi-segment linear structure model on the coordinate axis according to the one-dimensional sorted data corresponding to the sorted array and the position array; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold;

[0093] The index module 408 is used to input search conditions to determine the data to be searched and the clustering labels of the data to be searched, obtain the data points and spatial regions of the data to be searched; and index the data points according to the multi-segment linear structure model corresponding to the spatial region.

[0094] In one embodiment, the clustering module 402 is further used to randomly select k data points as initial cluster centers, randomly extract a data batch of size B from a data set D of the original data; calculate the distance from the data point x in each data batch to all cluster centers, and assign the data point x to the cluster center closest to it; use a temporary cluster set T to store the data point allocation results, and calculate the new center of each temporary cluster T[i]; check whether the cluster center changes after each iteration, and if the cluster center change is less than a threshold or reaches a maximum number of iterations m, obtain the final cluster; perform final cluster allocation on the entire data set D, assign each data point to the nearest cluster center, and obtain the final cluster {D1, D2, D3, ..., D k}.

[0095] In one of the embodiments, the dimensionality reduction module 404 is also used to calculate the position of each high-dimensional data point in the original data in the two-dimensional space, divide the two-dimensional space into several small areas using the Hilbert curve, assign each data point to the corresponding area, and return its corresponding sequence value in the area as the Hilbert value of the point; sort the Hilbert values ​​in ascending order from small to large to form a Hilbert value sorting array x of the spatial area, and a position array y corresponding to the spatial area.

[0096] In one of the embodiments, the model building module 406 is also used to calculate the average error of the linear segment based on the one-dimensional sorted data corresponding to the sorted array and the position array; calculate the error between the predicted value and the actual value of the multi-segment linear structure model; if the average error is not greater than the overall error threshold and the error between the predicted value and the actual value is not greater than the range error threshold, the current linear segment is valid, otherwise the linear segment parameters are readjusted until the range error threshold is met, thereby building a multi-segment linear structure model.

[0097] In one of the embodiments, the index module 408 is also used to input search conditions to determine the data to be found and the clustering label of the data to be found; determine the line segment where the data to be found is located, and predict the position of the data to be found based on the multi-segment linear structure model. If the Hilbert value corresponding to the position prediction is equal to the data to be found, return the predicted position; if not, search within the overall error threshold and range error threshold of the multi-segment linear structure model until the predicted position whose Hilbert value is equal to the data to be found is found.

[0098] In one of the embodiments, it also includes an update module, which is used to find the position to be inserted in the multi-segment linear structure model through an index, insert the data to be stored into the position to be inserted, and return the sorted array and position array of the data to be stored; according to the sorted array and position array of the data to be stored, check whether the overall error threshold and the range error threshold are exceeded in the multi-segment linear structure model, and if so, determine the next position to be inserted for insertion.

[0099] For the specific limitations of the updateable spatial learning index device based on partitioning and dimensionality reduction, please refer to the limitations of the updateable spatial learning index method based on partitioning and dimensionality reduction in the above text, which will not be repeated here. Each module in the above-mentioned updateable spatial learning index device based on partitioning and dimensionality reduction can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0100] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0102] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An updateable spatial learning indexing method based on partitioning and dimensionality reduction, characterized in that: The method comprises: Dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions; Mapping the original data into a two-dimensional space to obtain position information of the original data in the two-dimensional space, recursively dividing the two-dimensional space using a Hilbert curve to obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values; According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold; Enter the search conditions to determine the data to be found and the cluster labels of the data to be found, and obtain the data points and spatial areas of the data to be found; Indexing the data points according to a multi-segment linear structure model corresponding to the spatial region; Enter the search conditions to determine the data to be found and the cluster labels of the data to be found, and obtain the data points and spatial areas of the data to be found, including: Enter search conditions to determine the data to be found and the clustering labels of the data to be found; Determine the line segment where the data to be searched is located, and predict the position of the data to be searched based on the multi-segment linear structure model. If the Hilbert value corresponding to the position prediction is equal to the data to be searched, return the predicted position; If not, searching within the overall error threshold and the range error threshold of the multi-segment linear structure model until a predicted position whose Hilbert value is equal to the data to be searched is found; The method comprises: Find the to-be-inserted position of the multi-segment linear structure model through the index, insert the to-be-stored data into the to-be-inserted position, and return the sorted array and position array of the to-be-stored data; Check whether the overall error threshold and the range error threshold are exceeded in the multi-segment linear structure model according to the sort array and the position array of the data to be stored, and if so, determine the next position to be inserted for insertion; The update steps include: Use the point query algorithm to find the insertion position; If the insertion position returns a not found flag, search for a position less than that value by index search, and return the position after it as the insertion position; Insert new data into the global array, update the overall x array and y array, and insert new data into the current segment, update the x array and y array of the segment; Check whether the predicted position of the newly inserted value in the linear model of the segment differs from the actual position by more than the prediction threshold. If it does not exceed the threshold, insert it in place and judge and insert the next insertion point. If it exceeds the threshold, retrain the segment. If the inserted value causes the error to exceed the error range, the inserted value is inserted into the segment, and the piecewise linear fit is re-performed according to the multi-segment linear model construction algorithm to calculate the new model and segment; Returns the updated multi-segment linear model and a new sorted array.

2. The method according to claim 1, characterized in that Dividing the two-dimensional data space into a plurality of non-overlapping spatial regions, and clustering the original data into the spatial regions, comprises: Randomly select k data points as the initial cluster centers and randomly extract a data batch of size B from the dataset D of the original data; Calculate the distance from each data point x to all cluster centers in each data batch, and assign the data point x to the cluster center closest to it; Use the temporary cluster set T to store the distribution results of data points and calculate the new center of each temporary cluster T[i]; After each iteration, check whether the cluster center has changed. If the cluster center change is less than the threshold or reaches the maximum number of iterations m, the final cluster is obtained; The final cluster assignment is performed on the entire data set D, and each data point is assigned to the nearest cluster center to obtain the final cluster {D1, D2, D3, ..., D k }.

3. The method according to claim 2, characterized in that According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis, including: Calculate the position of each high-dimensional data point in the original data in the two-dimensional space, divide the two-dimensional space into several small areas using the Hilbert curve, assign each data point to the corresponding area, and return its corresponding sequence value in the area as the Hilbert value of the data point; Sort the Hilbert values ​​in ascending order from small to large to form a Hilbert value sorting array x of the spatial region and a position array y corresponding to the spatial region.

4. The method according to claim 3, characterized in that According to the one-dimensional sorting data corresponding to the sorting array and the position array, a multi-segment linear structure model is constructed on the coordinate axis, including: Calculating the average error of the linear segment according to the one-dimensional sorted data corresponding to the sorted array and the position array; Calculate the error between the predicted value and the actual value of the multi-segment linear structural model; If the average error is not greater than the overall error threshold and the error between the predicted value and the actual value is not greater than the range error threshold, the current linear segment is valid; otherwise, the linear segment parameters are readjusted until the range error threshold is met, thereby constructing a multi-segment linear structure model.

5. An updateable spatial learning indexing device based on partitioning and dimensionality reduction, used to implement the updateable spatial learning indexing method based on partitioning and dimensionality reduction according to any one of claims 1 to 4, characterized in that: The device comprises: A clustering module, used to divide the two-dimensional data space into a plurality of non-overlapping spatial regions, and cluster the original data into the spatial regions; A dimensionality reduction module is used to map the original data into a two-dimensional space to obtain the position information of the original data in the two-dimensional space, and recursively divide the two-dimensional space using a Hilbert curve to obtain a sorted array of Hilbert values ​​of the original data in the divided two-dimensional space and a position array of spatial region sequence values; A model building module, used to build a multi-segment linear structure model on a coordinate axis according to the one-dimensional sorted data corresponding to the sorted array and the position array; the multi-segment linear structure model satisfies the overall error threshold and the range error threshold; The index module is used to input search conditions to determine the data to be searched and the clustering labels of the data to be searched, obtain the data points and spatial regions of the data to be searched; and index the data points according to the multi-segment linear structure model corresponding to the spatial region.

6. The device according to claim 5, characterized in that The dimension reduction module is also used to calculate the position of each high-dimensional data point in the original data in the two-dimensional space, divide the two-dimensional space into several small areas using the Hilbert curve, assign each data point to the corresponding area, and return its corresponding sequence value in the area as the Hilbert value of the data point; sort the Hilbert values ​​in ascending order from small to large to form a Hilbert value sorting array x of the spatial area, and a position array y corresponding to the spatial area.

Images