Data indexing methods, apparatus, electronic devices and storage media

Abstract

This disclosure provides a data indexing method, apparatus, electronic device, and storage medium, relating to the field of data processing, specifically cloud computing, big data, the Internet of Things, and intelligent search. The specific implementation involves: acquiring data to be stored; extracting index features from the data to be stored; sorting and storing target data associated with the data to be stored and the same storage space based on the index features; and establishing an index for the data to be stored based on the index features and the sorted storage location. Embodiments of this disclosure can improve query efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, particularly to the fields of cloud computing, big data, the Internet of Things and intelligent search, and specifically to a data indexing method, apparatus, electronic device and storage medium. Background Technology

[0002] The continuous development of computer applications has led to a dramatic increase in the amount of data.

[0003] With the ever-increasing volume of large-scale data, better tools or technologies are needed to organize and manage files. Efficient data organization methods can help people quickly retrieve the data they want from large-scale background data when needed. Summary of the Invention

[0004] This disclosure provides a data indexing method, apparatus, electronic device, and storage medium.

[0005] According to one aspect of this disclosure, a method for establishing a data index is provided, comprising:

[0006] Retrieve the data to be stored;

[0007] Extract the index features of the data to be stored;

[0008] Based on the index characteristics of the data to be stored, the target data associated with the same storage space are sorted and stored.

[0009] An index is created for the data to be stored based on its index characteristics and the location of its sorted storage.

[0010] According to one aspect of this disclosure, a data query method is provided, comprising:

[0011] A data query request has been received;

[0012] Extract query features from the data query request;

[0013] Based on a pre-established index, data is continuously read at the positions corresponding to the query features, wherein the index is established using the data indexing method described in any embodiment of this disclosure.

[0014] According to one aspect of this disclosure, a data indexing apparatus is provided, comprising:

[0015] The data acquisition module is used to acquire data to be stored.

[0016] The index feature extraction module is used to extract the index features of the data to be stored.

[0017] The sorting and storage module is used to sort and store the target data associated with the data to be stored and the same storage space according to the index characteristics of the data to be stored;

[0018] The index building module is used to build an index for the data to be stored based on the index characteristics and the sorted storage location of the data to be stored.

[0019] According to one aspect of this disclosure, a data query apparatus is provided, comprising:

[0020] The query request acquisition module is used to receive data query requests;

[0021] The query feature extraction module is used to extract query features from the data query request;

[0022] The query value range determination module is used for...

[0023] The data query module is used to continuously read the queried data at the positions corresponding to the query features according to a pre-established index, wherein the index is established by the data index establishment method described in any embodiment of this disclosure.

[0024] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0025] At least one processor; and

[0026] A memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data indexing method described in any embodiment of this disclosure.

[0028] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the data indexing method described in any embodiment of this disclosure.

[0029] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data indexing method described in any embodiment of this disclosure.

[0030] The embodiments disclosed herein can improve query efficiency.

[0031] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0032] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0033] Figure 1 This is a flowchart of a data index creation method disclosed in an embodiment of the present disclosure;

[0034] Figure 2 This is a flowchart of another data index creation method disclosed in the embodiments of this disclosure;

[0035] Figure 3 This is a flowchart of another data index creation method disclosed in the embodiments of this disclosure;

[0036] Figure 4 This is a flowchart of a data query method disclosed in an embodiment of the present disclosure;

[0037] Figure 5 This is a scene diagram of a data indexing system disclosed in an embodiment of this disclosure;

[0038] Figure 6 This is a scene diagram of feature fusion according to an embodiment of the present disclosure;

[0039] Figure 7 This is a scene diagram of index value calculation according to an embodiment of the present disclosure;

[0040] Figure 8 This is a scenario diagram of a data query disclosed in an embodiment of this disclosure;

[0041] Figure 9 This is a structural diagram of a data indexing device disclosed in an embodiment of the present disclosure;

[0042] Figure 10 This is a structural diagram of a data query device disclosed in an embodiment of the present disclosure;

[0043] Figure 11 This is a block diagram of an electronic device used to implement the data indexing method or data query method of the embodiments of this disclosure. Detailed Implementation

[0044] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0045] Figure 1 This is a flowchart of a data index creation method disclosed in an embodiment of this disclosure. This embodiment can be applied to the case of data storage and index creation. The method of this embodiment can be executed by a data index creation device, which can be implemented in software and / or hardware and specifically configured in an electronic device with a certain data processing capability. The electronic device can be a client device or a server device. Client devices include mobile phones, tablet computers, vehicle terminals, and desktop computers.

[0046] S101, Obtain the data to be stored.

[0047] The system receives data storage requests and retrieves the data to be stored. Alternatively, data can be pre-stored in memory. When the amount of data in memory reaches a preset threshold, or when no new data has been written for a certain period, the data to be stored is retrieved from memory and stored on disk. This allows for batch storage of data from memory, reducing disk write operations, improving storage efficiency, and enhancing write performance. Furthermore, data to be stored can be extracted from data storage requests, written to a log, and then written to memory. This allows for recovery of the data to be stored from the log in the event of a system crash or restart, improving system fault tolerance.

[0048] S102, extract the index features of the data to be stored.

[0049] Index features are used to identify data to be stored, distinguishing it from other data and enabling retrieval of that data among numerous data sets. Index features can be the value of a parameter (field) within the data to be stored, or values ​​obtained through processing the data. For example, if the data to be stored is spatiotemporal data, index features could include time and spatial features. Similarly, if the data to be stored is product data, index features could include functional and name features.

[0050] S103, based on the index characteristics of the data to be stored, sort and store the target data associated with the same storage space as the data to be stored.

[0051] The data to be stored and the target data are stored together in the same storage space. The target data can be data that has been pre-stored in this storage space, or it can be data that is stored in the same storage space as the data to be stored, as a batch of data.

[0052] Specifically, sorted storage can be achieved by sorting the data to be stored and the target data according to the index characteristics of the data to be stored and the index characteristics of the target data, and then storing the data to be stored according to the sorting result, so that the target data and the data to be stored are stored in the same storage space in the order of the sorting result.

[0053] For example, index features can be mapped to numerical values, and the data to be stored and the target data can be sorted according to the numerical values.

[0054] S104, Based on the index characteristics and sorted storage location of the data to be stored, an index is established for the data to be stored.

[0055] An index of the data to be stored is used to locate the data, enabling queries. Index features represent information pointing to the data to be stored, and the sorted storage location is the real-time storage location of the data. A correspondence can be established between index features and sorted storage locations, thus creating an index for the data to be stored.

[0056] In existing technologies, data is stored in a discrete manner, which means that queries within the same range need to read data from multiple random locations, resulting in a large amount of input / output overhead and high disk load.

[0057] According to the technical solution disclosed herein, the data to be stored and the target data that need to be stored in the same storage space are sorted according to the index characteristics and stored sequentially, so as to continuously retrieve the data to be queried during data query, reduce the input and output overhead of the query, and thus improve the query performance.

[0058] Figure 2 This is a flowchart of another data indexing method disclosed in this embodiment, which is further optimized and extended based on the above technical solution and can be combined with the above optional implementation methods. Specifically, the step of sorting and storing the data to be stored and target data associated with the same storage space based on the index features of the data to be stored involves: fusing multiple features in the index features of the data to be stored to obtain the representation data corresponding to the data to be stored; determining the index value of the data to be stored based on the representation data and feature trend data corresponding to the data to be stored; sorting the data to be stored and the target data based on the index value of the data to be stored and the index value of the target data associated with the storage space; and storing the data to be stored in the storage space according to the sorting result.

[0059] S201, Obtain the data to be stored.

[0060] S202, Extract the index features of the data to be stored.

[0061] S203, multiple features in the index features of the data to be stored are fused to obtain the representation data corresponding to the data to be stored.

[0062] Index features comprise multiple features, which are then fused together to obtain the fused result, which is the representation data. This representation data identifies the data to be stored and, by integrating multiple features of the data, contains information about those features. In practice, building an index based on the representation data can reduce the dimensionality of the index information. This allows for queries using multiple fields to be merged into a single piece of information, enabling dimensionality-reduced queries and improving query efficiency.

[0063] S204, determine the index value of the data to be stored based on the characterization data and characteristic trend data corresponding to the data to be stored.

[0064] Feature trend data is used to describe the distribution trend of data characteristics within the same storage space. The index value of the data to be stored is used to quantify the distribution location of the data. Based on the characterization data and the feature trend data, the distribution location of the data to be stored within the storage space can be determined, and this distribution location can be quantized into the index value of the data to be stored.

[0065] The characteristic trend data can be obtained by fitting a large amount of historical data to the distribution. For example, the characteristic trend data includes a function, where the function value is a certain feature in the index features, and the independent variables are other features in the index features besides the aforementioned feature.

[0066] S205, Sort the data to be stored and the target data according to the index value of the data to be stored and the index value of the target data associated with the storage space.

[0067] The target data is indexed in the same way. Based on these index values, the data to be stored and the target data are sorted. Specifically, the data can be sorted according to the magnitude of the index values.

[0068] S206, Store the data to be stored in the storage space according to the sorting result.

[0069] You can insert the data to be stored into the storage space where the target data is already pre-stored, based on the sorting result. Alternatively, you can store the data to be stored and the target data sequentially in the storage space, based on the sorting result.

[0070] Optionally, the feature trend data includes a feature trend line, and the method further includes: fitting a feature trend line based on the representation data corresponding to each of the target data; determining the index value of the data to be stored based on the representation data and feature trend data corresponding to the data to be stored includes: mapping the representation data corresponding to the data to be stored onto the feature trend line to obtain the mapping point corresponding to the data to be stored; and determining the distance between the standard point and the mapping point corresponding to the data to be stored as the index value of the data to be stored.

[0071] Feature trend data includes a feature trend line, which describes the linear variation of the feature distribution of the target data. A multi-dimensional coordinate system is established based on the index features. The feature trend line is represented by a linear equation within this multi-dimensional coordinate system. Characteristic data can be mapped to multi-dimensional coordinate points. Based on the characteristic data corresponding to the target data, the parameters in the linear equation are solved to obtain the linear equation with known parameters, which is then determined as the feature trend line. The feature trend line can be obtained by fitting only the characteristic data corresponding to the target data, or it can be obtained by fitting both the characteristic data corresponding to the target data and the characteristic data corresponding to the data to be stored.

[0072] Mapping points are points on the characteristic trend line projected onto the representative data. These mapping points are used to map data from the same storage space onto a specific dimension to quantify the data's characteristics. Mapping the representative data onto the characteristic trend line involves establishing a perpendicular line passing through the representative data and the characteristic trend line. The intersection of this perpendicular line and the characteristic trend line is the mapping point corresponding to the data to be stored.

[0073] A standard point is a fixed point in a multidimensional coordinate system, used to quantize the mapping point. An example standard point is the origin of the multidimensional coordinate system. The distance between the standard point and the mapping point is determined as the index value of the data to be stored, and the index features of the data to be stored are quantized.

[0074] By fitting the characteristic trend line to the corresponding representation data of the target data, the characteristic distribution pattern of the target data can be accurately represented. The representation data of the data to be stored is then mapped onto this characteristic trend line. The distance between the mapped point and the standard point is calculated to quantify the index features of the data to be stored. The index feature information is reduced to one dimension and quantized to enable subsequent data sorting, improve the accuracy of sorting, and improve query efficiency based on one-dimensional information.

[0075] The data to be stored is spatiotemporal data, and the index features include at least one of the following: spatial features, temporal features, and at least one type of attribute feature. Spatiotemporal data is typically queried from both time and spatial dimensions.

[0076] Optionally, the index features of the data to be stored include spatial features and / or at least one type of attribute feature.

[0077] Spatial features are used to describe the spatial location of data, such as using latitude and longitude or coordinates. Attribute features are used to describe the attributes of the data. Attribute features can be understood as additional labeling information attached to the data to be stored. For example, attribute features include at least one of the following: the name of the data to be stored, the corresponding building or product's functional service, and the city to which it belongs. For example, if the data to be stored is spatiotemporal data, spatial features are the spatial information of the spatiotemporal data, and attribute features are other identifiable information about the spatiotemporal data besides spatiotemporal information. Attribute features can be customized, and at least one type of attribute feature can be configured to add index features to the data to be stored.

[0078] By configuring the index features of the data to be stored, the richness and representativeness of the index features can be increased, thereby enabling sequential storage and index creation, and improving the accuracy of index queries.

[0079] According to the technical solution disclosed herein, multiple features are fused to obtain the representation data corresponding to the data to be stored. Based on the feature trend data and the representation data, the index value of the data to be stored is determined. Based on the index value of the target data, the data to be stored and the target data are sorted. The data to be stored is then stored based on the sorting result. This method can fuse multiple features into one representation data, thereby reducing the dimensionality of multidimensional index information. The index value is then determined based on the dimensionality-reduced data to sort the data to be stored. This method can reduce the dimensionality of the index, while maintaining the richness of the index content, and simultaneously improving query efficiency and query performance.

[0080] Figure 3 This is a flowchart of another data indexing method disclosed in this embodiment, which is further optimized and extended based on the above technical solution and can be combined with the above optional implementation methods. The storage space includes the target data directory. The step of sorting and storing the data to be stored and the target data associated with the same storage space is specifically as follows: obtaining the generation time of the data to be stored and querying the target data directory corresponding to the generation time, wherein the generation time of the target data and the generation time of the data to be stored belong to the same time period; under the target data directory, sorting and storing the data to be stored and the target data associated with the same storage space according to the index characteristics of the data to be stored.

[0081] S301, Obtain the data to be stored.

[0082] S302, Extract the index features of the data to be stored.

[0083] S303, obtain the generation time of the data to be stored, and query the target data directory corresponding to the generation time. The generation time of the target data and the generation time of the data to be stored belong to the same time period, and the storage space includes the target data directory.

[0084] The generation time of the data to be stored refers to the time when the data was generated, which is usually determined at the acquisition device and is unrelated to the storage time. The data directory is used to store multiple data sets generated within a specific time period. The same storage space can refer to the same data directory. The target data directory corresponding to the generation time is used to store the data corresponding to that generation time. Target data and data to be stored are stored in the same target data directory. Therefore, different data can be partitioned and stored according to their generation time, specifically by storing them in separate directories. This achieves time-based data classification, essentially establishing a time index. Data directories can be determined based on time, and queries can be performed within these directories to retrieve the corresponding data. This allows for comprehensive queries combining time with other index features, improving query accuracy.

[0085] S304, under the target data directory, according to the index characteristics of the data to be stored, the target data associated with the same target data directory are sorted and stored.

[0086] The target data and the data to be stored are sorted and stored in the same target data directory.

[0087] S305, an index for the data to be stored is established based on the index characteristics and sorted storage location of the data to be stored.

[0088] To reduce storage space usage, you can configure an expiration time and delete the target data directory corresponding to the expiration time.

[0089] As in the previous example, the data to be stored is spatiotemporal data, and its index features include spatial features and / or at least one type of attribute feature. This allows for differentiated storage based on generation time, achieving a temporal index, and the creation of a spatial index within the same data directory, i.e., the same time period, achieving a spatial index. This balances efficient query performance across both time and space. Furthermore, the index features include at least one type of attribute feature, and custom data fields can be defined to enable custom indexing, increasing the query scope and improving query flexibility.

[0090] Optionally, the data indexing method further includes: obtaining the target index directory corresponding to the generation time; and storing the index data of the data to be stored in the target index directory.

[0091] The index directory is used to store indexes of multiple data items generated within a specific time period. The index directory is consistent with the data directory; that is, for a given time period, there is one data directory and one index directory. The index data can specifically include index features and the correspondence between these features and the data to be stored. The correspondence describes the address of the data to be stored that the index feature points to.

[0092] By storing index data separately according to its generation time, the index directory can be queried by time during indexing, and further data queries can be performed within the index directory to ultimately find the corresponding data. This enables comprehensive queries based on time and other features, improving query accuracy.

[0093] Optionally, the data index creation method further includes: obtaining the expiration time; and deleting the target data directory corresponding to the expiration time and the target index directory corresponding to the expiration time.

[0094] Expiration times are used to clear redundant data and free up storage resources. Based on the time period and expiration time corresponding to the data directory, the system checks whether the data directory and index directory meet the expiration conditions. Data directories that meet the expiration conditions are identified as the data directories corresponding to the expiration time and deleted; similarly, index directories that meet the expiration conditions are identified as the index directories corresponding to the expiration time period and deleted.

[0095] For example, the expiration condition could be that the time period corresponding to the directory (data directory and index directory) is within the expiration time range, or that the time period corresponding to the directory is before the expiration time. Other conditions can also be configured, which are not specifically limited.

[0096] By configuring expiration times and deleting the target data directory and target index directory corresponding to the expiration times, file-level deletion can be performed directly to quickly clean up data without having to modify file content. This greatly reduces the overhead of deletion operations. Furthermore, data and corresponding index information can be deleted simultaneously, freeing up more storage resources, deleting more redundant data, and improving the utilization rate of storage resources.

[0097] Optionally, the data to be stored includes: map spatiotemporal data.

[0098] Spatiotemporal data of a map can be data about points on a map, each point possessing both temporal and spatial information. Time refers to the time of creation, such as the creation time of the object represented by that point. Space refers to the location of the point, such as latitude and longitude, or spatial coordinates.

[0099] Geographic information services generate a large amount of spatial data, and in many scenarios, this spatial data carries a timestamp. For example, in map services, vehicle coordinates change over time. This spatial data with time attributes is called spatiotemporal data. It should be noted that the acquisition of the data to be stored is authorized by the user and is legal and compliant.

[0100] By defining the data to be stored as map spatiotemporal data, the application scenarios of map services can be expanded, and the accuracy and efficiency of data queries related to map services can be improved.

[0101] According to the technical solution disclosed herein, by storing data belonging to the same time period in the same data directory according to the generation time, different data can be stored separately according to the generation time. This allows for indexing by querying the data directory by time, performing further queries within the data directory, and retrieving the corresponding data. This enables comprehensive querying based on time and other index features, thereby improving query accuracy.

[0102] Figure 4 This is a flowchart of a data index creation method disclosed in an embodiment of this disclosure. This embodiment can be applied to situations where stored data is queried. The method of this embodiment can be executed by a data index creation device, which can be implemented in software and / or hardware and specifically configured in an electronic device with a certain data processing capability. The electronic device can be a client device or a server device. Client devices include mobile phones, tablet computers, vehicle terminals, and desktop computers.

[0103] S401, a data query request has been received.

[0104] The data query request is used to query data on the disk of the index created by the index creation method provided in this disclosure.

[0105] S402, Extract query features from the data query request.

[0106] A data query request includes query features of the data to be retrieved. Query features are used to query data, specifically to determine indexed data, thereby finding the data's storage location and retrieving it. Query features correspond to index features. Query features are the same as index features, or may only include a portion of the index features. For example, index features include spatial features, A-type attribute features, B-type attribute features, and C-type attribute features. Query features may include spatial features, A-type attribute features, B-type attribute features, and C-type attribute features. Alternatively, query features may also include spatial features and B-type attribute features. In a specific example, query features include spatial features, such as the range of coordinates a-c, and B-type attribute features, such as city XX. Features that exist in the index features but not in the query features can be empty and assigned a value of 0, etc.

[0107] S403, according to a pre-established index, the queried data is continuously read at the position corresponding to the query feature, wherein the index is established by a data indexing method as described in any embodiment of this disclosure.

[0108] When data is stored in a sequential order, and the query characteristic is a range query, it is usually read from the range, and the read data is stored contiguously. This allows data to be read continuously at the positions corresponding to the query characteristics to retrieve the queried data. It should be noted that the continuously read data may not be the data that the data query request is looking for. The continuously read data can be further queried in memory based on the query characteristics to improve the accuracy of the query.

[0109] The processing of query features can refer to the index feature processing method. The specific query method is as follows: multiple features in the query features are fused to obtain the fused result, and the range of query values ​​is determined based on the feature trend data. The range of storage locations is determined based on the range of query values, and data is continuously read within the storage location range.

[0110] The query value corresponds to the index value. The index value is generated by processing the query features accordingly, using the method of generating index features to produce the query value. Furthermore, multiple data points can be queried, resulting in multiple query values ​​and forming a range of query values. Alternatively, if the query feature represents a range of values, the calculated query value will also be a range of query values.

[0111] Optionally, the step of continuously reading the queried data at the location corresponding to the query feature according to a pre-established index includes: extracting the query time range from the data query request; and continuously reading the queried data at the location corresponding to the query feature in the data directory corresponding to the query time range according to the pre-established index.

[0112] The data query request retrieves spatiotemporal data, allowing for the extraction of both time and space information. This spatiotemporal data is stored in different data directories distributed according to its generation time. The query request can be used to extract the query time range, determine the corresponding data directory, and then retrieve data within each directory based on query characteristics, enabling multi-dimensional queries.

[0113] By first querying the time range to determine the data directory, and then further querying based on query characteristics within different data directories, multi-dimensional data querying can be achieved, improving query accuracy and narrowing large-scale queries to small-scale queries within a few data directories, thereby improving query efficiency.

[0114] According to the technical solution of this disclosure, the query is performed in the sequential storage space based on the range of query values ​​calculated according to the query characteristics. This enables the continuous acquisition of query data during data query, reduces the input and output overhead of the query, and thus improves the query performance.

[0115] Figure 5 This is a scenario diagram illustrating another data indexing and querying method disclosed in this embodiment. The data indexing system may include a log layer, a cache layer, an index layer, and a data layer. Each layer can be understood as a module that implements a specific function. The data stored and queried in the data indexing system is spatiotemporal data.

[0116] The first layer is the log layer, which is responsible for storing the write-ahead log. The write-ahead log refers to the raw data that users write to the data indexing system. When the data indexing system receives a user's write request, it first writes the user's data to the log. This way, if the system crashes or restarts, the user's data can be recovered from the write-ahead log. Multiple data entries can be stored in the same write-ahead log, or they can be stored in different write-ahead logs, for example... Figure 5 As shown, write to log 1, write to log 2... write to log N.

[0117] The second layer is the caching layer, which is further divided into write cache and query cache. Write cache means that data written to the data indexing system by users is not directly written to disk, but first written to memory. When the data in memory exceeds a certain size, or when no new data has been written for a certain period, the data in memory is then written to disk in batches. This reduces the number of disk writes, improving write performance. Furthermore, this approach allows for the compression of batch data before writing to disk, thereby reducing disk space usage. Query cache, on the other hand, stores the user's query results in memory. When encountering duplicate query conditions, there's no need to query the disk again; instead, the results in memory are returned directly.

[0118] The third layer is the index layer. The index layer is primarily responsible for filtering primary keys that meet the user's query conditions, defining them as index features. These primary keys then point to the actual spatiotemporal data, allowing the index to locate the data the user is querying. Index features can include tag features and spatial features. Tag features refer to tags in the spatiotemporal data, such as device identifiers (IDs). By creating a tag index for the device ID, the specified data can be quickly retrieved using the user-specified device ID. Spatial features refer to spatial fields in the user data, such as vehicle coordinates (points are two-dimensional points, Point(x,y)). By creating a spatial index for these coordinates, spatial queries, such as intersection or containment, can be used to locate the corresponding spatiotemporal data. The data in the index layer and the data in the data layer are linked one-to-one according to their generation time. That is, if the underlying spatiotemporal data is sharded according to 24-hour intervals, its index data is also sharded according to 24-hour intervals, ensuring that the spatiotemporal data and index data are deleted synchronously.

[0119] The fourth layer is the data layer, which stores sequentially ordered spatiotemporal data. This data is grouped according to its generation time. For example, the generation time of the spatiotemporal data can be divided into: generation time 1 = 0-100, generation time 2 = 100-200, and so on, with generation time N = (N-1)*100 to N*100. In a specific example, spatiotemporal data with a generation time belonging to generation time 2 will be stored together, usually in the same directory. Corresponding to the data in the index layer, the index data (i.e., spatial indexes and label indexes in the graph) is grouped and stored according to the spatiotemporal data it points to. This allows for quick data cleanup when data expires, through direct file-level deletion. For example, deleting spatiotemporal data from 0-100 can be achieved by directly deleting the directory corresponding to generation time 1, without needing to modify the file content, thus greatly reducing the overhead of deletion operations.

[0120] The index builder is a module used to construct the index layer based on the data from the data layer. This index includes tag indexes and spatial indexes. Tag indexes are created synchronously when user data is written and then written to the index layer. This is because tag queries are the most basic capability and therefore must be created in real-time. For spatial indexes, a build mode can be configured, including real-time build, custom build, or never build. Real-time build means that the spatial index is created when data is written and then synchronously written to the index layer. Custom build means that the spatial index is created according to user-configured rules, such as performing a spatial index build for all data throughout the day once a day. Never build means that no spatial index is created at all. The reason for setting these three spatial index build modes is that not all user data requires real-time spatial index creation. For example, a user might write data and then perform a spatial query one day later. In this case, the custom spatial index build method can create the spatial index through a daily task, thus eliminating the overhead of building the spatial index during real-time writing and further improving write performance.

[0121] It should be noted that, without constructing a spatial index, all spatial features can be set to 0. This allows label indexing to still be performed even when spatial features are missing.

[0122] The data cleaner is a module used for data cleanup, primarily for deleting data with expiration dates. Since both the index layer and data layer are stored in shards based on the generation time of the spatiotemporal data, when all data in the same directory has expired, it can be deleted entirely. Deletion doesn't directly erase the file; instead, it sets an expired status for the file, preventing new queries from accessing it and unaffecting ongoing queries. Once all queries on the file have been completed, the file is permanently deleted.

[0123] The scheme adopted in this embodiment is based on time-series data storage and integrates spatial indexing technology, thereby providing efficient spatial query capabilities under the premise of high write throughput.

[0124] The core of a data indexing system consists of the spatial index and the tag index in the index layer, but these two types of indexes cannot be directly combined. In existing query methods, for a user's query, either both indexes are queried simultaneously, and the intersection is used to obtain a set of primary keys, which are then used to query the data layer; or only one index is selected, and after retrieving the data, another condition is used to filter it in memory. Neither of these solutions utilizes both indexes simultaneously. However, this embodiment uses an index analysis task to reconstruct the index layer and the data layer. The first step in the reconstruction is to map the index to an n-dimensional point.

[0125] like Figure 6As shown, taking a single label as an example, since the number of labels is finite, features can be encoded and mapped. For example, in the figure, the value "XX" is mapped to the number 1. The number 1 and the spatial feature can form a two-dimensional point. If there are multiple features, they form an n-dimensional point. Therefore, the features of a batch of data can actually be viewed as a series of n-dimensional points, and these n-dimensional points can be processed using PCA (Principal Component Analysis), as follows. Figure 7 As shown, the horizontal axis represents spatial features, and the vertical axis represents the label feature "city." We can then perform a comprehensive analysis of the data within a period (a time interval) to calculate the line that best reflects the distribution trend of the points, which is then identified as the feature trend line. Each point is mapped onto this line, and the length of the line marked "fusion_index_val" in the diagram can be used as the index value for the data. This index value is then used to reorder the data in the data layer, resulting in the reconstructed data. The example in the diagram can be generalized to the case of n label values, in which case an n-dimensional line will be obtained, but the principle remains the same.

[0126] After obtaining the reconstructed data, queries can be performed using the new fused index. The query scheme is as follows: Figure 8 As shown, the line segment parallel to the horizontal axis represents the query target. On the city axis, this means querying data with city = 10; on the position axis, it's between 0.3 and 0.4, indicating a query within this spatial range. Mapping this line to the red axis yields the range of the index values ​​to be queried. Data retrieved through this range may include some non-target data, requiring secondary filtering using query conditions. The biggest advantage of this query scheme is the very low number of I / O operations. Because the index features are merged into a single value, querying the index requires only one I / O operation. After retrieving the index value, the data layer is sorted according to the new index value, so range queries also require only one I / O operation. Therefore, this scheme is very friendly to traditional mechanical hard drives with slow I / O speeds.

[0127] As we can see from the above introduction, the effectiveness of a query strategy is related to the distribution of the data. The closer the data distribution is to the calculated red line, the higher the accuracy of the query filtering. In actual engineering, geographical location and specific tags are generally strongly correlated, such as the city and spatial location in the figure. In this scenario, the query strategy will have better results. That is, attribute features and spatial features have a strong correlation. For example, the attribute feature is a place name. City XX has place A, but city YY does not have place A. Therefore, place A can only exist in city XX and will not exist in other cities. Thus, the place name of place A corresponds to the spatial location of city XX and has no relation to the spatial locations of other cities. Therefore, there is a strong correlation between place name and space, that is, there is a strong correlation between attribute features and spatial features.

[0128] This disclosure is based on a time-series database storage scheme, thus possessing powerful write capabilities. Furthermore, by configuring a suitable spatial index construction mode, the overhead of real-time writes can be further reduced, improving write performance. To quickly delete data, both index data and spatiotemporal data are stored in shards. The data cleaner cleans data through file deletion, which is much more efficient than the usual method of deleting data by modifying files. Simultaneously, spatial indexes are constructed using an index builder and a spatial index construction mode, balancing write performance and index creation. With the support of spatial indexes, the data to be queried can be efficiently filtered based on spatial query conditions without requiring spatial calculations on large amounts of data in memory. Moreover, the tag index and spatial index are merged into a numerical index through a series of mathematical operations, transforming the joint query of the tag index and spatial index into a one-dimensional index query, reducing query I / O overhead and thus improving query performance.

[0129] According to embodiments of this disclosure, Figure 9 This is a structural diagram of a data indexing device according to an embodiment of this disclosure. This embodiment is applicable to data storage and indexing. The device is implemented in software and / or hardware and is specifically configured in an electronic device with certain data processing capabilities.

[0130] like Figure 9 A data indexing device 900, as shown, includes: a data acquisition module 901, an index feature extraction module 902, a sorting and storage module 903, and an indexing module 904; wherein,

[0131] Data acquisition module 901 is used to acquire data to be stored;

[0132] Index feature extraction module 902 is used to extract the index features of the data to be stored;

[0133] The sorting and storage module 903 is used to sort and store the target data associated with the data to be stored and the same storage space according to the index characteristics of the data to be stored;

[0134] The index building module 904 is used to build an index of the data to be stored based on the index characteristics and the sorted storage location of the data to be stored.

[0135] According to the technical solution disclosed herein, the data to be stored and the target data that need to be stored in the same storage space are sorted according to the index characteristics and stored sequentially, so as to continuously retrieve the data to be queried during data query, reduce the input and output overhead of the query, and thus improve the query performance.

[0136] Furthermore, the sorting and storage module 903 includes: a characterization data detection unit, used to fuse multiple features in the index features of the data to be stored to obtain characterization data corresponding to the data to be stored; an index value detection unit, used to determine the index value of the data to be stored based on the characterization data and feature trend data corresponding to the data to be stored; a data sorting unit, used to sort the data to be stored and the target data based on the index value of the data to be stored and the index value of the target data associated with the storage space; and a data storage unit, used to store the data to be stored in the storage space according to the sorting result.

[0137] Furthermore, the feature trend data includes a feature trend line, and the data indexing device further includes: a feature trend fitting subunit, used to fit a feature trend line based on the representation data corresponding to each of the target data; the index value detection unit includes: a representation data mapping subunit, used to map the representation data corresponding to the data to be stored onto the feature trend line to obtain the mapping point corresponding to the data to be stored; and an index value calculation subunit, used to determine the distance between the standard point and the mapping point corresponding to the data to be stored as the index value of the data to be stored.

[0138] Furthermore, the index features of the data to be stored include spatial features and / or at least one type of attribute feature.

[0139] Furthermore, the storage space includes the target data directory, and the index building module 904 includes: a data partitioning determination unit, used to obtain the generation time of the data to be stored and query the target data directory corresponding to the generation time, wherein the generation time of the target data and the generation time of the data to be stored belong to the same time period; and a data partitioning storage unit, used to sort and store the target data associated with the data to be stored and the data to be stored in the same storage space under the target data directory, according to the index characteristics of the data to be stored.

[0140] Furthermore, the data indexing device further includes: an index partitioning determination unit, used to obtain the target index directory corresponding to the generation time; and an index partitioning storage unit, used to store the index data of the data to be stored in the target index directory.

[0141] Furthermore, the data indexing device further includes: an expiration time acquisition unit for acquiring the expiration time; and an expired data deletion unit for deleting the target data directory corresponding to the expiration time and the target index directory corresponding to the expiration time.

[0142] Furthermore, the data to be stored includes: map spatiotemporal data.

[0143] The aforementioned data indexing apparatus can execute the data indexing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the data indexing method.

[0144] According to embodiments of this disclosure, Figure 10 This is a structural diagram of a data query device according to an embodiment of this disclosure. This embodiment is applicable to situations where stored data is queried. The device is implemented using software and / or hardware and is specifically configured in an electronic device with a certain data processing capability.

[0145] like Figure 10 A data query device 1000, as shown, includes: a query request acquisition module 1001, a query feature extraction module 1002, and a data query module 1003; wherein,

[0146] The query request acquisition module 1001 is used to receive data query requests;

[0147] The query feature extraction module 1002 is used to extract query features from the data query request;

[0148] The data query module 1003 is used to continuously read the queried data at the positions corresponding to the query features according to a pre-established index, wherein the index is established by the data index establishment method as described in any one of claims 1-8.

[0149] According to the technical solution of this disclosure, the query is performed in the sequential storage space based on the range of query values ​​calculated according to the query characteristics. This enables the continuous acquisition of query data during data query, reduces the input and output overhead of the query, and thus improves the query performance.

[0150] Furthermore, the data query module 1003 includes: a query time range determination unit, used to extract the query time range from the data query request; and a time directory query unit, used to continuously read the queried data at the positions corresponding to the query features according to a pre-established index.

[0151] The data query device described above can execute the data query method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the data query method.

[0152] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0153] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0154] Figure 11 A schematic area diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0156] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as data indexing methods or data querying methods. For example, in some embodiments, the data indexing methods or data querying methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the data indexing methods or data querying methods described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform a data indexing method or a data query method.

[0158] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or area diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0162] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0163] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0164] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for creating a data index, comprising: Retrieve the data to be stored; Extract the index features of the data to be stored; Based on the index characteristics of the data to be stored, the target data associated with the same storage space are sorted and stored. An index is created for the data to be stored based on its index characteristics and the location of its sorted storage. The step of sorting and storing target data associated with the same storage space based on the index characteristics of the data to be stored includes: Multiple features in the index features of the data to be stored are fused to obtain the representation data corresponding to the data to be stored. The index value of the data to be stored is determined based on the characterization data and characteristic trend data corresponding to the data to be stored. Sort the data to be stored and the target data according to the index value of the data to be stored and the index value of the target data associated with the storage space; The data to be stored is stored in the storage space according to the sorting result; The characteristic trend data includes a characteristic trend line, and the method further includes: Based on the characterization data corresponding to each of the target data, a characteristic trend line is fitted to obtain the line. The step of determining the index value of the data to be stored based on the characterization data and feature trend data corresponding to the data to be stored includes: Map the characterization data corresponding to the data to be stored onto the feature trend line to obtain the mapping point corresponding to the data to be stored. The distance between the standard point and the mapping point corresponding to the data to be stored is determined as the index value of the data to be stored.

2. The method according to claim 1, wherein, The index features of the data to be stored include spatial features and / or at least one type of attribute feature.

3. The method according to claim 1, wherein, The storage space includes the target data directory. The step of sorting and storing the target data associated with the data to be stored and the data to be stored in the same storage space according to the index characteristics of the data to be stored includes: Obtain the generation time of the data to be stored, and query the target data directory corresponding to the generation time, wherein the generation time of the target data and the generation time of the data to be stored belong to the same time period; Under the target data directory, the target data to be stored and associated with the same storage space are sorted and stored according to the index characteristics of the data to be stored.

4. The method according to claim 3, further comprising: Obtain the target index directory corresponding to the generation time; The index data of the data to be stored is stored in the target index directory.

5. The method according to claim 4, further comprising: Get the expiration time; Delete the target data directory and the target index directory corresponding to the expiration time.

6. The method according to claim 1, wherein, The data to be stored includes: map spatiotemporal data.

7. A data query method, comprising: A data query request has been received; Extract query features from the data query request; Based on a pre-established index, data is continuously read at the positions corresponding to the query features, wherein the index is established by the data indexing method as described in any one of claims 1-6.

8. The method according to claim 7, wherein, The step of continuously reading the queried data at the positions corresponding to the query features based on a pre-established index includes: Extract the query time range from the data query request; In the data directory corresponding to the query time range, according to the pre-established index, the query data is continuously read at the positions corresponding to the query features.

9. A data indexing device, comprising: The data acquisition module is used to acquire data to be stored. The index feature extraction module is used to extract the index features of the data to be stored. The sorting and storage module is used to sort and store the target data associated with the data to be stored and the same storage space according to the index characteristics of the data to be stored; An index building module is used to build an index for the data to be stored based on the index characteristics and the sorted storage location of the data to be stored. The sorting and storage module includes: The characterization data detection unit is used to fuse multiple features in the index features of the data to be stored to obtain the characterization data corresponding to the data to be stored. An index value detection unit is used to determine the index value of the data to be stored based on the characterization data and feature trend data corresponding to the data to be stored. A data sorting unit is used to sort the data to be stored and the target data according to the index value of the data to be stored and the index value of the target data associated with the storage space; A data storage unit is used to store the data to be stored in the storage space according to the sorting result; The characteristic trend data includes a characteristic trend line, and the device further includes: The feature trend fitting subunit is used to fit a feature trend line based on the characterization data corresponding to each of the target data. The index value detection unit includes: The characterization data mapping subunit is used to map the characterization data corresponding to the data to be stored onto the feature trend line to obtain the mapping point corresponding to the data to be stored. The index value calculation subunit is used to determine the distance between the standard point and the mapping point corresponding to the data to be stored as the index value of the data to be stored.

10. The apparatus according to claim 9, wherein, The index features of the data to be stored include spatial features and / or at least one type of attribute feature.

11. The apparatus according to claim 9, wherein, The storage space includes the target data directory, and the index building module includes: A data partitioning determination unit is used to obtain the generation time of the data to be stored and query the target data directory corresponding to the generation time, wherein the generation time of the target data and the generation time of the data to be stored belong to the same time period; A data partitioning storage unit is used to sort and store target data associated with the data to be stored and the same storage space under the target data directory, according to the index characteristics of the data to be stored.

12. The apparatus of claim 11, further comprising: The index partitioning determination unit is used to obtain the target index directory corresponding to the generation time; An index partition storage unit is used to store the index data of the data to be stored in the target index directory.

13. The apparatus of claim 12, further comprising: The expiration time acquisition unit is used to acquire the expiration time; The expired data deletion unit is used to delete the target data directory and the target index directory corresponding to the expiration time.

14. The apparatus according to claim 9, wherein, The data to be stored includes: map spatiotemporal data.

15. A data query device, comprising: The query request acquisition module is used to receive data query requests; The query feature extraction module is used to extract query features from the data query request; The data query module is used to continuously read the queried data at the positions corresponding to the query features according to a pre-established index, wherein the index is established by the data indexing method as described in any one of claims 1-6.

16. The apparatus according to claim 15, wherein, The data query module includes: A query time range determination unit is used to extract the query time range from the data query request; The time catalog query unit is used to continuously read the query data at the positions corresponding to the query features according to a pre-established index.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data indexing method of any one of claims 1-6 or the data query method of any one of claims 7-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the data indexing method according to any one of claims 1-6 or the data query method according to any one of claims 7-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the data indexing method according to any one of claims 1-6 or the data querying method according to any one of claims 7-8.

Images