A distributed big data storage system and method

Abstract

The present application provides a distributed big data storage system and method. The method determines the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node; performs polygon retrieval judgment on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, and obtains the spatiotemporal storage dimension characteristics corresponding to each historical economic data in the spatiotemporally associated historical data set; determines the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and uses the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval. The method can store and retrieve the data based on the spatiotemporal storage dimension characteristics of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval of the distributed storage node.

Description

Technical Field

[0001] The present application relates to the technical field of data storage and data processing, and more specifically, to a distributed big data storage system and method. Background Art

[0002] In distributed storage systems, massive amounts of information, such as order transaction data and user payment data, are typically split and stored in multiple independent databases. This type of cloud computing-based distributed information storage strategy not only effectively addresses the data distribution and computing resource scheduling issues that arise when merging massive information files, but also leverages the scalability of distributed file systems to expand the reach of data storage and computing capabilities.

[0003] In the existing technology, the call of distributed storage information usually depends on the spatiotemporal correlation of data, and retrieval algorithms based on spatiotemporal polygons are widely used for data query. Since polygonal areas usually have characteristics such as wide boundaries and irregular shapes, they often need to be accurately expressed in the form of vectors containing a large number of vertices. In actual queries, in order to complete a polygon range query, the system often needs to traverse hundreds of millions or even billions of spatiotemporal data points, which brings huge computational overhead. Especially when facing the retrieval task of massive distributed storage data, this high computing cost has become a key bottleneck restricting the application efficiency and scalability of distributed storage systems. Therefore, how to effectively improve the retrieval efficiency of distributed storage information has become a technical problem that urgently needs to be solved. Summary of the Invention

[0004] The present application provides a distributed big data storage system and method, which can store and retrieve data based on the spatiotemporal storage dimension characteristics of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval of distributed storage nodes.

[0005] In a first aspect, the present application provides a distributed big data storage method, which can be executed by a network device, or by a chip configured in the network device, and the present application does not limit this.

[0006] Specifically, the method includes:

[0007] Acquire multiple distributed storage nodes, and determine the characteristic distances corresponding to the economic collection data to be stored and the corresponding distributed storage nodes according to the characteristic labels corresponding to the economic collection data to be stored;

[0008] Allocating the economic collection data to a distributed storage node with the smallest characteristic distance, and determining the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node;

[0009] Performing polygon search and determination on the historical economic data in the distributed storage node according to the spatiotemporal polygon search granularity to obtain a spatiotemporal associated historical data set, and obtaining spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal associated historical data set;

[0010] The storage location of the economic collection data is determined based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and the spatiotemporal storage dimension characteristics of the economic collection data are used as spatiotemporal retrieval information for distributed retrieval.

[0011] In combination with the first aspect, in certain implementations of the first aspect, in the process of determining the characteristic distances corresponding to the economic collection data and each distributed storage node, the characteristic values on each spatiotemporal characteristic dimension in the characteristic labels corresponding to the economic collection data to be stored are converted into distance detection vectors, and the characteristic vectors corresponding to the storage center data of each distributed storage node are obtained, and the Euclidean distance values between the distance detection vectors and each characteristic vector are used as the characteristic distances corresponding to the economic collection data and each distributed storage node.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, determining the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node specifically includes:

[0013] Obtain the historical economic data in the distributed storage node and build a historical economic data space;

[0014] In the spatiotemporal feature dimension space, the historical economic data space is obtained, and the difference between the maximum eigenvalue and the minimum eigenvalue corresponding to each spatiotemporal feature dimension of the historical economic data space is obtained as the data range corresponding to each spatiotemporal feature dimension, and the ratio between the data range corresponding to each spatiotemporal feature dimension and the corresponding standard data range is used as the data density feature corresponding to each spatiotemporal feature dimension;

[0015] Based on the dimension weights corresponding to each spatiotemporal feature dimension, the data density features within each spatiotemporal feature dimension are weightedly fused to obtain the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node.

[0016] In combination with the first aspect, in certain implementations of the first aspect, polygon retrieval judgment is performed on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, which specifically includes: mapping according to the spatiotemporal polygon retrieval granularity to obtain the spatiotemporal retrieval distance corresponding to each spatiotemporal feature dimension; for any spatiotemporal feature dimension, the farthest historical economic data within the corresponding spatiotemporal retrieval distance from the economic collection data is used as polygon retrieval vertex data; and a polygon retrieval boundary is formed according to the polygon retrieval vertex data corresponding to each spatiotemporal feature dimension; the polygon retrieval boundary is extracted to form a spatiotemporally associated historical data set with all historical economic data within the boundary.

[0017] In combination with the first aspect, in certain implementations of the first aspect, in the process of obtaining the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal associated historical data set, the spatiotemporal center point distance corresponding to each historical economic data in the spatiotemporal associated historical data set is used as the spatiotemporal storage dimension feature, and the spatiotemporal center point distance is based on the preset spatiotemporal data center point and the characteristic singular value of the collection label between the historical economic data and the spatiotemporal data center point as the spatiotemporal storage dimension feature, wherein the characteristic singular value is the weighted result of the characteristic difference value of the historical economic data and the spatiotemporal data center point in each spatiotemporal feature dimension.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the spatiotemporal storage dimension characteristics of the economic collection data are used as the spatiotemporal retrieval information for distributed retrieval, and the spatiotemporal storage dimension characteristics of the economic collection data are used as the characteristic values of the incremental retrieval dimension, and spatiotemporal polygon retrieval is performed based on multiple polygon boundary retrieval conditions and incremental retrieval dimension conditions.

[0019] In combination with the first aspect, in certain implementations of the first aspect, each historical economic data in the spatiotemporal correlation historical data set is stored sequentially based on the size of spatiotemporal storage dimension characteristics.

[0020] In a second aspect, the present application provides a distributed big data storage system, which includes a data storage unit, wherein the data storage unit includes:

[0021] A storage node selection module is used to obtain multiple distributed storage nodes and determine the characteristic distances corresponding to the economic collection data to be stored and each distributed storage node according to the characteristic labels corresponding to the economic collection data to be stored;

[0022] A data processing module is used to distribute the economic collection data to the distributed storage node with the smallest feature distance, and determine the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node;

[0023] The data processing module is further configured to perform polygon retrieval on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporal associated historical data set, and acquire spatiotemporal storage dimension features corresponding to each piece of historical economic data in the spatiotemporal associated historical data set;

[0024] The data storage control module determines the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and uses the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval.

[0025] In a third aspect, the present application provides a computer terminal device, which includes a memory and a processor, wherein the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned distributed big data storage method.

[0026] In a fourth aspect, the present application provides a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned distributed big data storage method.

[0027] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0028] In a distributed big data storage system and method provided by the present application, first, multiple distributed storage nodes are obtained, and the characteristic distances corresponding to the economic collection data and each distributed storage node are determined according to the characteristic labels corresponding to the economic collection data to be stored; the economic collection data are allocated to the distributed storage node with the smallest characteristic distance, and based on the historical economic data in the distributed storage node, the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node is determined; polygon retrieval judgment is performed on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, and the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporally associated historical data set are obtained; the storage location of the economic collection data is determined based on the spatiotemporal storage dimension features corresponding to each historical economic data, and the spatiotemporal storage dimension features of the economic collection data are used as spatiotemporal retrieval information for distributed retrieval.

[0029] Therefore, it can be seen that the present application first determines the corresponding spatiotemporal polygon retrieval granularity according to the spatiotemporal characteristics of the data stored in the distributed storage node, thereby effectively adjusting the number of samples in the spatiotemporal associated historical data set, and determines the storage location of the data to be stored based on the spatiotemporal storage dimension characteristics corresponding to each adjacent historical data sample in the spatiotemporal associated historical data set, and uses the spatiotemporal storage dimension characteristics of the economically collected data as the spatiotemporal retrieval information for distributed retrieval, constructs the spatiotemporal storage dimension characteristics of the distributed storage data, and converts the polygon retrieval process that originally relied on high-overhead coordinate calculation into a two-stage process of range judgment and spatial verification of the spatiotemporal storage dimension characteristics, thereby reducing the data traversal range during spatiotemporal polygon retrieval and improving the retrieval speed of the distributed storage node.

[0030] In summary, the present application can perform storage and retrieval based on the spatiotemporal storage dimension characteristics of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval on distributed storage nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is an exemplary flow chart of a distributed big data storage method according to some embodiments of the present application;

[0032] Figure 2 is a structural diagram of a data storage unit according to some embodiments of the present application;

[0033] Figure 3 This is a structural diagram of a computer terminal device that implements a distributed big data storage method according to some embodiments of the present application. DETAILED DESCRIPTION

[0034] The present application obtains multiple distributed storage nodes, determines the characteristic distances corresponding to the economic collection data and each distributed storage node according to the characteristic labels corresponding to the economic collection data to be stored; distributes the economic collection data to the distributed storage node with the smallest characteristic distance, and determines the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node; performs polygon retrieval judgment on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, and obtains the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporally associated historical data set; determines the storage location of the economic collection data based on the spatiotemporal storage dimension features corresponding to each historical economic data, and uses the spatiotemporal storage dimension features of the economic collection data as spatiotemporal retrieval information for distributed retrieval, which can be stored and retrieved based on the spatiotemporal storage dimension features of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval of distributed storage nodes.

[0035] In order to better understand the above technical solution, the following will be combined with the accompanying drawings and specific implementation methods to describe the above technical solution in detail. Figure 1 , which is an exemplary flow chart of a distributed big data storage method according to some embodiments of the present application. The distributed big data storage method 100 mainly includes the following steps:

[0036] In step S101, a plurality of distributed storage nodes are acquired, and characteristic distances corresponding to the economic collection data to be stored and the respective distributed storage nodes are determined based on characteristic labels corresponding to the economic collection data to be stored.

[0037] Optionally, in some embodiments, the feature label includes feature values on multiple spatiotemporal feature dimensions, wherein the spatiotemporal feature dimensions include: spatial dimension, such as the spatial longitude and latitude of the collection location associated with the economic collection data, H3 / S2 code, administrative area code, etc. as the feature value of the spatial dimension; time dimension: such as the collection timestamp as the feature value of the time dimension; in some other embodiments, other dimensional features that can reflect the economic collection data information can also be used, such as the label value corresponding to the business category to which the data belongs and the type of economic activity as the spatiotemporal feature dimension, and this application does not limit this.

[0038] Optionally, in some embodiments, in the process of determining the characteristic distances corresponding to the economic collection data and each distributed storage node, the characteristic values on each spatiotemporal characteristic dimension in the characteristic labels corresponding to the economic collection data to be stored can be converted into distance detection vectors, and the characteristic vectors corresponding to the storage center data of each distributed storage node can be obtained, and the Euclidean distance values between the distance detection vectors and each characteristic vector can be used as the characteristic distances corresponding to the economic collection data and each distributed storage node.

[0039] In a specific implementation, the mean data of all historical economic data in the distributed storage node can be used as the storage center data, and the feature mean of the historical economic data in each feature dimension can be used as the vector value of the feature vector.

[0040] In step S102, the economic collection data is distributed to the distributed storage node with the smallest feature distance, and based on the historical economic data in the distributed storage node, the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node is determined.

[0041] It should be noted that the spatiotemporal polygon retrieval granularity is used to indicate the distribution density of the spatiotemporal dimension characteristics of the data stored in the distributed storage node. When the spatiotemporal polygon retrieval granularity is large, it indicates that the spatiotemporal dimension characteristics of the data stored in the distributed storage node are densely distributed. The polygonal boundaries of the spatiotemporal dimension characteristics of the data are usually more complex and sensitive in high-density areas. Therefore, it is necessary to more accurately estimate the location of the data to be stored. If only a small amount of adjacent historical data is used to estimate the feature distribution or determine the storage location, errors are likely to occur due to local fluctuations. In this case, it is necessary to increase the sample size of the data adjacent to the spatiotemporal dimension characteristics of the data to be stored, so as to smooth out local fluctuations, form a more stable feature distribution trend, reduce the retrieval difficulty of the spatiotemporal polygon retrieval, and improve the retrieval efficiency. Optionally, in some embodiments, based on the historical economic data in the distributed storage node, determining the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node specifically includes:

[0042] Obtain the historical economic data in the distributed storage node and build a historical economic data space;

[0043] In the spatiotemporal feature dimension space, the historical economic data space is obtained, and the difference between the maximum eigenvalue and the minimum eigenvalue corresponding to each spatiotemporal feature dimension of the historical economic data space is obtained as the data range corresponding to each spatiotemporal feature dimension, and the ratio between the data range corresponding to each spatiotemporal feature dimension and the corresponding standard data range is used as the data density feature corresponding to each spatiotemporal feature dimension;

[0044] Based on the dimension weights corresponding to each spatiotemporal feature dimension, the data density features within each spatiotemporal feature dimension are weightedly fused to obtain the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node.

[0045] Optionally, in some embodiments, the data density features within each spatiotemporal feature dimension are weightedly fused based on the dimensional weights corresponding to each spatiotemporal feature dimension to obtain the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node, and the weighted weights corresponding to each spatiotemporal feature dimension are calibrated to constants based on experience.

[0046] In step S103, polygon retrieval is performed on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporal associated historical data set, and the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal associated historical data set are obtained.

[0047] Optionally, in some embodiments, polygon retrieval judgment is performed on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, specifically including: mapping according to the spatiotemporal polygon retrieval granularity to obtain the spatiotemporal retrieval distance corresponding to each spatiotemporal feature dimension, for any spatiotemporal feature dimension, the farthest historical economic data within the corresponding spatiotemporal retrieval distance from the economic collection data is used as polygon retrieval vertex data, and a polygon retrieval boundary is formed according to the polygon retrieval vertex data corresponding to each spatiotemporal feature dimension, and the polygon retrieval boundary is extracted to form all historical economic data within the boundary into a spatiotemporally associated historical data set.

[0048] In specific implementation, mapping is performed according to the spatiotemporal polygon retrieval granularity to obtain the spatiotemporal retrieval distance corresponding to each spatiotemporal feature dimension. Different spatiotemporal retrieval distance levels can be mapped according to the granularity range of the spatiotemporal polygon retrieval granularity based on a preset linear mapping table.

[0049] Preferably, in some embodiments, after obtaining polygonal search vertex data, a convex hull algorithm may be used to generate a closed polygonal area as a search boundary.

[0050] Optionally, in some embodiments, in the process of obtaining the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal associated historical data set, the spatiotemporal center point distance corresponding to each historical economic data in the spatiotemporal associated historical data set is used as the spatiotemporal storage dimension feature. It should be noted that the present application sets a spatiotemporal center point for each node in the distributed system, and weightedly integrates the difference values between each historical economic data and the center point in multiple acquisition label dimensions (such as time, geography, and event type) to obtain a one-dimensional spatiotemporal storage dimension feature value, and sorts and stores all historical data in each node from small to large according to the dimension feature, and converts complex polygons into a "maximum vector radius boundary" centered on the center point for approximate filtering, so that the first retrieval condition of the polygon boundary can be set to the maximum feature difference value of the center point in each spatiotemporal feature dimension, thereby reducing the data comparison range of the spatiotemporal polygon retrieval and improving the retrieval speed of the spatiotemporal polygon retrieval.

[0051] In a specific implementation, the spatiotemporal center point distance needs to be based on a preset spatiotemporal center point, and the characteristic singular value of the collection label between the historical economic data and the spatiotemporal center point is used as the spatiotemporal storage dimension feature, wherein the characteristic singular value is the weighted result of the characteristic difference value between the historical economic data and the spatiotemporal center point in each spatiotemporal feature dimension, wherein the weighted weight corresponding to each spatiotemporal feature dimension is calibrated as a constant based on experience. The spatiotemporal storage dimension feature reflects the deviation distance between the historical economic data and the spatiotemporal center point. The historical economic data in the spatiotemporally associated historical data set are stored sequentially according to the size of the spatiotemporal storage dimension feature, thereby ensuring that a unilateral search is performed based on the polygon data range during the spatiotemporal polygon search process. For example, when searching for the first randomly selected historical economic data, if it is determined that the spatiotemporal storage dimension feature of the historical economic data is higher than the maximum distance value between the corresponding polygon search boundary and the spatiotemporal center point, there is no need to continue traversing the entire distributed storage node for search and comparison. Only the storage order of the spatiotemporal storage dimension feature is required to search the preceding economic storage data, which greatly reduces the search range of the spatiotemporal polygon search and improves the speed of the search results.

[0052] In step S104, the storage location of the economic collection data is determined based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and the spatiotemporal storage dimension characteristics of the economic collection data are used as spatiotemporal retrieval information for distributed retrieval.

[0053] It should be noted that this application compresses the distance deviation values between historical data and the central points of the spatiotemporal data into spatiotemporal storage dimension features, and stores them in index order based on the features, thereby converting the complex spatiotemporal polygon retrieval problem into one-way feature value range filtering, thereby greatly reducing the retrieval complexity and system computing overhead.

[0054] Optionally, in some embodiments, determining the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data specifically includes: obtaining the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and determining the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to the economic collection data and the sequential storage rules of the spatiotemporal storage dimension characteristics.

[0055] Optionally, in some embodiments, the spatiotemporal storage dimension characteristics of the economic collection data are used as the spatiotemporal retrieval information for distributed retrieval, and the spatiotemporal storage dimension characteristics of the economic collection data are used as the characteristic values of the incremental retrieval dimension, and spatiotemporal polygon retrieval is performed according to multiple polygon boundary retrieval conditions and incremental retrieval dimension conditions.

[0056] In a specific implementation, for example, the boundary conditions of the spatiotemporal polygon retrieval algorithm are set as follows: time range: January 2024 to December 2024, spatial boundary: irregular area composed of polygon vertices (such as city, region), type condition: event type, industry classification, label, etc., and the spatiotemporal storage dimension characteristic value is set to a one-dimensional numerical dimension that approximately represents the offset distance of the data relative to the center point of the spatiotemporal data. The maximum allowable characteristic value of the incremental retrieval dimension (i.e., the maximum characteristic distance between the center point and the boundary) is determined based on the boundary conditions of the spatiotemporal polygon retrieval algorithm. The constraint conditions of the spatiotemporal storage dimension characteristic value are added to the spatiotemporal polygon retrieval algorithm to construct a joint retrieval rule, thereby improving the retrieval speed of the spatiotemporal polygon retrieval. If the minimum value of the spatiotemporal storage dimension characteristic value in a distributed node is higher than the maximum allowable characteristic value, it means that all data in the node is greater than the query characteristic value, and the distributed storage node can be directly skipped.

[0057] In addition, in another aspect of the present application, in some embodiments, the present application provides a distributed big data storage system, the device includes a data storage unit, reference Figure 2 , which is a schematic diagram of exemplary hardware and / or software structure of a data storage unit according to some embodiments of the present application. The data storage unit 200 includes: a storage node selection module 201, a data processing module 202, and a data storage control module 203, which are described as follows:

[0058] The storage node selection module 201 is used to obtain multiple distributed storage nodes and determine the characteristic distances between the economic collection data to be stored and each distributed storage node according to the characteristic labels corresponding to the economic collection data to be stored;

[0059] The data processing module 202 is used to distribute the economic collection data to the distributed storage node with the smallest feature distance, and determine the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node;

[0060] The data processing module 202 is further configured to perform polygon retrieval on the historical economic data in the distributed storage node based on the spatiotemporal polygon retrieval granularity to obtain a spatiotemporal associated historical data set, and acquire spatiotemporal storage dimension features corresponding to each piece of historical economic data in the spatiotemporal associated historical data set;

[0061] The data storage control module 203 determines the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and uses the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval.

[0062] The above describes in detail an example of a distributed big data storage system and method provided in an embodiment of the present application. It can be understood that in order to achieve the above functions, the corresponding device includes hardware structures and / or software modules corresponding to executing each function.

[0063] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function in the application is executed in hardware or in a computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Therefore, professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0064] In addition, the present application also provides a computer terminal device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned distributed big data storage method.

[0065] In some embodiments, reference Figure 3 , which is a schematic diagram of the structure of a computer terminal device that implements a distributed big data storage method according to some embodiments of the present application. A distributed big data storage method in the above embodiment can be achieved by Figure 3 The computer terminal device 300 shown in FIG. 1 is implemented as shown in FIG. 1 , and the computer terminal device 300 includes at least one communication bus 301 , a communication interface 302 , a processor 303 and a memory 304 .

[0066] The processor 303 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more processors for controlling the execution of a distributed big data storage method in the present application.

[0067] The communication bus 301 may include a path for transmitting information between the aforementioned components.

[0068] Memory 304 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. Memory 304 may be independent and connected to processor 303 via communication bus 301. Memory 304 may also be integrated with processor 303.

[0069] Memory 304 is used to store program code for executing the solution of the present application, and is controlled by processor 303 for execution. Processor 303 is used to execute the program code stored in memory 304. The program code may include one or more software modules. In the above embodiment, the determination of polygon retrieval granularity can be implemented by processor 303 and one or more software modules in the program code in memory 304.

[0070] The communication interface 302 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0071] Optionally, the computer terminal device 300 may further include a power supply 305 for providing power to various devices or circuits in the real-time computer terminal device.

[0072] In a specific implementation, as an example, a computer terminal device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0073] The aforementioned computer terminal device can be a general-purpose computer terminal device or a dedicated computer terminal device. In a specific implementation, the computer terminal device can be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer terminal device.

[0074] In addition, other aspects of the present application further provide a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned distributed big data storage method.

[0075] In summary, in a distributed big data storage system and method disclosed in an embodiment of the present application, first, multiple distributed storage nodes are obtained, and the characteristic distances corresponding to the economic collection data and each distributed storage node are determined according to the characteristic labels corresponding to the economic collection data to be stored; the economic collection data are allocated to the distributed storage node with the smallest characteristic distance, and based on the historical economic data in the distributed storage node, the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node is determined; polygon retrieval judgment is performed on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporally associated historical data set, and the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporally associated historical data set are obtained; the storage location of the economic collection data is determined based on the spatiotemporal storage dimension features corresponding to each historical economic data, and the spatiotemporal storage dimension features of the economic collection data are used as spatiotemporal retrieval information for distributed retrieval, which can be stored and retrieved based on the spatiotemporal storage dimension features of the data to be stored, thereby improving the retrieval efficiency of spatiotemporal polygon retrieval of distributed storage nodes.

[0076] The above description is merely an embodiment of the present application. Common knowledge such as the specific technical solutions or features of the solutions is not described in detail herein. It should be noted that those skilled in the art may make various modifications and improvements without departing from the technical solution of the present application, and these modifications and improvements should also be considered within the scope of protection of the present application. These modifications and improvements will not affect the effectiveness of the implementation of the present application or the practical application of the patent.

[0077] The scope of protection claimed by this application shall be determined by the content of the claims. The specific embodiments and other descriptions in the specification may be used to interpret the content of the claims. Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of the invention. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. A distributed big data storage method, characterized in that: include: Acquire multiple distributed storage nodes, and determine the characteristic distances corresponding to the economic collection data to be stored and the corresponding distributed storage nodes according to the characteristic labels corresponding to the economic collection data to be stored; Allocating the economic collection data to a distributed storage node with the smallest characteristic distance, and determining the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node; Performing polygon search and determination on the historical economic data in the distributed storage node according to the spatiotemporal polygon search granularity to obtain a spatiotemporal associated historical data set, and obtaining spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal associated historical data set; Determine the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and use the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval; Wherein, based on the historical economic data in the distributed storage node, determining the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node specifically includes: Obtain the historical economic data in the distributed storage node and build a historical economic data space; In the spatiotemporal feature dimension space, the historical economic data space is obtained, and the difference between the maximum eigenvalue and the minimum eigenvalue corresponding to each spatiotemporal feature dimension of the historical economic data space is obtained as the data range corresponding to each spatiotemporal feature dimension, and the ratio between the data range corresponding to each spatiotemporal feature dimension and the corresponding standard data range is used as the data density feature corresponding to each spatiotemporal feature dimension; Based on the dimension weights corresponding to each spatiotemporal feature dimension, the data density features within each spatiotemporal feature dimension are weightedly fused to obtain the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node; Performing polygon search and judgment on the historical economic data in the distributed storage node according to the spatiotemporal polygon search granularity to obtain a spatiotemporally associated historical data set specifically includes: mapping according to the spatiotemporal polygon search granularity to obtain the spatiotemporal search distance corresponding to each spatiotemporal feature dimension; for any spatiotemporal feature dimension, using the farthest historical economic data within the corresponding spatiotemporal search distance from the economic collection data as polygon search vertex data; and forming a polygon search boundary based on the polygon search vertex data corresponding to each spatiotemporal feature dimension; extracting the polygon search boundary and forming all historical economic data within the boundary into a spatiotemporally associated historical data set.

2. The method according to claim 1, wherein In the process of determining the characteristic distances corresponding to the economic collection data and each distributed storage node, the characteristic values on each spatiotemporal characteristic dimension in the characteristic labels corresponding to the economic collection data to be stored are converted into distance detection vectors, and the characteristic vectors corresponding to the storage center data of each distributed storage node are obtained, and the Euclidean distance values between the distance detection vectors and each characteristic vector are used as the characteristic distances corresponding to the economic collection data and each distributed storage node.

3. The method according to claim 1, wherein In the process of obtaining the spatiotemporal storage dimension features corresponding to each historical economic data in the spatiotemporal correlation historical data set, the spatiotemporal center point distance corresponding to each historical economic data in the spatiotemporal correlation historical data set is used as the spatiotemporal storage dimension feature, and the spatiotemporal center point distance is based on the preset spatiotemporal data center point and the characteristic singular value of the collection label between the historical economic data and the spatiotemporal data center point as the spatiotemporal storage dimension feature, wherein the characteristic singular value is the weighted result of the characteristic difference value of the historical economic data and the spatiotemporal data center point in each spatiotemporal feature dimension.

4. The method according to claim 1, wherein In the process of using the spatiotemporal storage dimension characteristics of the economic collection data as the spatiotemporal retrieval information for distributed retrieval, the spatiotemporal storage dimension characteristics of the economic collection data are used as the characteristic values of the incremental retrieval dimension, and spatiotemporal polygon retrieval is performed based on multiple polygon boundary retrieval conditions and incremental retrieval dimension conditions.

5. The method according to claim 1, wherein Each historical economic data in the spatiotemporal correlation historical data set is stored sequentially according to the size of the spatiotemporal storage dimension characteristics.

6. A distributed big data storage system comprising a data storage unit, wherein the data storage unit is configured to execute the distributed big data storage method according to any one of claims 1 to 5, wherein: The data storage unit includes: A storage node selection module is used to obtain multiple distributed storage nodes and determine the characteristic distances corresponding to the economic collection data to be stored and each distributed storage node according to the characteristic labels corresponding to the economic collection data to be stored; A data processing module is used to distribute the economic collection data to the distributed storage node with the smallest feature distance, and determine the spatiotemporal polygon retrieval granularity corresponding to the distributed storage node based on the historical economic data in the distributed storage node; The data processing module is further configured to perform polygon retrieval on the historical economic data in the distributed storage node according to the spatiotemporal polygon retrieval granularity to obtain a spatiotemporal associated historical data set, and acquire spatiotemporal storage dimension features corresponding to each piece of historical economic data in the spatiotemporal associated historical data set; The data storage control module determines the storage location of the economic collection data based on the spatiotemporal storage dimension characteristics corresponding to each historical economic data, and uses the spatiotemporal storage dimension characteristics of the economic collection data as spatiotemporal retrieval information for distributed retrieval.

7. A computer terminal device, characterized in that: The computer terminal device includes a memory and a processor, the memory stores a code, and the processor is configured to obtain the code and execute the distributed big data storage method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing at least one computer program, characterized in that: The computer program is loaded and executed by a processor to implement the operations performed by the distributed big data storage method according to any one of claims 1 to 5.

Images