Distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology

Abstract

The present invention relates to the field of data processing technology, and proposes a distributed storage scheduling method for Beidou spatiotemporal big data based on container technology, comprising: deploying a distributed storage system, recording a plurality of request data in a plurality of historical data groups of each node, as well as migrated request data; determining the authenticity of the temporal dependency of each request data in each data group of each node; obtaining the strong consistency demand of each data group of the same node; obtaining the persistence of the strong consistency demand of each node; obtaining the vertical expansion CPU resource of each node; determining the attenuation of the strong consistency demand between each node; obtaining the horizontal expansion of the container instance between each node, and then obtaining the overall horizontal expansion of the container of each node; expanding each node of the distributed storage system, and receiving new Beidou spatiotemporal big data for distributed storage. The present invention aims to solve the problem of response delay caused by Beidou spatiotemporal data when node load increases sharply and node pressure is shared by expansion.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a distributed storage scheduling method for Beidou spatiotemporal big data based on container technology. Background Art

[0002] Beidou spatiotemporal big data refers to the massive amount of spatiotemporal data provided by the Beidou satellite navigation system, including satellite positioning, navigation, and timing information. As the number and accuracy of satellites increase, the speed and scale of Beidou spatiotemporal data generation continues to grow, and the volume of data is also increasing. Container technology enables efficient resource management and deployment, and supports flexible scalability and load balancing within distributed systems. Therefore, container-based technology can effectively address the storage and processing needs of Beidou spatiotemporal big data. Container orchestration tools such as Kubernetes enable automated management of data storage, task scheduling, and resource allocation, reducing manual intervention and improving operational efficiency.

[0003] When distributedly storing massive amounts of Beidou spatiotemporal big data using Kubernetes containerized service orchestration technology, Kubernetes dynamically determines whether to add more containers (horizontal scaling) or adjust the resources of existing containers (vertical scaling) based on node load. Vertical scaling, which allocates more resources to existing containers, is typically necessary when node load remains high. Horizontal scaling, which distributes the storage pressure on a single node by adding container instances to other nodes, ensures efficient distributed data storage.

[0004] In the existing technology, Beidou spatiotemporal data, which contains multiple information such as timestamps, position coordinates, speed, direction, timing errors, etc., usually does not exist in isolation. The data at the previous moment and the data at the subsequent moment are continuous and closely related. The system usually needs to perform prediction, correction and analysis based on historical data in the time series. Therefore, some Beidou spatiotemporal data have significant time-series dependence; when the load increases sharply, it is generally necessary to share the node pressure through horizontal expansion and store subsequent data in other nodes; and horizontal expansion of data with strong time-series dependence may increase the complexity of cross-node transaction processing, making it difficult to meet the response requirements in some scenarios, seriously affecting the Beidou system, especially the low-latency response requirements in military applications. Summary of the Invention

[0005] The present invention provides a distributed storage and scheduling method for Beidou spatiotemporal big data based on container technology to solve the problem that the existing Beidou spatiotemporal data will generate response delays when the node load increases sharply by expanding and sharing the node pressure. The technical solutions adopted are as follows:

[0006] The present invention proposes a distributed storage scheduling method for Beidou spatiotemporal big data based on container technology, which includes the following steps:

[0007] Deploy a distributed storage system, obtain nodes and their loads, and record several request data in several historical data groups of each node, as well as the migrated request data and node information before and after the migration; the request data includes data value, request IP address, and request time;

[0008] Based on the request time distribution of request data under the same request IP in the same data group of each node, as well as the difference in data values ​​at adjacent request times, the authenticity of the temporal dependency of each request data in each data group of each node is determined. The strong consistency demand of each data group on the same node is obtained by combining the authenticity of the temporal dependency of all request data in the same data group of the same node. The fluctuation of the strong consistency demand of each data group on the same node and the node load are further analyzed to obtain the persistence of the strong consistency demand of each node. The vertical expansion CPU resources of each node are obtained by combining the CPU resources of each node.

[0009] Based on the authenticity of the temporal dependencies of request data migrated between different nodes and its migration process, the attenuation of strongly consistent demand between nodes is determined. Combined with the persistence of strongly consistent demand and the load of each node, as well as the number of container instances, the horizontal scaling of container instances between nodes is achieved, and thus the overall horizontal scaling of containers on each node is achieved.

[0010] Based on the vertical expansion of CPU resources of each node and the horizontal expansion of container instances, each node of the distributed storage system is expanded, and new Beidou spatiotemporal big data is received for distributed storage.

[0011] Optionally, the method of determining the authenticity of the timing dependency of each request data in each data group of each node based on the request time distribution of the request data under the same request IP in the same historical data group of each node, and the difference in data values ​​at adjacent request times, includes the following specific methods:

[0012] Analyze the distribution of request times for request data from the same request IP in the same historical data group of each node to determine the temporal dependency of each request data;

[0013] Get the history of distributed storage node A The absolute value of the difference between the data values ​​of any two adjacent request data in the data group is used as the data value difference of the two request data, and the mean of the data value differences of all two adjacent request data is obtained as the first The difference in the values ​​of adjacent requests of data groups; the historical In the data group The timing of the request data depends on the authenticity The calculation method is:

[0014]

[0015] in, Indicates the history of distributed storage node A In the data group The timing dependency of request data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A The difference in the values ​​of adjacent requests for data groups, represents the absolute value function, Represents an exponential function with a natural constant as its base.

[0016] Optionally, the timing dependency of each request data may be obtained by:

[0017] Distributed storage node A history In the data group The request IP for requesting data is , get the All request data and request time under the request IP in the data group are taken as the first In the data group The request data of the same IP address and the time of the request;

[0018] Get the difference between two adjacent requests from the same IP address in all requests from the same IP address, and based on the standard deviation of all the differences and the number of requests from the same IP address, get the historical number of requests from the distributed storage node A. In the data group The timing dependency is negatively correlated with the standard deviation of all the differences and positively correlated with the number of request data from the same IP.

[0019] Optionally, the strong consistency requirement of each data group of the same node is obtained by:

[0020] Based on the history of distributed storage node A The cumulative sum of the timing dependence authenticity of all request data in the data group, combined with the The number of requested data in the data group is obtained by the distributed storage node A. The strong consistency requirement of each data group is positively correlated with the cumulative sum of the timing dependence authenticity of all the requested data and the number of the requested data.

[0021] Optionally, the specific method of obtaining the strong consistency requirement persistence of each node includes:

[0022] Based on the load of each distributed storage node and the strong consistency requirements of each historical data group, the load factor and strong consistency factor of each distributed storage node are obtained;

[0023] The inversely proportional normalized result of the standard deviation of the strong consistency requirement of all historical data groups of distributed storage node A is multiplied by its load factor and strong consistency factor as the strong consistency requirement persistence of distributed storage node A.

[0024] Optionally, the load factor and strong consistency factor of each distributed storage node are obtained, including the following specific methods:

[0025] Linearly normalize the load of all distributed storage nodes, and use the result as the load factor of each distributed storage node;

[0026] Obtain the mean of the strong consistency requirements of each historical data group of distributed storage node A as the strong consistency requirement of distributed storage node A, perform linear normalization on the strong consistency requirements of all distributed storage nodes, and use the result as the strong consistency factor of each distributed storage node.

[0027] Optionally, the obtaining of vertically expanded CPU resources of each node includes the following specific methods:

[0028]

[0029] in, Indicates the vertical expansion of CPU resources for the distributed storage node A container instance. Indicates the strong consistency requirement persistence of distributed storage node A, Indicates the existing CPU resources of the container instance of distributed storage node A.

[0030] Optionally, the strong consistency requirement attenuation between the nodes is obtained by:

[0031] Obtaining some request data for migration from distributed storage node A to distributed storage node B as migration request data from node B to node A;

[0032] Based on the authenticity of the temporal dependence of the migration request data, the migration weight of each migration request data is obtained; and the migration duration of all migration request data is weighted and summed using the migration weight, and the result is used as the strong consistency demand attenuation of the distributed storage node B for the distributed storage node A.

[0033] Optionally, the horizontal expansion of container instances between nodes may be achieved by:

[0034] Based on the load of distributed storage node B and its strong consistent demand attenuation for distributed storage node A, a horizontal expansion reference of node B for node A is obtained; the horizontal expansion reference is negatively correlated with the load and the strong consistent demand attenuation;

[0035] Normalize the weights of all nodes' horizontal scaling references to node A, and use the result as the horizontal scaling reference weight of each node to node A.

[0036] The product of the difference between 1 and the strong consistency requirement persistence of distributed storage node A and the number of container instances of distributed storage node A is used as the total number of instances required for horizontal expansion of distributed storage node A.

[0037] According to the reference weight of the horizontal expansion of distributed storage node B for node A, the total number of instances required for the horizontal expansion of distributed storage node A is weighted to obtain the horizontal expansion of container instances of distributed storage node B for node A.

[0038] Optionally, the obtaining of the overall container horizontal expansion of each node includes the following specific methods:

[0039] According to the horizontal expansion of container instances of any distributed storage node for other nodes, the overall horizontal expansion of the container of the distributed storage node is accumulated.

[0040] The beneficial effects of the present invention are as follows: the present invention analyzes the strong consistency requirement of request data in nodes in a distributed storage system, vertically expands nodes based on the persistence of strong consistency requirement, and considers the attenuation of strong consistency requirement during the migration of request data; wherein, when analyzing the request IP and request time of request data in each historical data group of a single distributed storage node, the regular distribution of request time of the same request IP and the small data value difference of adjacent request data are used to quantify the authenticity of the temporal dependence of the request data, and then determine the strong consistency requirement for the data group, reflecting the dependency of the request data and the strong requirement for the storage node, that is, the request data cannot be migrated alone to affect the temporal dependency; further analyze the fluctuation of the small strong consistency requirement under high load of all historical data groups of the distributed storage node, so as to quantify Strong consistency demand persistence, and based on this, obtain the vertical expansion CPU resources of the node to ensure the storage of requested data under high load and high demand; through the migration duration of the requested data migrated between distributed storage nodes and the authenticity of its timing dependence, analyze the impact of migration on the response of subsequent request data operations, quantify the strong consistency demand attenuation between nodes, and quantify the horizontal expansion of container instances between nodes in combination with the node load to ensure that under the premise of sufficient container instances, horizontal expansion will not too much affect the operational response of subsequent request data, and then obtain the overall container horizontal expansion of each node, and combine the vertical expansion of the corresponding node to perform subsequent node container instance expansion; to reduce the delay impact of continuous high load demand on subsequent operations, so as to perform horizontal expansion on the basis of vertical expansion, and ensure high availability, persistence and fault tolerance of data in the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A schematic flow chart of a distributed storage and scheduling method for BeiDou spatiotemporal big data based on container technology provided by one embodiment of the present invention;

[0043] Figure 2 This is an example diagram of nodes in a distributed storage system;

[0044] Figure 3 This is an example diagram of vertical expansion of nodes in a distributed storage system;

[0045] Figure 4 This is an example diagram of vertical and horizontal expansion of nodes in a distributed storage system. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] See also Figure 1 , which shows a flow chart of a distributed storage scheduling method for Beidou spatiotemporal big data based on container technology provided by an embodiment of the present invention, the method comprising the following steps:

[0048] Step S001: deploy a distributed storage system, obtain nodes and their loads, and record several request data in several historical data groups of each node, as well as migrated request data and node information before and after migration.

[0049] The purpose of this embodiment is to perform distributed storage of Beidou spatiotemporal big data. Beidou spatiotemporal data is closely related to geographical location. Distributed storage nodes can be sharded according to the geographical area of ​​the data to ensure that the storage nodes are as close as possible to the physical location of the data to reduce response delay.

[0050] Specifically, different distributed storage nodes are determined according to geographical locations, such as Figure 2 As shown; select a distributed storage system (such as Ceph), use Helm Chart to quickly deploy and manage applications in Kubernetes, and use StatefulSets to deploy applications that require persistent data, thereby completing the deployment of the distributed storage system; the deployment of the distributed storage system is achieved through existing technologies and will not be repeated in this embodiment.

[0051] Furthermore, for any distributed storage node, each historical day is regarded as a collection grouping period, and the same type of data within each day is collected as a data group consisting of request data. Then, several request data in several historical data groups of each node are recorded. The request data includes data value, request IP and request time.

[0052] Furthermore, during the historical migration process of the distributed storage system, there is requested data migration between nodes. The migration duration of the requested data migration process, that is, the time it takes for the requested data to migrate from one node to another, as well as the node information before and after the migration, that is, the node before migration and the node after migration, are recorded.

[0053] Step S002: Determine the authenticity of the timing dependency of each request data in each data group of each node based on the request time distribution of the request data under the same request IP in the same historical data group of each node, and the difference in data values ​​under adjacent request times; obtain the strong consistency demand of each data group of the same node by combining the authenticity of the timing dependency of all request data in the same data group of the same node; further analyze the fluctuation of the strong consistency demand of each data group of the same node and the node load to obtain the persistence of the strong consistency demand of each node; combine the CPU resources of each node to obtain the vertical expansion CPU resources of each node.

[0054] Preferably, in one embodiment of the present invention, the authenticity of the timing dependency of each request data in each data group of each node is determined based on the distribution of request times under the same request IP in the same historical data group of each node and the difference in data values ​​under adjacent request times. The specific method includes:

[0055] It should be noted that for each request data in each historical data group in a single distributed storage node, if the number of requests is frequent and regular under the same request IP, the request data under the current request IP depends on the status or results of the previous request data; as time goes by, historical requests will form a continuous and dependent relationship, that is, the request data is no longer independent, but will be closely related to the previous request data in time series, showing obvious time series dependence; the time series dependence of the request data can be quantified through the regular performance of the request time differences between the request data under the same request IP in the same data group.

[0056] Specifically, the history of distributed storage node A In the data group The request IP for requesting data is , get the All request data and request time under the request IP in the data group are taken as the first In the data group The same IP request data of the request data, and the same IP request time (the same IP request time includes the The difference between two adjacent requests from the same IP is obtained, and the inverse proportional normalization result of the standard deviation of all the differences is multiplied by the number of requests from the same IP as the historical number of distributed storage node A. In the data group The timing dependency of request data; it should be noted that the request time of the same IP address is distributed in time sequence, and the difference between two adjacent request times of the same IP address is the difference obtained by subtracting the time of the previous request time from the time of the latter request time of the same IP address.

[0057] It should be noted that the smaller the fluctuation of the difference between adjacent request times in the same IP request time, and the smaller the standard deviation of all differences, the more regular the request time of the same IP is. At the same time, the larger the number of request data of the same IP, the more requests the corresponding request IP has, the greater the possibility that the corresponding request data is frequently and regularly requested in time sequence, and the stronger its time sequence dependence.

[0058] It should be further explained that, based on the timing dependency, due to the security of network transmission, not all requested data are truly available; however, due to the high frequency of Beidou spatiotemporal data collection and the sudden handover, the difference in data values ​​between adjacent request times in the same data group will not be too large. Therefore, the smaller the difference between the data values ​​of adjacent request times relative to the average difference of the request data at the overall adjacent request times in the data group, the higher the authenticity of the request data, and this is used to quantify the timing dependency authenticity of the request data.

[0059] Specifically, obtain the history of distributed storage node A The absolute value of the difference between the data values ​​of any two adjacent request data in the data group is used as the data value difference of the two request data, and the mean of the data value differences of all two adjacent request data is obtained as the first The difference in the adjacent request values ​​of the data groups, where adjacent means time sequence adjacent, that is, the request data at adjacent request times; then the distributed storage node A has the In the data group The timing of the request data depends on the authenticity The calculation method is:

[0060]

[0061] in, Indicates the history of distributed storage node A In the data group The timing dependency of request data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A The difference in the values ​​of adjacent requests for data groups, represents the absolute value function, Represents an exponential function with a natural constant as its base.

[0062] What needs to be explained is that the closer the data value difference between the request data and the previous adjacent request data is to the data value difference between the adjacent requests in the data group, that is, the closer the ratio is to 1, the smaller the possibility of mutation of the request data is, and the higher its real availability is. Combined with the greater timing dependency, the timing dependency authenticity of the request data is greater.

[0063] Preferably, in one embodiment of the present invention, the timing dependency authenticity of all request data in the same data group at the same node is integrated to obtain the strong consistency requirement of each data group at the same node, including the specific method of:

[0064] It should be noted that for a historical data group of a single distributed storage node, under high load, that is, high data volume, if the temporal dependency authenticity of all data is relatively large, the corresponding data group is not only stored independently, but is interdependent. Subsequent request data operations usually require prediction, correction, and analysis based on the previous data status. The stronger the location requirement of the storage node for the data group, the stronger the consistency requirement for the data group.

[0065] Specifically, the history of distributed storage node A The cumulative sum of the timing dependence authenticity of all request data in the data group is the same as the The product of the number of requested data in the data group is used as the historical Strong consistency requirements for data sets.

[0066] It should be further explained that for all historical data groups of a single distributed storage node, under high load and the accumulation of high data volume for all historical data groups, the strong consistency demand of all data groups fluctuates less. Then, under high load, the corresponding single distributed storage node has a stronger demand for the location of each requested data stored in the storage node, thereby quantifying the persistence of the node's strong consistency demand.

[0067] Preferably, in one embodiment of the present invention, the fluctuation of the strong consistency demand of each data group at the same node and the node load are further analyzed to obtain the persistence of the strong consistency demand of each node, including the specific method of:

[0068] The load of all distributed storage nodes (i.e., the cumulative sum of the data volume of all historical data groups of a single distributed storage node) is linearly normalized, and the result is used as the load factor of each distributed storage node; the average of the strong consistency requirements of each historical data group of distributed storage node A is obtained as the strong consistency requirement of distributed storage node A, and the strong consistency requirements of all distributed storage nodes are linearly normalized, and the result is used as the strong consistency factor of each distributed storage node.

[0069] Furthermore, the inversely proportional normalized result of the standard deviation of the strong consistency requirement of all historical data groups of distributed storage node A is multiplied by its load factor and strong consistency factor as the strong consistency requirement persistence of distributed storage node A.

[0070] It should be noted that the larger the load factor, the greater the node load. At the same time, the smaller the standard deviation between the strong consistency requirements of the data group, the smaller the fluctuation. The node's historical data group continues to maintain a high strong consistency requirement, and the corresponding strong consistency requirement is more persistent.

[0071] Preferably, in one embodiment of the present invention, the vertically expanded CPU resources of each node are obtained by combining the CPU resources of each node, including the specific method of:

[0072] It should be noted that compared to horizontal scaling, which expands the system capacity by adding more container instances, vertical scaling is more flexible and has stronger scalability. Vertical scaling responds to load growth by improving the hardware configuration of a single node. However, the resources of a single node are limited. As demand grows, the expansion of a single node will encounter bottlenecks and cannot be increased indefinitely. Therefore, vertical scaling can only be performed when it is truly necessary to better alleviate the bottleneck problem.

[0073] It's also important to note that for vertical scaling, if a node experiences sustained high load, it generally means additional CPU resources are needed. However, not all high-load data is truly essential to a node; only data that meets strong consistency requirements cannot be migrated. Therefore, vertical scaling is only necessary when a single storage node experiences sustained high consistency requirements under high load.

[0074] Specifically, the calculation method for vertically scaling CPU resources of distributed storage node A is:

[0075]

[0076] in, Indicates the vertical expansion of CPU resources for the distributed storage node A container instance. Indicates the strong consistency requirement persistence of distributed storage node A, Indicates the existing CPU resources of the container instance of distributed storage node A.

[0077] Furthermore, the storage node A in the distributed storage system is vertically expanded based on the vertical expansion of CPU resources, such as Figure 3 As shown, Figure 3 After the expansion, the CPU resources of node 1 are the sum of the existing CPU resources and the vertically expanded CPU resources.

[0078] So far, by analyzing the request IP and request time of the request data in each historical data group of a single distributed storage node, the regular distribution of request time of the same request IP and the small data value difference of adjacent request data are used to quantify the authenticity of the timing dependence of the request data, and then determine the strong consistency requirement for the data group, reflecting the dependency of the request data and the strong requirement for the storage node, that is, the request data cannot be migrated alone to affect the timing dependency; further analyze the fluctuation of the smaller strong consistency requirement under high load of all historical data groups of the distributed storage node, so as to quantify the persistence of the strong consistency requirement, and based on this, obtain the vertical expansion CPU resources of the node to ensure the storage of request data under high load and high demand.

[0079] Step S003: Based on the authenticity of the timing dependence of the request data migrated between different nodes and its migration process, determine the attenuation of the strong consistency demand between each node; combine the strong consistency demand persistence and load of each node, as well as the number of container instances, to obtain the horizontal expansion of the container instances between each node, and then obtain the overall container horizontal expansion of each node.

[0080] Preferably, in one embodiment of the present invention, based on the authenticity of the temporal dependence of request data migrated between different nodes and its migration process, the strong consistency demand attenuation between nodes is determined, including the specific method of:

[0081] It should be noted that for request data that has been historically migrated between distributed storage nodes, if the request data itself has a large timing dependency on authenticity, subsequent request data will need to rely on the migrated request data during operation. If the migration time between nodes is long, subsequent operations will require a longer response time, and the corresponding strong consistency requirement will decay. The greater the decay of the strong consistency requirement between nodes, the greater the decay.

[0082] Specifically, several request data migrated from distributed storage node A to distributed storage node B are obtained as the migration request data of node B to node A; the timing dependency authenticity of all migration request data is weighted normalized, and the result obtained is used as the migration weight of each migration request data; the migration duration of all migration request data is weighted and summed by the migration weight, and the result obtained is used as the strong consistency demand attenuation of distributed storage node B for distributed storage node A.

[0083] Preferably, in one embodiment of the present invention, the strong consistency demand persistence and load of each node, as well as the number of container instances, are combined to obtain the horizontal expansion of container instances between each node, and then the overall horizontal expansion of containers of each node is obtained, including the specific method of:

[0084] It should be noted that a single distributed storage node needs to balance the load by expanding container instances through other distributed storage nodes. If the load on other nodes is light, there will be more container instances for horizontal expansion. If the load on other nodes is heavy, there will be fewer container instances available for expansion, and the distributed storage nodes will tend to be expanded vertically. At the same time, in order to meet the response requirements, horizontal expansion needs to meet the strong consistency requirements as much as possible, so the request data of a single node can be migrated to other nodes with weaker strong consistency requirement attenuation to reduce the impact of migration.

[0085] It should be further explained that for a single distributed storage node, after vertical expansion, its high load unutilization is analyzed, that is, the total number of container instances required for horizontal expansion of other nodes is obtained based on the persistence of strong consistent demand; then, the required horizontal expansion container instances are allocated based on the load size of each node and the attenuation of its strong consistent demand for the node.

[0086] Specifically, obtain the inversely proportional normalized result of the load of distributed storage node B, and multiply it by the inversely proportional normalized result of its strong consistency demand attenuation for distributed storage node A, as the reference for the horizontal expansion of node B for node A; perform weight normalization on the horizontal expansion reference of all nodes for node A, and use the result as the reference weight for the horizontal expansion of each node for node A; subtract the difference between 1 and the strong consistency demand persistence of distributed storage node A, and multiply it by the number of container instances of distributed storage node A, as the total number of instances required for the horizontal expansion of distributed storage node A.

[0087] Furthermore, the product of the reference weight of the horizontal expansion of the distributed storage node B for node A and the total number of instances required for the horizontal expansion of the distributed storage node A is rounded up and used as the horizontal expansion of the container instance of the distributed storage node B for node A.

[0088] It should be noted that the total number of instances required for horizontal expansion is obtained by quantifying the unutilized capacity through the persistence of strongly consistent demand corresponding to high load demand. Other nodes are given a larger horizontal expansion reference weight based on smaller loads and weaker strongly consistent demand decay, so as to obtain horizontal expansion of container instances on other nodes.

[0089] It should be further explained that since the nodes in the distributed storage system run in parallel, it is necessary to balance the load of each node, that is, after vertical expansion, obtain the horizontal expansion of the container instances of other nodes for the node through horizontal expansion, and then obtain the overall horizontal expansion of the container of each node, that is, the comprehensive horizontal expansion of the container instances that meets the strong consistency requirements.

[0090] Specifically, the container instance horizontal expansion of the distributed storage node B for each other node is obtained, and the cumulative sum of the horizontal expansion of all container instances is used as the overall container horizontal expansion of the distributed storage node B.

[0091] At this point, the migration duration of the request data migrated between distributed storage nodes and the authenticity of its timing dependence are analyzed to analyze the impact of migration on the response of subsequent request data operations, quantify the strong consistency demand decay between nodes, and quantify the horizontal expansion of container instances between nodes in combination with node load to ensure that under the premise of sufficient container instances, horizontal expansion will not excessively affect the operational response of subsequent request data, and then obtain the overall horizontal expansion of the container of each node, and combine the vertical expansion of the corresponding node to perform subsequent node container instance expansion.

[0092] Step S004: Based on the vertical expansion of CPU resources of each node and the horizontal expansion of container instances, each node of the distributed storage system is expanded, and new Beidou spatiotemporal big data is received for distributed storage.

[0093] Specifically, after the distributed storage node B is vertically expanded, it is horizontally expanded according to the horizontal expansion of its overall container, such as Figure 4 As shown, Figure 4 After the vertical expansion of node 1, the overall horizontal expansion of the container includes the cumulative value of the horizontal expansion of the container instances from node 2 to node 5; based on this, the distributed storage nodes are expanded with a balanced load while meeting the strong consistency requirement based on the historical request data.

[0094] Furthermore, a stream processing platform (such as Kafka) is used to receive new data streams; after the Beidou spatiotemporal data is generated in real time, Kafka caches the data stream and distributes it on demand, and the data stream will be passed to a specific container instance (Pod) in the Kubernetes cluster. The corresponding container instance is responsible for processing the data and storing it in a distributed storage system; the requested data is routed and processed according to a predetermined sharding strategy (based on geographical areas) to ensure that the data is evenly distributed to each storage node in the cluster; the sharded data will be stored in an appropriate distributed storage system (Ceph) to achieve distributed storage and scheduling of Beidou spatiotemporal big data; the data stream reception and cache distribution, as well as the subsequent routing processing and storage in the distributed storage system, are all achieved through existing technologies and will not be repeated in this embodiment.

[0095] So far, by analyzing the strong consistency requirements of requested data in nodes in the distributed storage system, vertical expansion of nodes is carried out based on the persistence of strong consistency requirements, and the decay of strong consistency requirements is considered during the migration of requested data to reduce the delay impact of continuous high load requirements on subsequent operations. In this way, horizontal expansion is carried out on the basis of vertical expansion to ensure the high availability, persistence and fault tolerance of data in the distributed storage system.

[0096] It should be noted that this embodiment adopts Model to present inverse proportional relationship and normalization processing, represents an exponential function with a natural constant as the base, As the input of the model, the implementer can set the inverse proportional function and normalization function according to the actual situation.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed storage and scheduling method for BeiDou spatiotemporal big data based on container technology, characterized in that: The method comprises the following steps: Deploy a distributed storage system, obtain nodes and their loads, and record several request data in several historical data groups of each node, as well as the migrated request data and node information before and after the migration; the request data includes data value, request IP address, and request time; Based on the request time distribution of request data under the same request IP in the same data group of each node, as well as the difference in data values ​​at adjacent request times, the authenticity of the temporal dependency of each request data in each data group of each node is determined. The strong consistency demand of each data group on the same node is obtained by combining the authenticity of the temporal dependency of all request data in the same data group of the same node. The fluctuation of the strong consistency demand of each data group on the same node and the node load are further analyzed to obtain the persistence of the strong consistency demand of each node. The vertical expansion CPU resources of each node are obtained by combining the CPU resources of each node. Based on the authenticity of the temporal dependencies of request data migrated between different nodes and its migration process, the attenuation of strongly consistent demand between nodes is determined. Combined with the persistence of strongly consistent demand and the load of each node, as well as the number of container instances, the horizontal scaling of container instances between nodes is achieved, and thus the overall horizontal scaling of containers on each node is achieved. Based on the vertical expansion of CPU resources of each node and the horizontal expansion of container instances, each node of the distributed storage system is expanded and new Beidou spatiotemporal big data is received for distributed storage; The method for obtaining the authenticity of the temporal dependency is: analyzing the distribution regularity of the request time of the request data under the same request IP in the same historical data group of each node to determine the temporal dependency of each request data; Get the history of distributed storage node A The absolute value of the difference between the data values ​​of any two adjacent request data in the data group is used as the data value difference of the two request data, and the mean of the data value differences of all two adjacent request data is obtained as the first The difference in the values ​​of adjacent requests of data groups; the historical In the data group The timing of the request data depends on the authenticity The calculation method is: ;in, Indicates the history of distributed storage node A In the data group The timing dependency of request data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A In the data group The data value of the requested data, Indicates the history of distributed storage node A The difference in the values ​​of adjacent requests for data groups, represents the absolute value function, Represents an exponential function with a natural constant as its base.

2. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The specific method for obtaining the timing dependency of each request data is as follows: Distributed storage node A history In the data group The request IP for requesting data is , get the All request data and request time under the request IP in the data group are taken as the first In the data group The request data of the same IP address and the time of the request; Get the difference between two adjacent requests from the same IP address in all requests from the same IP address, and based on the standard deviation of all the differences and the number of requests from the same IP address, get the historical number of requests from the distributed storage node A. In the data group The timing dependency of request data; The timing dependency is negatively correlated with the standard deviation of all the differences, and positively correlated with the number of the same-IP request data.

3. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The strong consistency requirement of each data group of the same node is specifically obtained by: Based on the history of distributed storage node A The cumulative sum of the timing dependence authenticity of all request data in the data group, combined with the The number of requested data in the data group is obtained by the distributed storage node A. The strong consistency requirement of each data group is positively correlated with the cumulative sum of the timing dependence authenticity of all the requested data and the number of the requested data.

4. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The specific method for obtaining the strong consistency requirement persistence of each node includes: Based on the load of each distributed storage node and the strong consistency requirements of each historical data group, the load factor and strong consistency factor of each distributed storage node are obtained; The inversely proportional normalized result of the standard deviation of the strong consistency requirement of all historical data groups of distributed storage node A is multiplied by its load factor and strong consistency factor as the strong consistency requirement persistence of distributed storage node A.

5. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 4 is characterized in that: The specific method for obtaining the load factor and strong consistency factor of each distributed storage node is as follows: Linearly normalize the load of all distributed storage nodes, and use the result as the load factor of each distributed storage node; Obtain the mean of the strong consistency requirements of each historical data group of distributed storage node A as the strong consistency requirement of distributed storage node A, perform linear normalization on the strong consistency requirements of all distributed storage nodes, and use the result as the strong consistency factor of each distributed storage node.

6. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The specific method of obtaining the vertically expanded CPU resources of each node includes: in, Indicates the vertical expansion of CPU resources for the distributed storage node A container instance. Indicates the strong consistency requirement persistence of distributed storage node A, Indicates the existing CPU resources of the container instance of distributed storage node A.

7. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The strong consistency demand attenuation between the nodes is specifically obtained as follows: Obtaining some request data for migration from distributed storage node A to distributed storage node B as migration request data from node B to node A; Based on the temporal dependency authenticity of the migration request data, the migration weight of each migration request data is obtained; The migration duration of all migration request data is weighted and summed using the migration weight, and the result is used as the strong consistency demand attenuation of distributed storage node B for distributed storage node A.

8. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The specific method for obtaining horizontal expansion of container instances between nodes is as follows: Based on the load of distributed storage node B and its strong consistent demand attenuation for distributed storage node A, the horizontal expansion reference of node B for node A is obtained; The horizontal expansion reference is negatively correlated with the load and the strong consistent demand attenuation; Normalize the weights of all nodes' horizontal scaling references to node A, and use the result as the horizontal scaling reference weight of each node to node A. The product of the difference between 1 and the strong consistency requirement persistence of distributed storage node A and the number of container instances of distributed storage node A is used as the total number of instances required for horizontal expansion of distributed storage node A. According to the reference weight of the horizontal expansion of distributed storage node B for node A, the total number of instances required for the horizontal expansion of distributed storage node A is weighted to obtain the horizontal expansion of container instances of distributed storage node B for node A.

9. The distributed storage scheduling method for BeiDou spatiotemporal big data based on container technology according to claim 1 is characterized in that: The specific method for obtaining the horizontal expansion of the overall container of each node includes: According to the horizontal expansion of container instances of any distributed storage node for other nodes, the overall horizontal expansion of the container of the distributed storage node is accumulated.

Images