Data storage method, data storage device, electronic device, and storage medium

Abstract

The present disclosure relates to a data storage method, a data storage device, an electronic device and a storage medium. The data storage method comprises: in response to obtaining to-be-stored data, determining a data stage of the to-be-stored data according to a time length of obtaining the to-be-stored data and a time threshold, the data stage comprising a first data stage and a second data stage, wherein the data in the first data stage has a frequency of access greater than or equal to a frequency threshold, and the data in the second data stage has a frequency of access less than the frequency threshold; and in response to the to-be-stored data being in the second data stage, storing the to-be-stored data in the second data stage to a second storage node, the second storage node being constructed based on a cloud storage object and a cloud computing node. Through the present disclosure, the problem of uneven resource utilization of hot data nodes and warm data nodes is solved, and the resource utilization of data storage and calculation in different data life cycle stages is improved.

Description

Technical Field

[0001] This disclosure relates to the field of distributed data storage, and more particularly to a data storage method, data storage device, electronic device, and storage medium. Background Technology

[0002] Data generated by the Elastic Search Engine (ES) can be divided into two phases based on time: the hot data phase and the warm data phase. The hot data phase is for computationally intensive scenarios, requiring high computational performance but low storage performance. The warm data phase, on the other hand, is for storage-intensive scenarios, requiring high storage performance but low computational performance.

[0003] Because Elasticsearch supports multiple data types, it cannot separate data storage based on the frequency of data access. If all data is stored on high-performance hardware disks, there will be problems with high data storage costs and / or wasted storage resources. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a data storage method, a data storage device, an electronic device, and a storage medium.

[0005] According to a first aspect of the present disclosure, a data storage method is provided, comprising: in response to acquiring data to be stored, determining the data stage of the data to be stored based on the acquisition duration and a time threshold of the data to be stored, wherein the data stage includes a first data stage and a second data stage, wherein the access frequency of data in the first data stage is greater than or equal to a frequency threshold, and the access frequency of data in the second data stage is less than the frequency threshold; in response to the data to be stored being in the second data stage, storing the data to be stored in the second data stage to a second storage node, wherein the second storage node is constructed based on a cloud storage object and a cloud computing node.

[0006] In one embodiment, determining the data stage of the data to be stored based on the acquisition duration and time threshold includes: determining that the data to be stored is in a first data stage in response to the acquisition duration being less than or equal to the time threshold; and determining that the data to be stored is in a second data stage in response to the acquisition duration being greater than the time threshold.

[0007] In one embodiment, the method further includes: in response to the data to be stored being in a first data stage, storing the data to be stored in the first data stage to a first storage node, the first storage node being constructed based on a hardware storage medium and a hardware computing module; storing the data to be stored in a second data stage to a second storage node includes: migrating the data to be stored in the second data stage from the first storage node to the second storage node.

[0008] In one embodiment, the second storage node includes multiple second sub-storage nodes, each corresponding to a different cloud computing node; the second storage node is constructed as follows: multiple second sub-storage nodes are determined, and cloud computing nodes are configured for each second sub-storage node based on preset container orchestration technology and the computing performance requirements of each second sub-storage node; the multiple second sub-storage nodes, each configured with a cloud computing node, are determined as the second storage node.

[0009] In one embodiment, the data to be stored includes different types of sub-data; storing the data to be stored in the second data stage to the second storage node includes: storing each sub-data item to the corresponding second sub-storage node according to the correspondence between data categories and second sub-storage nodes, and the data category of each sub-data item included in the data to be stored in the second data stage.

[0010] According to a second aspect of the present disclosure, a data storage apparatus is provided, comprising: a determining unit, configured to, in response to acquiring data to be stored, determine the data stage of the data to be stored based on the acquisition duration and a time threshold of the data to be stored, wherein the data stage includes a first data stage and a second data stage, wherein the access frequency of data in the first data stage is greater than or equal to a frequency threshold, and the access frequency of data in the second data stage is less than the frequency threshold; and a processing unit, configured to, in response to the data to be stored being in the second data stage, store the data to be stored in the second data stage to a second storage node, wherein the second storage node is constructed based on a cloud storage object and a cloud computing node.

[0011] In one embodiment, the determining unit determines the data stage of the data to be stored based on the acquisition duration and time threshold of the data to be stored in the following manner: in response to the acquisition duration being less than or equal to the time threshold, the data to be stored is determined to be in a first data stage; in response to the acquisition duration being greater than the time threshold, the data to be stored is determined to be in a second data stage.

[0012] In one embodiment, the processing unit is further configured to: in response to the data to be stored being in a first data stage, store the data to be stored in the first data stage to a first storage node, the first storage node being constructed based on a hardware storage medium and a hardware computing module; the processing unit stores the data to be stored in a second data stage to a second storage node in the following manner: migrate the data to be stored in the second data stage from the first storage node to the second storage node.

[0013] In one embodiment, the second storage node includes multiple second sub-storage nodes, each corresponding to a different cloud computing node; the second storage node is constructed as follows: multiple second sub-storage nodes are determined, and cloud computing nodes are configured for each second sub-storage node based on preset container orchestration technology and the computing performance requirements of each second sub-storage node; the multiple second sub-storage nodes, each configured with a cloud computing node, are determined as the second storage node.

[0014] In one embodiment, the data to be stored includes different types of sub-data; the processing unit stores the data to be stored in the second data stage to the second storage node in the following manner: according to the correspondence between data categories and second sub-storage nodes, and the data category of each sub-data item included in the data to be stored in the second data stage, each sub-data item is stored to the corresponding second sub-storage node.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the data storage method described in the first aspect or any embodiment of the first aspect.

[0016] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the data storage method described in the first aspect or any embodiment of the first aspect.

[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: After acquiring the data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and time threshold of the data to be stored. The data stage includes a first data stage and a second data stage. The access frequency of data in the first data stage is greater than or equal to a frequency threshold, while the access frequency of data in the second data stage is less than the frequency threshold. When the data to be stored is in the second data stage, the data in the second data stage is stored in a second storage node, wherein the second storage node is a storage node constructed based on cloud storage objects and cloud computing nodes. This disclosure solves the problem of uneven resource utilization between hot data nodes and warm data nodes, improving the resource utilization rate of data storage and computation at different data lifecycle stages.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a flowchart illustrating a data storage method according to an exemplary embodiment.

[0021] Figure 2 This is a flowchart illustrating a method for determining the data level of data to be stored, according to an exemplary embodiment.

[0022] Figure 3 This is a flowchart illustrating a method for storing data to be stored according to an exemplary embodiment.

[0023] Figure 4 This is a flowchart illustrating a method for constructing a second storage node according to an exemplary embodiment.

[0024] Figure 5 This is a flowchart illustrating a method for storing data to be stored in the second data stage to a second storage node according to an exemplary embodiment.

[0025] Figure 6 This is a schematic diagram illustrating the structure of a data storage system according to an exemplary embodiment of the present disclosure.

[0026] Figure 7 This is a block diagram illustrating a data storage device according to an exemplary embodiment.

[0027] Figure 8This is a block diagram illustrating an apparatus for data storage according to an exemplary embodiment.

[0028] Figure 9 This is a block diagram illustrating an apparatus for data storage according to an exemplary embodiment. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0030] The data storage method provided in this disclosure is applied to scenarios involving the storage of data generated by a data search and analysis engine.

[0031] Big data-based elastic search engines (such as Elasticsearch, ES) are generally distributed, highly scalable, and real-time search and data analysis engines. Big data-based search engines possess the ability to search, analyze, and explore user data input based on massive amounts of data. Fully leveraging the horizontal scalability of big data-based search engines can make data more valuable in production environments. The implementation principle of a big data-based search engine includes the following steps: First, the user's required data is submitted to the big data-based search engine database. The corresponding sentences are segmented into words using a word segmentation controller. The word segmentation results and their corresponding weights are stored in the database. When the user searches for data using the search engine, the word segmentation results are ranked and scored according to their weights. The returned results are then presented to the user.

[0032] In related technologies, data acquired by big data-based search engines in response to user search operations can be divided into two phases based on time sequence: the hot data phase and the warm data phase. The hot data phase targets computationally intensive scenarios. Data in this phase requires frequent access, placing high demands on the computational performance of storage nodes while placing lower demands on their storage performance. In one example, data in the hot data phase exhibits high resource utilization of computing units (such as the Central Processing Unit, CPU) (e.g., above 70%), but low utilization of storage units (such as disks) (e.g., less than 20%). The warm data phase, on the other hand, targets storage-intensive scenarios. Data in this phase involves fewer read / write operations, but due to storage economy, the amount of stored data is large, placing high demands on storage performance while placing lower demands on computational performance. In one example, data in the warm data phase exhibits low CPU utilization (e.g., less than 10%), but high disk utilization (e.g., above 70%).

[0033] In related technologies, data retrieved by big data-based search engines based on user search operations is stored using local hardware (such as CPU and hard drive) in a storage-compute architecture. However, these technologies do not consider the differences in storage and computing performance requirements between hot and warm data phases. Sharing storage and computing resources on the same server has several drawbacks: The original storage-compute architecture of big data-based search engines shares storage and computing resources on the same deployment node. Considering the differences in storage and computing performance requirements between hot and warm data phases, the aforementioned technologies suffer from uneven resource utilization. Storage-compute architectures cannot simply expand resources based on bottlenecks to alleviate the load on the bottleneck resource. However, this further exacerbates the waste of underutilized resources. Furthermore, if one resource has high utilization, other resources cannot be reduced in cost as needed, becoming a bottleneck for cost reduction. Therefore, the aforementioned existing technologies suffer from high operational costs. In related technologies, cluster scaling involves the application and return of host resources. This process is lengthy and requires advance inspections based on thresholds to identify host resource bottlenecks, generate alerts, and then manually assess and prepare and deploy resources. The entire process involves many steps and places high demands on operations and maintenance personnel. Therefore, these technologies suffer from a lengthy host management process. Furthermore, after scaling up or down the storage cluster used for data storage, data rebalancing is required, involving data migration between different nodes. This data balancing process may cause increased load on some newly added nodes, temporarily affecting cluster stability and requiring manual intervention in severe cases. Therefore, these technologies suffer from high data migration costs. Finally, the scaling of storage clusters in these technologies relies on physical machine hardware, lacking elastic operation capabilities and exhibiting poor operational and deployment flexibility. The related technologies employ a multi-level storage architecture, which gradually reduces the data storage medium as the storage period of time-series data increases. For example, hot data is stored on solid-state drives (SSDs), warm and cold data is stored on hard disk drives (HDDs), and frozen data is stored on distributed file systems (HDFS, Hadoop Destributed File System). As the media migrates, the performance of data retrieval will gradually decrease. For example, the retrieval performance of HDDs is half that of SSDs, while the data retrieval performance of HDFS can be 20 times worse than that of SSDs in extreme cases. Therefore, the related technologies suffer from the problem of high performance loss when storing cold backup data.

[0034] In view of this, this disclosure proposes a data storage method. After acquiring the data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and a time threshold. The data stage includes a first data stage and a second data stage. Data in the first data stage has an access frequency greater than or equal to a frequency threshold, while data in the second data stage has an access frequency less than the frequency threshold. When the data to be stored is in the second data stage, it is stored in a second storage node, which is a storage node constructed based on cloud storage objects and cloud computing nodes. This disclosure solves the problem of uneven resource utilization between hot data nodes and warm data nodes, improves the resource utilization of data storage and computation at different data lifecycle stages, and ensures data import and data access performance. It also improves data access performance in the warm data stage. By constructing a storage-compute separation architecture and introducing object storage, the unit price of warm data storage is greatly reduced, thereby reducing storage costs.

[0035] The data storage method proposed in this disclosure is mainly applied to the construction and use of search engine log scenarios based on big data. The data has relatively clear time sequence and hot / cold characteristics, and provides different media storage solutions for data at different stages.

[0036] Figure 1 This is a flowchart illustrating a data storage method according to an exemplary embodiment. For example... Figure 1 As shown, the method includes the following steps.

[0037] In step S101, in response to acquiring the data to be stored, the data stage of the data to be stored is determined according to the acquisition duration and time threshold of the data to be stored. The data stage includes a first data stage and a second data stage, wherein the access frequency of the data in the first data stage is greater than or equal to the frequency threshold, and the access frequency of the data in the second data stage is less than the frequency threshold.

[0038] In step S102, in response to the data to be stored being in the second data stage, the data to be stored in the second data stage is stored in the second storage node, which is constructed based on cloud storage objects and cloud computing nodes.

[0039] In this embodiment of the disclosure, after acquiring the data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and a time threshold. The data stages include a first data stage and a second data stage. Data in the first data stage has an access frequency greater than or equal to a frequency threshold, while data in the second data stage has an access frequency less than the frequency threshold. When the data to be stored is in the second data stage, the data in the second data stage is stored in a second storage node, wherein the second storage node is a storage node constructed based on cloud storage objects and cloud computing nodes.

[0040] In this embodiment, the data storage method is applied in a big data-based elastic search engine. The data to be stored is the data obtained by the elastic search engine in response to user queries and access behaviors. After the data to be stored is obtained, its access frequency gradually decreases as the acquisition time increases. It is understood that for frequently accessed data, the elastic search engine needs to frequently respond with the corresponding data based on received access requests. Therefore, frequently accessed data requires higher computing performance and is generally stored in the hardware storage area of ​​the device to quickly respond to received access requests. Correspondingly, for infrequently accessed data (i.e., data in the second data stage), it is generally relatively inactive but still requires periodic access. This type of data is usually not frequently changed but still needs to be queried, and it tends to accumulate over time, requiring a large amount of storage space. This disclosure sets up corresponding second storage nodes for infrequently accessed data. This disclosure uses container orchestration methods to allocate different cloud computing nodes to different cloud storage objects, thus constructing the first storage node. It is understandable that the first storage node, built based on cloud storage objects and cloud computing nodes, has lower computing performance, i.e. weaker data read and write capabilities, and larger storage space compared to local hardware storage nodes. Therefore, the second storage node in this disclosure is mainly suitable for data to be stored that is accessed less frequently. The frequency threshold in this disclosure is the frequency value corresponding to the read and write capabilities of the second storage node.

[0041] In this embodiment of the disclosure, a second storage node is constructed based on a storage-compute separation architecture, which can be regarded as a storage-compute separation node. Storage-compute separation is an architectural design concept that separates data storage and data computation resources, enabling the second storage node to be independently expanded and optimized, providing strong support for building scalable, flexible, and efficient data storage systems.

[0042] This disclosure addresses the issue of uneven resource utilization between hot and warm data nodes, improving resource utilization for data storage and computation at different stages of the data lifecycle, and ensuring data import and access performance. It also enhances data access performance during the warm data stage. By constructing a storage-compute separation architecture and introducing low-cost cloud object storage, the storage cost of warm data is reduced, thereby lowering the overall storage cost.

[0043] The present disclosure describes a method for determining the data level of data to be stored.

[0044] Figure 2 This is a flowchart illustrating a method for determining the data level of data to be stored, according to an exemplary embodiment. Figure 2 As shown, the method includes the following steps.

[0045] In step S201, in response to obtaining the data to be stored, the acquisition time of the data to be stored is determined.

[0046] In step S202A, in response to the acquisition duration being less than or equal to a time threshold, it is determined that the data to be stored is in the first data stage.

[0047] In step S202B, in response to the acquisition duration being greater than the time threshold, it is determined that the data to be stored is in the second data stage.

[0048] In this embodiment of the disclosure, after acquiring the data to be stored, the acquisition duration of the data to be stored is determined. If the acquisition duration is less than a time threshold, the corresponding data is considered to be frequently accessed and is determined to be in the hot data stage, i.e., in the first data stage. If the acquisition duration is less than or equal to the time threshold, the access frequency of the corresponding data is considered to be decreasing and is determined to be in the warm data stage, i.e., in the second data stage.

[0049] In this embodiment, the data storage method is applied in a big data-based elastic search engine. The data to be stored is the data obtained by the elastic search engine in response to user queries and access behaviors. After the data to be stored is obtained, the frequency of access to the data to be stored gradually decreases as the acquisition time increases. As mentioned above, data to be stored with different access frequencies have different impacts on the computing and storage performance of the storage nodes. This disclosure divides the data stages corresponding to the data to be stored into a first data stage and a second data stage based on a time threshold and the acquisition duration of the data to be stored. The time threshold, as a critical parameter for dividing the data stages, can be set according to the user's habits of accessing the data to be stored using the elastic search engine. By continuously monitoring the user's data access behavior, the time period during which the user frequently accesses the same data is determined, and this time period is defined as the time threshold. For example, the elastic search engine can determine the time threshold at regular intervals based on the user's usage habits, thereby dividing the data stages of the data to be stored based on the latest acquired time threshold. Users can also set the aforementioned time thresholds themselves in the settings interface of the corresponding elastic search engine based on their personal usage habits. The effects of these time thresholds can be explained: data with a retrieval duration exceeding the time threshold will be stored in the cloud, not occupying local memory, but with a longer response time; data with a retrieval duration less than or equal to the time threshold will be stored locally, occupying local memory, but can be accessed quickly. If the user does not define a custom time threshold, and the corresponding elastic search engine cannot determine the time threshold based on the user's usage habits, the time thresholds in this disclosure can be default time thresholds, such as, but not limited to, 3 days, 7 days, and 14 days.

[0050] In this embodiment of the disclosure, considering that the data to be stored has relatively clear temporal and hot / cold characteristics, where the hot / cold characteristics correspond to the different data stages mentioned above, including a first data stage and a second data stage, and that the storage and computing performance requirements of the storage nodes are inconsistent for data in different stages, this disclosure provides storage schemes for different media for data in different stages. The following embodiments of this disclosure describe the method for storing the data to be stored.

[0051] Figure 3 This is a flowchart illustrating a method for storing data to be stored according to an exemplary embodiment. Figure 3 As shown, the method includes the following steps.

[0052] In step S301, in response to acquiring the data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and time threshold of the data to be stored.

[0053] In step S302, in response to the data to be stored being in the first data stage, the data to be stored in the first data stage is stored in the first storage node, which is constructed based on hardware storage media and hardware computing units.

[0054] In step S303, in response to the data to be stored being in the second data stage, the data to be stored being in the second data stage is migrated from the first storage node to the second storage node.

[0055] In this embodiment, a first storage node based on a storage-computing architecture is provided for the data to be stored in the first data stage. Considering the characteristics of the data to be stored in the first stage (i.e., hot data), namely high read / write throughput but a small amount of data to be stored, this disclosure uses a machine model with a multi-core storage module plus a computing module (such as CPU + SSD) to deploy the hot data node. This ensures both performance and stability while also guaranteeing good resource utilization. The first storage node in this disclosure can also be called a hot data node. Hot data nodes have high utilization of the computing module and high requirements for the data output capability of the storage module. This disclosure stores hot data through a first storage node constructed based on hardware storage media and hardware computing units, ensuring the performance of accessing hot data.

[0056] In this embodiment, for data in the second data stage (warm data stage), it is assumed that the requirements for read / write performance decrease, while the requirements for storage capacity increase. Therefore, after the data to be stored transitions from the first data stage to the second data stage, this disclosure transfers the data to be stored in the second data stage from the hardware storage node to the cloud storage node, thereby reducing the storage cost for warm data while ensuring the storage performance of the local hardware storage node.

[0057] In this embodiment of the disclosure, the second storage node includes multiple second sub-storage nodes, each corresponding to a different cloud computing node. The following describes the construction of the second storage node.

[0058] Figure 4 This is a flowchart illustrating a method for constructing a second storage node according to an exemplary embodiment. Figure 4 As shown, the method includes the following steps.

[0059] In step S401, multiple second sub-storage nodes are determined, and cloud computing nodes are configured for each second sub-storage node based on preset container orchestration technology and the computing performance requirements of each second sub-storage node.

[0060] In step S402, multiple second sub-storage nodes, each configured with a cloud computing node, are identified as second storage nodes.

[0061] In this embodiment, a storage-compute separation architecture is used to construct the second storage node. This architecture isolates storage resources from computing resources. The disclosure provides computing resources through a cloud computing node cluster corresponding to a pre-defined container orchestration technology (such as Kubernetes, MiKS), and uses a high-performance distributed file system (such as JuiceFS) designed for cloud-native environments for file storage. The distributed file system stores the data to be stored in the second data stage on cloud storage objects (such as Object Storage, OSS). This disclosure uses a Container Storage Interface Driver (CSIDriver) mode to mount different cloud computing nodes (PODs) in the container orchestration cluster. Each cloud computing node requests an independent volume to store node index data. A StatefulSet controller mode is used to allocate cloud computing nodes to different second sub-storage nodes, enabling recovery based on state after node failure. It retains user requests for storage resources (PersistentVolumeClaim, PVC) and Domain Name System (DNS) information, correctly locating the corresponding storage volume and network address.

[0062] In this embodiment, computing resources are allocated based on container orchestration clusters, using storage modules and memory resources as templates. In scenarios where warm data nodes consume relatively few computing resources, a single server can deploy 10 computing PODs, achieving virtualization of computing resources and eliminating the waste of computing resources caused by limitations on storage resources. At the storage level, this disclosure combines a high-performance distributed file system designed for cloud-native environments with the introduction of cloud storage objects, significantly reducing unit storage costs while ensuring performance, achieving further cost reduction in storage. Simultaneously, while adopting cheaper storage, it abandons the HDFS snapshot archiving mechanism in related technologies, which greatly affects the access performance of frozen data. Switching to cloud storage objects results in lower costs, reducing storage costs while ensuring access performance. This disclosure uses cloud storage, which can reduce the number of replicas of components in big data-based search engines (such as ES components). The multi-replica mechanism provided by distributed storage reduces the actual storage quantity, achieving cost optimization. After implementing the storage-compute separation architecture, the expansion of warm data nodes in the search engine service no longer depends on the utilization rate of physical server storage resources. Computing resources can be scaled up or down in minutes, provided that the cloud computing nodes of the container orchestration cluster have sufficient resources. This improves cluster operation and maintenance efficiency and reduces the impact of physical machine hardware failures on service availability.

[0063] In this embodiment of the disclosure, the data to be stored includes different types of sub-data. The following describes a method for storing data in the second data stage to a second storage node.

[0064] Figure 5 This is a flowchart illustrating a method for storing data to be stored in the second data stage to a second storage node, according to an exemplary embodiment. Figure 5 As shown, the method includes the following steps.

[0065] In step S501, in response to the acquisition of data to be stored, and the data to be stored is in the second data stage.

[0066] In step S502, each sub-data item is stored in the corresponding second sub-storage node according to the correspondence between the data category and the second sub-storage node, and the data category of each sub-data item included in the data to be stored in the second data stage.

[0067] In this embodiment, the data to be stored includes different types of sub-data, and the data to be stored in the second data stage can be further subdivided into different types. When storing the data to be stored in the second storage node, according to the correspondence between data categories and second sub-storage nodes, different types of data to be stored in the second data stage are stored in the corresponding cloud sub-storage nodes. This ensures that different types of data to be stored can be stored in the optimal second sub-storage node.

[0068] In this embodiment, the data to be stored in the second data stage can be further divided into different data categories. These data categories can correspond to the inherent data types of the data to be stored. For example, the data to be stored in the second data stage can be further divided into log data from an elastic search engine, hot data from historical searches, etc. It is understood that different types of data to be stored accumulate at different rates, resulting in different amounts of data. That is, different data types require different storage capacity from their respective storage nodes. Since different second sub-storage nodes in this disclosure have different storage capacities, the correspondence between data categories and second sub-storage nodes can be seen as a correspondence between data categories and the storage capacity of second sub-storage nodes. Second sub-storage nodes with larger storage capacities are set for data to be stored with high storage capacity requirements, while second sub-storage nodes with smaller storage capacities are set for data to be stored with low storage capacity requirements. The data categories in this disclosure can also correspond to user preferences. For example, the data to be stored in the second data stage can be further divided into data of interest and other data. Users will access data of interest more frequently than other data. That is, data of interest has higher requirements for the read and write capabilities of storage nodes. Different second sub-storage nodes in this disclosure will have different numbers of cloud computing nodes. Therefore, the correspondence between data categories and second sub-storage nodes in this disclosure can be the correspondence between data categories and the number of cloud computing nodes in the second sub-storage nodes. Second sub-storage nodes with more cloud computing nodes are set for data of interest, and second sub-storage nodes with fewer cloud computing nodes are set for other data.

[0069] In this embodiment, the data storage method is applied to a big data-based search engine (such as Elasticsearch, ES). The big data-based search engine includes a master node and client nodes. For the master node, a hybrid deployment approach is adopted: two are deployed on physical machine nodes, and one on a container node, ensuring the availability (HA) of the master node and improving its stability. For client nodes, all are deployed on container nodes, reducing usage costs. This also enhances the elastic scaling capability of the client as a load balancer. This disclosure introduces an elastic deployment solution based on container orchestration technology (Kubernetes) to improve operational efficiency. Based on the method of encapsulating, deploying, and managing complex applications (Kubernetes Operator pattern), this disclosure provides a deployment orchestration tool for products used for search, logging, security, observability, and data analysis (Elastic products). It extends the orchestration capabilities of container orchestration technology, supporting the setting and management of corresponding components (such as Elasticsearch and Kibana) on container orchestration technology. Using this tool in conjunction with the orchestration capabilities of Kubernetes, cluster rolling upgrades, failure detection and recovery, and configuration update upgrades can be easily achieved. It reduces the reliance on manual labor in operation and maintenance, resulting in a significant improvement in efficiency.

[0070] In an exemplary embodiment of this disclosure, Figure 6 This is a schematic diagram illustrating the structure of a data storage system according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the proxy server structure of the data storage system includes a first storage node built on physical servers and a second storage node built on container orchestration technology, a cloud-based distributed file system, and public cloud storage objects. Container orchestration technology is used to allocate compute nodes to different second-level sub-storage nodes. Container orchestration technology includes a master node, warm data nodes, and client nodes. The warm data nodes in container orchestration technology establish a connection with the cloud-based distributed file system through a container storage interface built by a container storage interface driver. The cloud-based distributed file system includes different volumes (e.g., volumes 1, 2, and 3) for storing warm data indexes. The cloud-based distributed file system stores the acquired warm data into public cloud storage objects, which may include different cloud storage objects (e.g., cloud storage A, B, C, and D). The physical servers include different control groups, each corresponding to different nodes for storing hot data. Different nodes correspond to different hard drives (e.g., hard drives 1, 2, and 3).

[0071] In this embodiment, after acquiring the data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and time threshold. The data stage includes a first data stage and a second data stage. Data in the first data stage has an access frequency greater than or equal to a frequency threshold, while data in the second data stage has an access frequency less than the frequency threshold. When the data to be stored is in the second data stage, it is stored in a second storage node, which is a storage node constructed based on cloud storage objects and cloud computing nodes. This disclosure solves the problem of uneven resource utilization between hot data nodes and warm data nodes, improves the resource utilization of data storage and computation at different data lifecycle stages, and ensures data import and data access performance. It also improves data access performance in the warm data stage. By constructing a storage-compute separation architecture and introducing object storage, the unit price of warm data storage is greatly reduced, thereby reducing storage costs. While ensuring sufficient resources for hot data nodes, and fully utilizing physical hardware to guarantee read and write performance, the search performance of the search engine is greatly improved.

[0072] Based on the same concept, this disclosure also provides a data storage device 100.

[0073] It is understood that the data storage device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0074] Figure 7 This is a block diagram illustrating a data storage device 100 according to an exemplary embodiment. (Refer to...) Figure 7 The device includes a determining unit 101 and a processing unit 102.

[0075] The determining unit 101 is used to determine the data stage of the data to be stored based on the acquisition duration and time threshold of the data to be stored in response to the acquisition of the data to be stored. The data stage includes a first data stage and a second data stage, wherein the access frequency of the data in the first data stage is greater than or equal to the frequency threshold, and the access frequency of the data in the second data stage is less than the frequency threshold.

[0076] The processing unit 102 is used to, in response to the data to be stored being in the second data stage, store the data in the second data stage to a second storage node, the second storage node being constructed based on cloud storage objects and cloud computing nodes.

[0077] In one embodiment, the determining unit 101 determines the data stage of the data to be stored based on the acquisition duration and a time threshold as follows: In response to an acquisition duration less than or equal to the time threshold, the data to be stored is determined to be in a first data stage. In response to an acquisition duration greater than the time threshold, the data to be stored is determined to be in a second data stage.

[0078] In one embodiment, the processing unit 102 is further configured to: in response to the data to be stored being in a first data stage, store the data in the first data stage to a first storage node, the first storage node being constructed based on a hardware storage medium and a hardware computing module. The processing unit 102 stores the data to be stored being in a second data stage to a second storage node by migrating the data in the second data stage from the first storage node to the second storage node.

[0079] In one embodiment, the second storage node includes multiple second sub-storage nodes, each corresponding to a different cloud computing node. The second storage node is constructed as follows: multiple second sub-storage nodes are determined, and based on preset container orchestration technology and the computing performance requirements of each second sub-storage node, a cloud computing node is configured for each second sub-storage node. These multiple second sub-storage nodes, each configured with a cloud computing node, are then identified as the second storage node.

[0080] In one embodiment, the data to be stored includes different types of sub-data. The processing unit 102 stores the data to be stored in the second data stage to the second storage node in the following manner: according to the correspondence between data categories and second sub-storage nodes, and the data category of each sub-data item included in the data to be stored in the second data stage, each sub-data item is stored to the corresponding second sub-storage node.

[0081] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0082] Figure 8 This is a block diagram illustrating a data storage device 200 according to an exemplary embodiment. The device 200 can be provided as an electronic device. For example, the device 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0083] Reference Figure 8 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0084] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0085] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0086] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0087] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0088] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0089] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0090] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0091] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0092] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0093] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0094] Figure 9 This is a block diagram illustrating an apparatus 300 for data storage according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 9 The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned data storage method.

[0095] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0096] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0097] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0098] It is further understood that the terms “center,” “longitudinal,” “lateral,” “front,” “rear,” “up,” “down,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer,” etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation.

[0099] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0100] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0101] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0102] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data storage method, characterized in that, include: In response to acquiring data to be stored, the data stage of the data to be stored is determined based on the acquisition duration and time threshold of the data to be stored. The data stage includes a first data stage and a second data stage, wherein the access frequency of data in the first data stage is greater than or equal to a frequency threshold, and the access frequency of data in the second data stage is less than a frequency threshold. In response to the data to be stored being in the second data stage, the data in the second data stage is stored in the second storage node, which is constructed based on cloud storage objects and cloud computing nodes.

2. The method according to claim 1, characterized in that, The step of determining the data stage of the data to be stored based on the acquisition duration and time threshold of the data to be stored includes: In response to the acquisition duration being less than or equal to the time threshold, it is determined that the data to be stored is in the first data stage; In response to the acquisition duration being greater than the time threshold, it is determined that the data to be stored is in the second data stage.

3. The method according to claim 1, characterized in that, The method further includes: In response to the data to be stored being in the first data stage, the data to be stored in the first data stage is stored in the first storage node, which is constructed based on hardware storage media and hardware computing modules; The step of storing the data to be stored in the second data stage to the second storage node includes: The data to be stored in the second data stage is migrated from the first storage node to the second storage node.

4. The method according to any one of claims 1-3, characterized in that, The second storage node includes multiple second sub-storage nodes, each of which corresponds to a different cloud computing node; The second storage node is constructed in the following manner: Multiple second sub-storage nodes are identified, and cloud computing nodes are configured for each second sub-storage node based on preset container orchestration technology and the computing performance requirements of each second sub-storage node. Multiple second sub-storage nodes, each configured with a cloud computing node, are identified as the second storage node.

5. The method according to claim 4, characterized in that, The data to be stored includes different types of sub-data; The step of storing the data to be stored in the second data stage to the second storage node includes: Based on the correspondence between data categories and second sub-storage nodes, and the data category of each sub-data item included in the data to be stored in the second data stage, each sub-data item is stored in the corresponding second sub-storage node.

6. A data storage device, characterized in that, include: A determining unit is configured to, in response to acquiring data to be stored, determine the data stage of the data to be stored based on the acquisition duration and time threshold of the data to be stored. The data stage includes a first data stage and a second data stage, wherein the access frequency of data in the first data stage is greater than or equal to a frequency threshold, and the access frequency of data in the second data stage is less than a frequency threshold. The processing unit is configured to, in response to the data to be stored being in the second data stage, store the data in the second data stage to a second storage node, wherein the second storage node is constructed based on cloud storage objects and cloud computing nodes.

7. The apparatus according to claim 6, characterized in that, The determining unit determines the data stage of the data to be stored based on the acquisition duration and time threshold of the data to be stored in the following manner: In response to the acquisition duration being less than or equal to the time threshold, it is determined that the data to be stored is in the first data stage; In response to the acquisition duration being greater than the time threshold, it is determined that the data to be stored is in the second data stage.

8. The apparatus according to claim 6, characterized in that, The processing unit is also used for: In response to the data to be stored being in the first data stage, the data to be stored in the first data stage is stored in the first storage node, which is constructed based on hardware storage media and hardware computing modules; The processing unit stores the data to be stored in the second data stage to the second storage node in the following manner: The data to be stored in the second data stage is migrated from the first storage node to the second storage node.

9. The apparatus according to any one of claims 6-8, characterized in that, The second storage node includes multiple second sub-storage nodes, each of which corresponds to a different cloud computing node; The second storage node is constructed in the following manner: Multiple second sub-storage nodes are identified, and cloud computing nodes are configured for each second sub-storage node based on preset container orchestration technology and the computing performance requirements of each second sub-storage node. Multiple second sub-storage nodes, each configured with a cloud computing node, are identified as the second storage node.

10. The apparatus according to claim 9, characterized in that, The data to be stored includes different types of sub-data; The processing unit stores the data to be stored in the second data stage to the second storage node in the following manner: Based on the correspondence between data categories and second sub-storage nodes, and the data category of each sub-data item included in the data to be stored in the second data stage, each sub-data item is stored in the corresponding second sub-storage node.

11. An electronic device, characterized in that, include: processor: Memory used to store processor-executable instructions; The processor is configured to execute the data storage method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by a processor, enable the processor to perform the data storage method according to any one of claims 1 to 5.

Images