A data storage method and device, and a storage medium

Abstract

A data storage method and device, and a storage medium, comprising: when creating an index of to-be-processed data, obtaining cluster parameter information of a storage to-be-processed data, a historical number of documents written into the index, and a historical size of documents written into the index (S101); and according to the cluster parameter information, the number of documents, and the size of documents, allocating a plurality of index shards for the to-be-processed data, and writing the to-be-processed data into the plurality of index shards, the plurality of index shards being on at least one node of a cluster (S102).

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a data storage method, apparatus, and storage medium. Background Technology

[0002] The open-source distributed search engine (Elasticsearch, ES) can both store and manage large-scale unstructured data, and also enable full-text retrieval of the stored data. This has led to the widespread adoption of ES in search engine development in recent years.

[0003] In existing technologies, when using Elasticsearch to shard an index, it is necessary to manually obtain the cluster configuration parameters, determine the number of index shards based on the configuration parameters, and then shard the created index according to the determined number of index shards. When there is a lot of index data and multiple indexes need to be created, it is necessary to manually obtain the cluster configuration parameters repeatedly to determine the number of each index shard, thereby creating multiple index shards. This reduces the intelligence of creating index shards. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of the present invention aim to provide a data storage method, apparatus, and storage medium that can improve the intelligence of the data storage device when creating index shards.

[0005] The technical solution of this invention is implemented as follows:

[0006] This application provides a data storage method, the method comprising:

[0007] When creating an index for the data to be processed, obtain the cluster parameter information for storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index;

[0008] Based on the cluster parameter information, the number of documents, and the document size, multiple index shards are allocated to the data to be processed, and the data to be processed is written into the multiple index shards, wherein the multiple index shards are located on at least one node of the cluster.

[0009] In the above scheme, the cluster parameter information includes at least one of the following:

[0010] The cluster's total number of processor cores, processor utilization, cluster memory information, cluster memory utilization, cluster network throughput information, network interface card data information in the cluster, total number of disk read / write operations in the cluster, and average number of disk read / write operations in the cluster.

[0011] In the above scheme, allocating multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size includes:

[0012] Based on the cluster parameter information, the number of documents, and the document size, the number of index shards corresponding to the data to be processed is determined;

[0013] The multiple index shards are allocated to the data to be processed based on the number of index shards.

[0014] In the above scheme, determining the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size includes:

[0015] Based on the cluster parameter information, determine the first number of index shards;

[0016] Based on the number of documents, determine the second number of index shards;

[0017] Based on the document size, determine the third number of index shards;

[0018] The number of index shards is determined based on the first quantity, the second quantity, and the third quantity.

[0019] In the above scheme, the step of allocating multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size further includes:

[0020] Create index aliases corresponding to the multiple index shards;

[0021] The data to be processed is indexed in the cluster using the index alias.

[0022] In the above scheme, the method further includes:

[0023] The number of nodes in the cluster is adjusted based on the cluster's processor parameters, memory information, and total number of disk read / write operations.

[0024] Update the cluster parameter information based on the cluster after adjusting the number of nodes.

[0025] This application provides a data storage device, the device comprising:

[0026] The acquisition unit is used to acquire cluster parameter information storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index when creating an index for the data to be processed.

[0027] An allocation unit is configured to allocate multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size, and to write the data to be processed into the multiple index shards, wherein the multiple index shards are located on at least one node of the cluster.

[0028] In the above scheme, the cluster parameter information includes at least one of the following:

[0029] The total number of processor cores storing the data to be processed, the utilization rate of the processors, the cluster's memory information, the cluster's memory utilization rate, the cluster's network throughput information, the network interface card data information in the cluster, the total number of disk read / write operations in the cluster, and the average number of disk read / write operations in the cluster.

[0030] In the above scheme, the device further includes a determining unit;

[0031] The determining unit is configured to determine the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size; and to allocate the multiple index shards to the data to be processed based on the number of index shards.

[0032] In the above scheme, the determining unit is used to determine a first number of index shards based on the cluster parameter information; determine a second number of index shards based on the number of documents; determine a third number of index shards based on the document size; and determine the total number of index shards based on the first number, the second number, and the third number.

[0033] In the above scheme, the device further includes a creation unit;

[0034] The creation unit is used to create index aliases corresponding to the multiple index shards; and to index the data to be processed in the cluster using the index aliases.

[0035] In the above scheme, the device further includes an adjustment unit;

[0036] The adjustment unit is used to adjust the number of nodes in the cluster based on the cluster's processor parameter information, cluster memory information, and the total number of disk read / write operations in the cluster; and to update the cluster parameter information based on the cluster after adjusting the number of nodes.

[0037] This application provides a data storage device, the device comprising:

[0038] The system includes a memory, a processor, and a communication bus. The memory communicates with the processor via the communication bus. The memory stores a data storage program that can be executed by the processor. When the data storage program is executed, the processor performs the data storage creation method described above.

[0039] This application provides a storage medium storing a computer program applied to a data storage device. When the computer program is executed by a processor, it implements the data storage method described above.

[0040] This invention provides a data storage method, apparatus, and storage medium, comprising: when creating an index for data to be processed, obtaining cluster parameter information for storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index; allocating multiple index shards for the data to be processed based on the cluster parameter information, the number of documents, and the document size, and writing the data to be processed into the multiple index shards, wherein the multiple index shards reside on at least one node of the cluster. Using the above method, when creating an index for the data to be processed, the data storage device directly obtains the cluster parameter information, the number of documents historically written to the index, and the size of the documents historically written to the index, and determines the number of index shards based on these parameters, thereby allocating multiple index shards for the data to be processed. This eliminates the need for manual acquisition of cluster configuration parameters and manual determination of the number of index shards based on the cluster configuration parameters, improving the intelligence of the data storage device when creating index shards. Attached Figure Description

[0041] Figure 1 A flowchart illustrating a data storage method provided in this application embodiment;

[0042] Figure 2 A schematic diagram of the composition structure of a data storage device provided in this application embodiment. Figure 1 ;

[0043] Figure 3 A schematic diagram of the composition structure of a data storage device provided in this application embodiment. Figure 2 . Detailed Implementation

[0044] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0045] Example 1

[0046] This application provides a data storage method. Figure 1 A flowchart of a data storage method provided in an embodiment of this application is shown below. Figure 1 As shown, data storage methods may include:

[0047] S101. When creating an index for the data to be processed, obtain the cluster parameter information for storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index.

[0048] The data storage method provided in this application embodiment is applicable to scenarios where a data storage device stores data to be processed.

[0049] In this embodiment, the data to be processed can be data obtained by the data storage device and stored in ES. The data to be processed can be data in binary form, data in hexadecimal form, or other forms of data. The specific form can be determined according to the actual situation, and this embodiment does not limit it.

[0050] In this embodiment of the application, the cluster parameter information may include the total number of processor cores in the cluster, the processor utilization rate, the cluster memory information, the cluster memory utilization rate, the cluster network throughput information, the network card data information in the cluster, the total number of disk read / write operations in the cluster, and the average number of disk read / write operations in the cluster, etc.

[0051] In this embodiment of the application, the number of documents historically written to the index can be the number of documents in which historical data to be processed is written to the index in the form of documents before the data to be processed is created.

[0052] It should be noted that the number of documents can be one or multiple, and the specific number can be determined according to the actual situation. This application embodiment does not limit this.

[0053] In this embodiment of the application, the size of the historical documents written to the index can be the size of the documents when the historical data to be processed is written to the index in the form of documents before the data to be processed is created.

[0054] It should be noted that the size of a document can be the sum of the sizes of all documents when writing to the index, or it can be the size of each document when writing to the index. The specific size can be determined according to the actual situation, and this application embodiment does not limit it.

[0055] In this embodiment, the data storage device can continuously write data to the index. When the amount of data in the index reaches a preset data volume threshold, the data processing device will recreate the index. Alternatively, the data storage device can use a thread to detect the amount of data in the index within a first preset time period. When the data storage device determines that the amount of data in the index has reached the preset data volume threshold, the data storage device will create an index for the data to be processed. The specific method can be determined according to the actual situation, and this embodiment does not limit it.

[0056] In this application embodiment, the first preset time period can be a time period configured for the data storage device, or it can be a time period information obtained by the data storage device from the received instruction information. The specific time period can be determined according to the actual situation, and this application embodiment does not limit it.

[0057] In this embodiment of the application, the first preset time period can be within 5ns, 1ms, or 2s, and the specific time period can be determined according to the actual situation. This embodiment of the application does not limit this.

[0058] In this embodiment of the application, the cluster parameter information includes at least one of the following: the total number of processor cores in the cluster, the processor utilization rate, the cluster memory information, the cluster memory utilization rate, the cluster network throughput information, the network card data information in the cluster, the total number of disk read / write operations in the cluster, and the average number of disk read / write operations in the cluster.

[0059] In this embodiment, the total number of processor cores in the cluster can be the total number of CPU cores in the cluster; the processor utilization rate can be the CPU utilization rate in the cluster; the cluster memory information can be the total memory information of each node in the cluster; the cluster memory utilization rate can be the memory utilization rate information of each node in the cluster; the cluster network throughput information can be the network throughput information in the cluster; and the network interface card (NIC) data information in the cluster can be the size information of the NICs in the cluster.

[0060] S102. Based on cluster parameter information, document quantity and document size, allocate multiple index shards for the data to be processed, and write the data to be processed into the multiple index shards. The multiple index shards are located on at least one node of the cluster.

[0061] In this embodiment of the application, after the data storage device obtains the cluster parameter information, the number of documents written to the historical index, and the size of the documents written to the historical index, the data storage device can allocate multiple index shards for the data to be processed according to the cluster parameter information, the number of documents written to the historical index, and the size of the documents written to the historical index, and write the data to be processed into the multiple index shards.

[0062] In this embodiment of the application, the process by which the data storage device allocates multiple index shards to the data to be processed based on cluster parameter information, the number of documents, and the document size can be described as follows: the data storage device determines the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size; after the data storage device determines the number of index shards, the data storage device allocates multiple index shards to the data to be processed based on the number of index shards.

[0063] In this embodiment of the application, the process by which the data storage device determines the number of index shards corresponding to the data to be processed based on cluster parameter information, the number of documents, and the document size can be as follows: the data storage device determines a first number of index shards based on cluster parameter information; a second number of index shards based on the number of documents; and a third number of index shards based on the document size. After the data storage device determines the first, second, and third numbers, it then determines the number of index shards based on these numbers.

[0064] In this embodiment, the data storage device can determine the first number of index shards based on the cluster parameter information, the number of documents historically written to the index, and the size of the documents historically written to the index, respectively; determine the second number of index shards based on the number of documents; and determine the third number of index shards based on the document size. Alternatively, the data storage device can first determine the first number of index shards based on the cluster parameter information, then determine the second number of index shards based on the number of documents, and finally determine the third number of index shards based on the document size. Another option is to first determine the third number of index shards based on the document size, then determine the second number of index shards based on the number of documents, and finally determine the first number of index shards based on the cluster parameter information. The specific determination can be made according to the actual situation, and this embodiment does not limit this.

[0065] In this embodiment, the data storage device determines the number of index shards based on the first quantity, the second quantity, and the third quantity. Alternatively, the data storage device can determine the average of the first, second, and third quantities and use this average as the number of index shards. Another method is to determine the first product of the first weight corresponding to the first quantity and the first quantity, the second product of the second weight corresponding to the second quantity and the second quantity, and the third product of the third weight corresponding to the third quantity and the third quantity. The data storage device then obtains the sum of the first, second, and third products and uses this sum as the number of index shards. The specific method can be determined according to actual circumstances, and this embodiment does not limit this approach.

[0066] It should be noted that the first weight, the second weight, and the third weight are preset weight values ​​in the data storage device. Alternatively, the first weight, the second weight, and the third weight can be weight values ​​determined by the data storage device based on the received weight allocation instruction. The specific weights can be determined according to the actual situation, and this application embodiment does not limit this.

[0067] In this embodiment of the application, the process of allocating multiple index shards for the data to be processed by the data storage device according to cluster parameter information, number of documents and document size also includes: the data storage device creating index aliases corresponding to the multiple index shards; after the data storage device creates the index aliases, the data storage device uses the index aliases to index the data to be processed in the cluster.

[0068] In this embodiment, the data storage device sets an index alias. Since the index alias can point to one index or multiple indexes, and one index corresponds to multiple index shards, when the index alias points to multiple indexes, the data storage device can directly retrieve the data information corresponding to the index alias from the data to be processed when indexing the data to be processed according to the index alias. This improves the speed of the data storage device when performing retrieval and increases the utilization rate of physical resources.

[0069] In this embodiment, the data storage device can create index aliases during the process of allocating multiple index shards for the data to be processed based on cluster parameter information, document quantity, and document size. The data storage device can also create index aliases after allocating multiple index shards for the data to be processed based on cluster parameter information, document quantity, and document size. The specific method can be determined according to the actual situation, and this embodiment does not limit it.

[0070] In this embodiment of the application, the data storage device can also adjust the number of nodes in the cluster according to the processor parameter information of the cluster, the memory information of the cluster, and the total number of read and write operations on the disk in the cluster; after the data storage device adjusts the number of nodes in the cluster, the data storage device updates the cluster parameter information according to the cluster after adjusting the number of nodes.

[0071] It should be noted that the data storage device can adjust the number of nodes in the cluster in a way that either increases or decreases the number of nodes in the cluster. The specific method can be determined according to the actual situation, and this application embodiment does not limit it.

[0072] In this embodiment, the data storage device can adjust the number of nodes in the cluster in either a preset time period or when it receives an instruction to adjust the number of nodes in the cluster. The specific method can be determined according to the actual situation, and this embodiment does not limit it.

[0073] In this application embodiment, the preset time period can be a time period configured for the data storage device, or it can be a time period information obtained by the data storage device from the received instruction information. The specific time period can be determined according to the actual situation, and this application embodiment does not limit it.

[0074] In this embodiment of the application, the preset time period can be within 2ns, 5ms, or 1s. The specific time period can be determined according to the actual situation, and this embodiment of the application does not limit it.

[0075] In this embodiment, the data storage device adjusts the number of nodes in the cluster based on the cluster's processor parameter information, cluster memory information, and the total number of disk read / write operations in the cluster. When the number of nodes in the cluster is too large, the data storage device reduces the number of nodes in the cluster; when the number of nodes in the cluster is too small, the data storage device increases the number of nodes in the cluster, thereby improving the utilization rate of node resources in the cluster.

[0076] In this embodiment of the application, the cluster parameter information may include the parameter information of each node in the cluster, such as the central processing parameter information of each node, the memory information of each node, the network throughput information of each node, and the disk read and write operation count information of each node.

[0077] Understandably, when creating an index for the data to be processed, the data storage device directly obtains cluster parameter information, the number of documents previously written to the index, and the size of those documents. Based on these information, it determines the number of index shards and allocates multiple index shards to the data to be processed. This eliminates the need for manual access to cluster configuration parameters and manual determination of the number of index shards, thus improving the intelligence of the data storage device when creating index shards.

[0078] Example 2

[0079] Based on the same inventive concept as Embodiment 1, this application provides a data storage device 1, corresponding to a data storage method; Figure 2 A schematic diagram of the composition structure of a data storage device provided in this application embodiment. Figure 1 The data storage device 1 may include:

[0080] The acquisition unit 11 is used to acquire cluster parameter information storing the data to be processed, the number of documents historically written to the index, and the size of documents historically written to the index when creating an index for the data to be processed.

[0081] The allocation unit 12 is used to allocate multiple index shards to the data to be processed according to the cluster parameter information, the number of documents and the document size, and write the data to be processed into the multiple index shards, wherein the multiple index shards are located on at least one node of the cluster.

[0082] In some embodiments of this application, the cluster parameter information includes at least one of the following:

[0083] The total number of processor cores storing the data to be processed, the utilization rate of the processors, the cluster's memory information, the cluster's memory utilization rate, the cluster's network throughput information, the network interface card data information in the cluster, the total number of disk read / write operations in the cluster, and the average number of disk read / write operations in the cluster.

[0084] In some embodiments of this application, the apparatus further includes a determining unit;

[0085] The determining unit is configured to determine the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size; and to allocate the multiple index shards to the data to be processed based on the number of index shards.

[0086] In some embodiments of this application,

[0087] The determining unit is configured to determine a first number of index shards based on the cluster parameter information; determine a second number of index shards based on the number of documents; determine a third number of index shards based on the document size; and determine the total number of index shards based on the first number, the second number, and the third number.

[0088] In some embodiments of this application, the apparatus further includes a creation unit;

[0089] The creation unit is used to create index aliases corresponding to the multiple index shards; and to index the data to be processed in the cluster using the index aliases.

[0090] In some embodiments of this application, the device further includes an adjustment unit;

[0091] The adjustment unit is used to adjust the number of nodes in the cluster based on the cluster's processor parameter information, cluster memory information, and the total number of disk read / write operations in the cluster; and to update the cluster parameter information based on the cluster after adjusting the number of nodes.

[0092] It should be noted that, in practical applications, the aforementioned acquisition unit 11 and allocation unit 12 can be implemented by the processor 13 on the data storage device 1, specifically by a CPU (Central Processing Unit), MPU (Microprocessor Unit), DSP (Digital Signal Processor), or Field Programmable Gate Array (FPGA), etc.; the aforementioned data storage can be implemented by the memory 14 on the data storage device 1.

[0093] This invention also provides a data storage device 1, such as... Figure 3 As shown, the data storage device 1 includes a processor 13, a memory 14, and a communication bus 15. The memory 14 communicates with the processor 13 through the communication bus 15. The memory 14 stores programs executable by the processor 13. When the program is executed, the data storage method described above is executed by the processor 13.

[0094] In practical applications, the aforementioned memory 14 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 13.

[0095] This application provides a computer-readable storage medium having a computer program thereon, which, when executed by a processor 13, implements the data storage method as described above.

[0096] Understandably, when creating an index for the data to be processed, the data storage device directly obtains cluster parameter information, the number of documents previously written to the index, and the size of those documents. Based on these information, it determines the number of index shards and allocates multiple index shards to the data to be processed. This eliminates the need for manual access to cluster configuration parameters and manual determination of the number of index shards, thus improving the intelligence of the data storage device when creating index shards.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

[0102] Industrial applicability

[0103] This application provides a data storage method, apparatus, and storage medium. Using the above method, when creating an index for the data to be processed, the data storage device directly obtains cluster parameter information, the number of documents historically written to the index, and the size of those documents. Based on these information, it determines the number of index shards, thereby allocating multiple index shards to the data to be processed. This eliminates the need for manual acquisition of cluster configuration parameters and manual determination of the number of index shards based on those parameters, thus improving the intelligence of the data storage device when creating index shards.

Claims

1. A data storage method, the method comprising: When creating an index for the data to be processed, obtain the cluster parameter information for storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index; Based on the cluster parameter information, the number of documents, and the document size, multiple index shards are allocated to the data to be processed, and the data to be processed is written into the multiple index shards, wherein the multiple index shards are located on at least one node of the cluster.

2. The method according to claim 1, wherein, The cluster parameter information includes at least one of the following: The cluster's total number of processor cores, processor utilization, cluster memory information, cluster memory utilization, cluster network throughput information, network interface card data information in the cluster, total number of disk read / write operations in the cluster, and average number of disk read / write operations in the cluster.

3. The method according to claim 1, wherein, The step of allocating multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size includes: Based on the cluster parameter information, the number of documents, and the document size, the number of index shards corresponding to the data to be processed is determined; The multiple index shards are allocated to the data to be processed based on the number of index shards.

4. The method according to claim 3, wherein, The step of determining the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size includes: Based on the cluster parameter information, determine the first number of index shards; Based on the number of documents, determine the second number of index shards; Based on the document size, determine the third number of index shards; The number of index shards is determined based on the first quantity, the second quantity, and the third quantity.

5. The method according to claim 3, wherein, The step of allocating multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size further includes: Create index aliases corresponding to the multiple index shards; The data to be processed is indexed in the cluster using the index alias.

6. The method according to claim 1, wherein, The method further includes: The number of nodes in the cluster is adjusted based on the cluster's processor parameters, memory information, and total number of disk read / write operations. Update the cluster parameter information based on the cluster after adjusting the number of nodes.

7. A data storage device, the device comprising: The acquisition unit is used to acquire cluster parameter information storing the data to be processed, the number of documents historically written to the index, and the size of the documents historically written to the index when creating an index for the data to be processed. An allocation unit is configured to allocate multiple index shards to the data to be processed based on the cluster parameter information, the number of documents, and the document size, and to write the data to be processed into the multiple index shards, wherein the multiple index shards are located on at least one node of the cluster.

8. The apparatus according to claim 7, wherein, The cluster parameter information includes at least one of the following: The total number of processor cores storing the data to be processed, the utilization rate of the processors, the cluster's memory information, the cluster's memory utilization rate, the cluster's network throughput information, the network interface card data information in the cluster, the total number of disk read / write operations in the cluster, and the average number of disk read / write operations in the cluster.

9. The apparatus according to claim 7, wherein, The device further includes a determining unit; The determining unit is configured to determine the number of index shards corresponding to the data to be processed based on the cluster parameter information, the number of documents, and the document size; and to allocate the multiple index shards to the data to be processed based on the number of index shards.

10. The apparatus according to claim 9, wherein, The determining unit is configured to determine a first number of index shards based on the cluster parameter information; determine a second number of index shards based on the number of documents; determine a third number of index shards based on the document size; and determine the total number of index shards based on the first number, the second number, and the third number.

11. The apparatus according to claim 9, wherein, The device also includes a creation unit; The creation unit is used to create index aliases corresponding to the multiple index shards; and to index the data to be processed in the cluster using the index aliases.

12. The apparatus according to claim 7, wherein, The device also includes an adjustment unit; The adjustment unit is used to adjust the number of nodes in the cluster based on the cluster's processor parameter information, cluster memory information, and the total number of disk read / write operations in the cluster. Update the cluster parameter information based on the cluster after adjusting the number of nodes.

13. A data storage device, the device comprising: The system includes a memory, a processor, and a communication bus, wherein the memory communicates with the processor via the communication bus, the memory stores a data storage program executable by the processor, and when the data storage program is executed, the processor performs the method as described in any one of claims 1 to 6.

14. A storage medium having a computer program stored thereon, used in a data storage device, wherein, When executed by a processor, the computer program implements the method described in any one of claims 1 to 6.

Images