Index fragment number adjustment method and device, equipment and storage medium

By monitoring and adjusting the write rate and shard data volume of the elastic search cluster, dynamically adjusting the number of index shards, the problem of insufficient number of index shards is solved, and the stability and robustness of the cluster is improved.

CN119025524APending Publication Date: 2024-11-26CHONGQING JINKANG NEW ENERGY VEHICLE CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411122385.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In elastic search engines, the number of shards of the original index is not enough to support efficient query and write operations, resulting in uneven cluster disk occupation, difficulty in shard migration and recovery, and poor cluster stability and robustness.

Method used

By monitoring the current write rate and shard data volume of each node of the elastic search cluster, adjust the number of index shards, and dynamically create new indexes to ensure the reasonable number of shards, determine the number of shards based on the real-time business data volume, and avoid unreasonable shards Adjust the number of films.

Benefits of technology

It improves the real-time adjustment of the number of index shards, reduces unnecessary shard migration, enhances the stability and robustness of the cluster, and ensures the rationality and adaptability of the number of shards.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025524A_ABST
    Figure CN119025524A_ABST
Patent Text Reader

Abstract

The invention provides an index fragment number adjustment method and device, equipment and a storage medium, and the method comprises the steps: monitoring the current writing rate of each index in each node of an elastic search cluster, and the data size of each fragment under the index; if monitoring that the data volume of each fragment under any index meets a preset condition, taking the index as a target index; calculating according to the maximum data volume corresponding to the target index and the current writing rate to obtain a target fragment number, the maximum data volume of each fragment under the target index being consistent; and creating an index under the node bearing the target index based on the target fragment number so as to complete the adjustment of the index fragment number. In this way, reasonable adjustment of the index fragments is achieved, and the stability and robustness of the cluster are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of index technology, and in particular to a method, device, equipment and storage medium for adjusting the number of index shards. Background Art

[0002] ElasticSearch (ES) is a distributed, highly scalable, and highly real-time search and data analysis engine used in search services, log analysis, data analysis, and storage. In ElasticSearch, a cluster consists of one or more nodes that jointly hold data and provide indexing and search functions. A node is a server in a cluster that stores data and participates in the cluster's indexing and search functions. An index is a collection of documents. An index consists of one or more shards for storing and searching data. These shards can be distributed and stored on different nodes. With the continuous growth of data volume and the complexity of business needs, the number of shards in the original index may not be sufficient to support efficient query and write operations, and the number of shards in the index needs to be readjusted.

[0003] In related technologies, the allocation quantity is usually adjusted based on the current indexed data volume and the predicted data growth rate. However, the growth rate prediction may be biased, and the calculated number of shards may be unreasonable, resulting in uneven cluster disk occupancy, difficulty in shard migration and recovery, and poor cluster stability and robustness. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] In view of the above-mentioned shortcomings of the prior art, the present application discloses a method, device, equipment and storage medium for adjusting the number of index shards to solve the above-mentioned technical problem of how to reasonably adjust the number of index shards.

[0006] In the first aspect, the present application provides a method for adjusting the number of index shards, including: monitoring the current write rate of each index in each node of the elastic search cluster, and the data volume of each shard under the index; if it is monitored that the data volume of each shard under any of the indexes meets the preset conditions, the index is used as the target index; according to the maximum data volume corresponding to the target index and the current write rate, a target number of shards is obtained by calculation, and the maximum data volume of each shard under the target index is consistent; based on the target number of shards, a new index is created under the node carrying the target index to complete the adjustment of the number of index shards.

[0007] In one embodiment of the present application, the monitoring of the current write rate of each index in the elastic search cluster node includes: detecting the total amount of data of the index within the preset cycle time at intervals of a preset cycle time; calculating according to the preset cycle time and the total amount of data to obtain the write rate of the index within the preset cycle time; and performing median calculation on the write rates corresponding to each of the preset cycle times of the current preset number of cycles to determine the current write rate.

[0008] In one embodiment of the present application, after obtaining the target number of shards, it also includes: comparing the target number of shards with the preset maximum number of shards and the preset minimum number of shards respectively; if the target number of shards is less than the preset minimum number of shards, adjusting the target number of shards to the preset minimum number of shards; if the target number of shards is greater than the preset maximum number of shards, adjusting the target number of shards to the preset maximum number of shards, and generating a warning message.

[0009] In one embodiment of the present application, if it is monitored that the data volume of each shard under any of the indexes meets a preset condition, the index is used as the target index, including: comparing the data volume of each shard under the index with the maximum data volume; if the data volume of each shard under the index is greater than the maximum data volume, a new index is created and the index is used as the target index; if the data volume of any of the shards under the index is less than or equal to the maximum data volume, no new index is created.

[0010] In one embodiment of the present application, before monitoring the current write rate of each index in each node of the elastic search cluster, it also includes: determining a mapping relationship between each node and the index under the elastic search cluster; configuring a cluster table of the elastic search cluster based on the mapping relationship, the cluster table at least including the number of nodes, the upper limit of the number of shards that each node can carry for the index, the maximum amount of data of the shards under each index, the preset minimum number of shards and the preset maximum number of shards under each index, and the preset cycle time.

[0011] In one embodiment of the present application, after configuring the cluster table of the elastic search cluster based on the mapping relationship, it also includes: configuring an index table, each index table corresponds to an index template, the index table includes multiple indexes, the index table includes at least the name and version number of the index template, the current number of shards of each index under the index template, and the name of each index, the current write rate, the total data volume and the write rate within the current preset cycle time, the name of the index includes the name and creation time of the index template; wherein the index table is updated at every preset cycle time.

[0012] In one embodiment of the present application, creating a new index under the node carrying the target index based on the target number of shards includes: comparing the target number of shards and the current number of shards of the target index, and determining the index template corresponding to the target index as the target index template; if the target number of shards is consistent with the current number of shards, creating a new index under the target index template according to the name of the target index template and the current time; if the target number of shards is inconsistent with the current number of shards, updating the version number corresponding to the target index template and updating the current number of shards to the target number of shards, and then creating a new index under the target index template according to the name of the target index template and the current time.

[0013] In the second aspect, the present application provides a device for adjusting the number of index shards, the device comprising: a monitoring module for monitoring the current write rate of each index in each node of the elastic search cluster, and the data volume of each shard under the index; a determination module for taking the index as the target index if it is monitored that the data volume of each shard under any of the indexes meets the preset conditions; a calculation module for calculating according to the maximum data volume corresponding to the target index and the current write rate to obtain the target number of shards, the maximum data volume of each shard under the target index being consistent; a creation module for creating a new index under the node carrying the target index based on the target number of shards to complete the adjustment of the number of index shards.

[0014] In a third aspect, the present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method described in the above embodiments.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes the method described in the above embodiments.

[0016] Beneficial effects of the present application: The present application proposes a method, device, equipment and storage medium for adjusting the number of index shards. First, by monitoring the amount of data of each shard under each index in each node of the elastic search cluster, it is determined whether the index needs to be newly created, that is, whether the number of shards needs to be adjusted. If adjustment is required, the index is used as the target index. In this way, a new index is dynamically created based on the number of shards under the real-time monitoring index. The real-time adjustment of the number of index shards is high, which to a certain extent avoids the situation where a large number of indexes need to be re-sharded, affecting the performance of the cluster, and improving the stability of the cluster; second, the target number of shards is calculated according to the maximum amount of data corresponding to the target index and the current write rate of the index, and a new index is created under the node carrying the target index based on the target number of shards to complete the adjustment of the number of index shards. In this way, a reasonable target number of shards is calculated based on the current write rate of the index obtained by real-time monitoring, that is, the adjustment of the number of index shards is determined based on the real-time business data volume of the index, and the adjusted number of index shards is reasonable, which further improves the stability and robustness of the cluster.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 is a schematic diagram of an index shard quantity adjustment architecture shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a task scheduling process in an architecture shown by an exemplary embodiment of the present application;

[0021] Figure 3 is a flowchart of a method for adjusting the number of index shards shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an index template table configuration shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a block diagram of a device for adjusting the number of index fragments shown in an exemplary embodiment of the present application;

[0024] Figure 6It is a structural diagram of a computer system suitable for implementing the electronic device of the present application, shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0026] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0027] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0028] See also Figure 1 and Figure 2 , are schematic diagrams of an index shard quantity adjustment architecture and a task scheduling process in the architecture, respectively, shown as an exemplary embodiment of the present application. Figure 1 The architecture shown includes a system setting module, an index data management module, a new index creation determination module, an index sharding calculation module and a task scheduling module, wherein the system setting module is configured with at least one cluster table; the index data management module is configured with multiple index tables corresponding to the cluster table, and the index tables and index templates correspond one to one; the new index module is used to determine whether there is an index in the index table that needs to be recreated as the target index; the index sharding calculation module is used to calculate the target sharding data of the target index, and to create a new index based on the target sharding data and the index template corresponding to the target index; the task scheduling module is used to schedule the execution of the index data management module, the new index creation determination module, and the index sharding calculation module, and to create a scheduling table to record the execution content of each scheduling. Figure 2As shown, the scheduling process of the task scheduling module is: when the task scheduling time is reached, that is, every preset periodic time, the data in the index table corresponding to the index template in the index data management module is read and updated, and then the new index module is scheduled to determine whether a new index is needed based on the updated data, that is, the current data volume of each shard under the index, and mark the index as the target index, and finally schedule the index shard calculation module to calculate the target number of shards, that is, the number of new shards, based on the updated data, that is, the current write rate of the index, and mark the number of new shards and the index template corresponding to the target index. The task scheduling module can also call the index data management module to create a new index template when it is needed. In this way, the automatic adjustment of index shards is improved.

[0029] See also Figure 3 , is a flowchart of a method for adjusting the number of index shards shown in an exemplary embodiment of the present application. The method can be used Figure 1 In the architecture shown, Figure 3 As shown, in an exemplary embodiment, the method for adjusting the number of index shards includes at least steps S310 to S340, which are described in detail as follows:

[0030] Step S310, monitor the current write rate of each index in each node of the elastic search cluster, and the data volume of each shard under the index.

[0031] Specifically, a mapping relationship between each node and index under an elastic search cluster is determined, and a cluster table of the elastic search cluster is configured based on the mapping relationship. The cluster table includes at least the number of nodes, the upper limit of the number of shards that each node can carry for any index, the maximum amount of data in the shards under each index, the preset minimum number of shards and the preset maximum number of shards under each index, and the preset cycle time.

[0032] In one embodiment of the present application, an elastic search cluster is configured with a cluster table, and each data included in the cluster table can be configured with default initial data, or adjusted according to actual needs, which is not limited here. The cluster table is configured with the number of nodes, which can be obtained according to the actual situation of the cluster. In actual applications, the cluster table is configured in the form of table structure data. For example, the field es_data_number=6 indicates that the number of nodes is 6, es_data_max_index_number=2 indicates that the upper limit of the number of shards of an index that each node can carry is 2, es_index_max_shards=es_data_number*es_data_max_index_number indicates that the maximum number of shards of each index is the number of nodes, that is, the preset maximum number of shards is equal to the number of nodes, es_min_shards=1 indicates that the minimum number of shards of each index is 1, that is, the preset minimum number of shards is 1, shard_max_data=30G (gigabytes) indicates that the maximum amount of data for each shard under the index is 30G, and k=2h (hours) indicates that the preset cycle time of the monitoring interval is set to 2 hours.

[0033] Specifically, an index table is configured, each index table corresponds to an index template, the index table includes multiple indexes, and the index table includes at least the name and version number of the index template, the current number of shards of each index under the index template, and the name of each index, the current write rate, the total data volume and write rate within the current preset cycle time, and the name of the index includes the name and creation time of the index template; wherein the index table is updated at every preset cycle time interval, and each time the current number of shards is updated, the version number of the index template increases by a natural number.

[0034] In one embodiment of the present application, different index templates have different settings for each data, and different index templates can be set according to different business requirements and data characteristics to meet different query requirements. In order to facilitate management, the prefix of the index name under the same index mode is set to the name of the index template, and the suffix is ​​set to the creation time of the index, and the creation time is accurate to seconds.

[0035] Figure 4 FIG. 1 is a schematic diagram of an index template table configuration shown in an exemplary embodiment of the present application. Figure 4As shown, Index_pattern_name represents the name of the index template, that is, the name of the index table; version represents the version number of the index template, and the initial value is set to 1; index_name = index_pattern_name + current time represents the name of the index; shards represents the current number of shards of the index, and the initial value is set to the number of nodes; index_name_current_data represents the write rate of the index within the current preset cycle time k, and the initial value is set to 0; index_name_current_data represents the total amount of data of the index within the preset cycle time k, and the initial value is set to 0; rate represents the current write rate, and the initial value is set to 0.

[0036] By configuring the cluster table and index table in the above way, you can better manage the indexes in the cluster. The cluster can add new index templates or adjust the settings of existing index templates according to changes in business needs, thereby improving the scalability of the cluster.

[0037] Specifically, the current write rate is determined as follows: the total amount of data of the index within the preset cycle time is detected at every preset cycle time interval; the write rate of the index within the preset cycle time is calculated based on the preset cycle time and the total amount of data; the median of the write rates corresponding to each preset cycle time of the current preset cycle number is calculated to determine the current write rate.

[0038] In one embodiment of the present application, in order to avoid the write rate of a single cycle time being affected by various temporary factors, such as a sudden increase in system load, network delay, etc., which may cause large fluctuations in the write rate, the current write rate of the index is calculated based on multiple write rates under multiple consecutive preset cycle times. For example, if the number of preset cycles is set to 10, the write rate of the index in 10 consecutive preset cycle times can reflect the long-term trend of the write rate, smooth out temporary fluctuations in data writing, and make the obtained current write rate more accurate and reliable.

[0039] Step S320: If it is monitored that the data volume of each shard under any index meets the preset condition, the index is used as the target index.

[0040] Specifically, the data volume of each shard under the index is compared with the maximum data volume; if the data volume of each shard under the index is greater than the maximum data volume, a new index is created and the index is used as the target index; if the data volume of any shard under the index is less than or equal to the maximum data volume, no new index is created.

[0041] In one embodiment of the present application, the preset condition is that the data volume of each shard under the index is greater than the maximum data volume. If data_size represents the data volume of the shard and shard_max_data represents the maximum data volume, the expression for comparing the data volume of a shard with the maximum data volume is is_need_create_index=data_size>shard_max_data? True:False.

[0042] In one embodiment of the present application, multiple target indexes may exist simultaneously.

[0043] Step S330, calculate according to the maximum data volume corresponding to the target index and the current write rate to obtain the target number of shards, and the maximum data volume of each shard under the target index is consistent.

[0044] In one embodiment of the present application, new_shard represents the target number of shards, rate represents the current write rate, shard_max_data represents the maximum data volume, and the expression for calculating the target number of shards is new_shards=int(ceil(rate / shard_max_data)). In this way, when the amount of business data indexed increases, the number of shards increases, and when the amount of business data decreases, the number of shards decreases. There is no need to predict the size of the business data volume, and the number of shards can be adjusted flexibly and reasonably.

[0045] Through the above method, the write rate of multiple consecutive cycles is monitored in real time, and the number of index shards can be dynamically adjusted to adapt to different business needs. When the amount of business data indexed increases, the number of shards increases, and when the amount of business data decreases, the number of shards decreases, which solves the problems of unreasonable number of shards and poor real-time adjustment of the number of shards. In this way, the migration speed of the shards is fast, the recovery ability of the shards is high, and the stability and robustness of the cluster are improved.

[0046] In one embodiment of the present application, in order to prevent the number of shards of the index from being too small, resulting in too much data on a single shard affecting performance, and the difficulty of shard migration and long recovery time when the number of re-created index shards is too large, which wastes cluster resources, when adjusting the number of index shards, it is necessary to limit the range of adjustment of the number of shards. Therefore, it is necessary to compare the target number of shards with the preset maximum number of shards and the preset minimum number of shards respectively; if the target number of shards is less than the preset minimum number of shards, the target number of shards is adjusted to the preset minimum number of shards; if the target number of shards is greater than the preset maximum number of shards, the target number of shards is adjusted to the preset maximum number of shards, and a warning message is generated. In this way, the number of shards of the newly created index is ensured to be within the appropriate range, which improves the stability of the cluster, and when the number of shards exceeds the appropriate range, an alarm message will be generated to remind relevant personnel, even if the cluster is expanded or the data volume is filtered, to further ensure the stability of the cluster.

[0047] Step S340: Create a new index under the node that carries the target index based on the target number of shards to complete the adjustment of the number of index shards.

[0048] In one embodiment of the present application, because the cluster needs to modify the number of shards through the Reindex (i.e., re-create the index) operation, that is, recreate the old index as a new index, and specify the required number of shards when creating it, and then migrate the data in the old index to the new index until the data of the new index is written. The old index can be replaced or deleted. Therefore, if the number of index shards needs to be adjusted, it is necessary to create a new index based on the required number of shards.

[0049] Specifically, the target number of shards and the current number of shards of the target index are compared, and the index template corresponding to the target index is determined as the target index template; if the target number of shards is consistent with the current number of shards, a new index is created under the target index template according to the name of the target index template and the current time; if the target number of shards is inconsistent with the current number of shards, the version number corresponding to the target index template is updated and the current number of shards is updated to the target number of shards, and then a new index is created under the target index template according to the name of the target index template and the current time.

[0050] In one embodiment of the present application, if the target number of shards is consistent with the current number of shards of the target index template, then the current number of shards can remain unchanged, and a new index can be directly created in the index table corresponding to the target index template. The name of the new index is formed by combining the target index template and the current time; if the target number of shards is inconsistent with the current number of shards of the target index template, then it is necessary to increase the version number under the target index template by a natural number, set the current number of shards to the target number of shards, and create a new index in the index table corresponding to the target index template.

[0051] The index shard quantity adjustment method provided by the present application is suitable for mass log information storage and query, and can support log management of any project business party. It has the following advantages: first, by monitoring the data volume of each shard under each index in each node of the elastic search cluster, it is determined whether the index needs to be newly created, that is, whether the number of shards needs to be adjusted. If adjustment is required, the index is used as the target index. In this way, a new index is dynamically created based on the number of shards under the real-time monitoring index, and the real-time adjustment of the number of index shards is high, which to a certain extent avoids the situation where a large number of indexes need to be re-sharded, affecting the performance of the cluster, and improving the stability of the cluster; second, the target number of shards is calculated according to the maximum data volume corresponding to the target index and the current write rate of the index, and a new index is created under the node carrying the target index based on the target number of shards to complete the adjustment of the number of index shards. In this way, a reasonable target number of shards is calculated based on the current write rate of the index obtained by real-time monitoring, that is, the adjustment of the number of index shards is determined based on the real-time business data volume of the index, and the adjusted number of index shards is reasonable, which further improves the stability and robustness of the cluster.

[0052] See also Figure 5 , is a block diagram of an index shard quantity adjustment device shown in an exemplary embodiment of the present application. Figure 5 As shown, in an exemplary embodiment, the index shard quantity adjustment device at least includes a monitoring module 510, a determination module 520, a calculation module 530, and a creation module 540, which are described in detail as follows:

[0053] Monitoring module 510, used to monitor the current write rate of each index in each node of the elastic search cluster, and the amount of data in each shard under the index;

[0054] The determination module 520 is used to use the index as the target index if it is monitored that the data volume of each shard under any index meets the preset condition;

[0055] A calculation module 530 is used to calculate the target number of shards according to the maximum data volume corresponding to the target index and the current write rate, and the maximum data volume of each shard under the target index is consistent;

[0056] The creation module 540 is used to create a new index under the node carrying the target index based on the target number of shards to complete the adjustment of the number of index shards.

[0057] It should be noted that the index shard quantity adjustment device provided in the above embodiment and the index shard quantity adjustment method provided in the above embodiment belong to the same concept, wherein the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0058] The present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method for adjusting the number of index shards as in the above embodiment.

[0059] See also Figure 6 , shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. It should be noted that, Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0060] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603, such as executing the method in the above embodiment. In the RAM 603, various programs and data required for system operation are also stored. The CPU 601, ROM 602 and RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0061] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom is installed into the storage section 608 as needed.

[0062] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 609, and / or installed from a removable medium 611. When the computer program is executed by a central processing unit (CPU) 601, various functions defined in the system of the present application are executed.

[0063] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the vehicle-side data forwarding method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0064] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, system or device. A computer program contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0065] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0066] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0067] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for adjusting the number of index shards, characterized in that: include: Monitor the current write rate of each index in each node of the elastic search cluster, as well as the amount of data in each shard under the index; If it is monitored that the data volume of each shard under any of the indexes meets the preset conditions, the index is used as the target index; Calculate the target number of shards according to the maximum data volume corresponding to the target index and the current write rate, and the maximum data volume of each shard under the target index is consistent; Based on the target number of shards, a new index is created under the node that carries the target index to complete the adjustment of the number of index shards.

2. The method for adjusting the number of index fragments according to claim 1, characterized in that: The monitoring of the current write rate of each index in the elastic search cluster node includes: Detecting the total amount of data of the index within the preset period at intervals of a preset period; Calculating according to the preset cycle time and the total amount of data, obtaining a write rate of the index within the preset cycle time; The median of the write rates corresponding to the preset cycle times of the current preset cycle number is calculated to determine the current write rate.

3. The method for adjusting the number of index fragments according to claim 1, characterized in that: After obtaining the target number of shards, the method further includes: Compare the target number of shards with a preset maximum number of shards and a preset minimum number of shards respectively; If the target number of shards is less than the preset minimum number of shards, adjusting the target number of shards to the preset minimum number of shards; If the target number of shards is greater than the preset maximum number of shards, the target number of shards is adjusted to the preset maximum number of shards, and a warning message is generated.

4. The method for adjusting the number of index fragments according to any one of claims 1 to 3, characterized in that: If it is monitored that the data volume of each shard under any index meets a preset condition, the index is used as a target index, including: Compare the data volume of each of the shards under the index with the maximum data volume respectively; If the data volume of each of the shards under the index is greater than the maximum data volume, a new index is created and the index is used as the target index; If the data volume of any of the shards under the index is less than or equal to the maximum data volume, no new index is created.

5. The method for adjusting the number of index fragments according to claim 2, characterized in that: Before monitoring the current write rate of each index in each node of the elastic search cluster, the method further includes: Determine a mapping relationship between each of the nodes and the index under the elastic search cluster; A cluster table of the elastic search cluster is configured based on the mapping relationship, and the cluster table includes at least the number of nodes, the upper limit of the number of shards that each node can carry for the index, the maximum amount of data of the shards under each index, the preset minimum number of shards and the preset maximum number of shards under each index, and the preset cycle time.

6. The method for adjusting the number of index fragments according to claim 5, characterized in that: After configuring the cluster table of the elastic search cluster based on the mapping relationship, the method further includes: Configure an index table, each index table corresponds to an index template, the index table includes multiple indexes, the index table includes at least the name and version number of the index template, the current number of shards of each index under the index template, and the name of each index, the current write rate, the total data volume and the write rate within the current preset cycle time, and the name of the index includes the name and creation time of the index template; Wherein, the index table is updated at intervals of the preset periodic time.

7. The method for adjusting the number of index fragments according to claim 6, characterized in that: The creating a new index under the node carrying the target index based on the target number of shards includes: Compare the target number of shards and the current number of shards of the target index, and determine the index template corresponding to the target index as the target index template; If the target number of shards is consistent with the current number of shards, a new index is created under the target index template according to the name of the target index template and the current time; If the target number of shards is inconsistent with the current number of shards, the version number corresponding to the target index template is updated and the current number of shards is updated to the target number of shards, and then a new index is created under the target index template according to the name of the target index template and the current time.

8. A device for adjusting the number of index fragments, characterized in that: include: A monitoring module is used to monitor the current write rate of each index in each node of the elastic search cluster, and the amount of data in each shard under the index; A determination module, configured to use the index as a target index if it is detected that the data volume of each shard under any index meets a preset condition; A calculation module, configured to calculate a target number of shards according to a maximum data volume corresponding to the target index and the current write rate, wherein the maximum data volume of each shard under the target index is consistent; A creation module is used to create a new index under the node carrying the target index based on the target number of shards to complete the adjustment of the number of index shards.

9. An electronic device, characterized in that: include: processor, memory, and communication bus; The communication bus is used to connect the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to make a computer execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Resource scheduling method, device and system for elastic search cluster

    CN117093357A