A data management method, device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose a data management method, apparatus, computer device, and storage medium. Embodiments of the present application can obtain management data containing data stream objects in a search engine, where the data stream objects are associated with multiple indexes of time series data in the search engine, and the data stream objects include multiple index shards; query the management data to obtain index description parameters of the data stream objects in multiple different dimensions; calculate the reference index shard numbers of the data stream objects in different dimensions according to the index description parameters in different dimensions; generate the target adjusted shard number of the data stream objects according to the current index shard number of the data stream objects and the reference index shard numbers of the data stream objects in different dimensions; and adjust the index shards of the data stream objects based on the target adjusted shard number to obtain the adjusted index shards. Through the embodiments of the present application, the management and maintenance costs of time series data in the search engine can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a data management method, apparatus, computer equipment, and storage medium. Background Technology

[0002] A search engine is a retrieval technology that uses specific strategies and algorithms to retrieve information from the internet and provide it to users based on their needs. Search engines allow for centralized data storage and the performance of operations such as searching, indexing, and analyzing that data. Time-series data includes data recorded in chronological order. Generally, time-series data is retained for a limited period and reflects the development trajectory of a business over a specific timeframe. When a search engine stores time-series data, due to its time-sensitive nature, it needs to periodically process expired data to improve the reliability of the processing. The inventors of this application have discovered in their practice with existing technologies that the management of time-series data in search engines suffers from high maintenance costs. Summary of the Invention

[0003] This application provides a data management method, apparatus, computer equipment, and storage medium that can reduce the management and maintenance costs of time-series data in search engines.

[0004] This application provides a data management method, including:

[0005] The management data of the search engine containing data stream objects is obtained, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards;

[0006] The management data is queried to obtain the index description parameters of the data stream object in multiple different dimensions;

[0007] Based on the index description parameters of different dimensions, calculate the number of reference index fragments of the data stream object in different dimensions;

[0008] Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, generate the target adjustment number of shards for the data stream object.

[0009] The index shards of the data stream object are adjusted based on the target number of shards to obtain the adjusted index shards.

[0010] Accordingly, embodiments of this application also provide a data management device, including:

[0011] The acquisition unit is used to acquire management data containing data stream objects in the search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards;

[0012] The query unit is used to query the management data to obtain the index description parameters of the data stream object in multiple different dimensions;

[0013] The calculation unit is used to calculate the number of reference index fragments of the data stream object in different dimensions based on the index description parameters of different dimensions.

[0014] The generation unit is used to generate the target adjustment number of the data stream object based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions;

[0015] The adjustment unit is used to adjust the index shards of the data stream object based on the target number of adjustment shards, so as to obtain the adjusted index shards.

[0016] In one embodiment, the generation unit may include:

[0017] A subunit is defined to determine the target number of reference index fragments from the number of reference index fragments in the different dimensions;

[0018] The comparison subunit is used to compare the target reference index shard number with the current index shard number of the data stream object to obtain a comparison result.

[0019] A generation subunit is used to generate the target adjusted number of shards for the data stream object based on the comparison result.

[0020] In one embodiment, the comparison subunit may include:

[0021] An amplification module is used to amplify the current number of index fragments to obtain an amplified number of index fragments.

[0022] The comparison module is used to compare the target reference index fragment number with the magnified index fragment number;

[0023] In one embodiment, the generating subunit may include:

[0024] The determination module is used to determine the target reference index fragment number as the target adjustment fragment number when the target reference index fragment number is greater than the magnified index fragment number.

[0025] In one embodiment, the generation unit may further include:

[0026] The shrinking subunit is used to shrink the current number of index fragments when the target reference index fragment number is less than or equal to the amplified index fragment number, so as to obtain the shrunken index fragment number.

[0027] The narrowing comparison subunit is used to compare the target reference index fragment number with the narrowed index fragment number;

[0028] A subunit is defined to determine the number of reduced index fragments as the target adjusted fragment number when the target reference index fragment number is less than the reduced index fragment number.

[0029] In one embodiment, the data management device may further include:

[0030] The data acquisition unit is used to acquire the time-series data to be updated when data updates are performed.

[0031] The mode determination unit is used to determine the writing mode of the time-series data to be updated based on the service information of the data to be updated.

[0032] The writing unit is used to write the timing data to be updated into the target write index of the data stream object based on the writing mode.

[0033] In one embodiment, the writing unit may include:

[0034] The parsing subunit is used to parse the time series data to be updated to obtain the time information of the time series data to be updated;

[0035] The first identification subunit is used to identify the time range information corresponding to each index in the data stream object;

[0036] The matching subunit is used to match the time information of the time series data to be updated with the time range information corresponding to the index to obtain the matching result;

[0037] The first index determination subunit is used to determine the target write index corresponding to the time-series data to be updated in the index of the data stream object based on the matching result.

[0038] The first write subunit is used to write the timing data to be updated into the target write index.

[0039] In one embodiment, the writing unit may include:

[0040] The second identification subunit is used to identify the creation identifier corresponding to each index in the data stream object;

[0041] The third identification subunit is used to identify the index to be written in the index of the data stream object based on the creation identifier;

[0042] The second index determination subunit is used to identify the status information of the index to be written, and determine the target write index based on the status information of the index to be written;

[0043] The second write subunit is used to write the timing data to be updated into the target write index.

[0044] In one embodiment, the second index determining subunit may include:

[0045] The matching module is used to match the status information to be written to the index with preset status conditions;

[0046] An index determination module is used to determine the index to be written as the target write index when the status information of the index to be written meets the preset status conditions.

[0047] An index identification module is used to identify associated indexes related to the index to be written when the status information of the index to be written does not meet the preset status conditions.

[0048] A write module is used to determine the associated index as the target write index.

[0049] In one embodiment, the query unit may include:

[0050] The query subunit is used to query the management data to obtain the initial index description parameters of the data flow object in multiple different dimensions;

[0051] The pattern recognition subunit is used to identify the pattern of parameter changes of the initial index description parameters;

[0052] The denoising subunit is used to denoise the initial index description parameters according to the parameter variation pattern, so as to obtain the index description parameters of the data stream object in multiple different dimensions.

[0053] In one embodiment, the data management device may further include:

[0054] The display unit is used to display the search engine configuration interface in response to configuration trigger operations;

[0055] The parameter acquisition unit is used to acquire configuration parameters through the search engine configuration interface;

[0056] An object generation unit is used to generate a data stream object according to the configuration parameters;

[0057] The encapsulation unit is used to encapsulate the data stream object to obtain the management data.

[0058] In one embodiment, the object generation unit may include:

[0059] The strategy creation subunit is used to create an index lifecycle management strategy based on the configuration parameters, wherein the index lifecycle management strategy includes a strategy name;

[0060] The template generation subunit is used to generate a mapping relationship between the index and the business content according to the configuration parameters, and to generate an index template according to the mapping relationship and the policy name of the index lifecycle management strategy.

[0061] The object generation subunit is used to generate the data stream object based on the configuration parameters and the template name of the index template.

[0062] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0063] Accordingly, this application also provides a storage medium storing instructions that, when executed by a processor, implement any of the data management methods provided in this application.

[0064] This application embodiment can obtain management data of a data stream object contained in a search engine. The data stream object is associated with multiple indexes of time-series data in the search engine, and the data stream object includes multiple index shards. The management data is queried to obtain index description parameters of the data stream object in multiple different dimensions. Based on the index description parameters of different dimensions, the number of reference index shards of the data stream object in different dimensions is calculated. Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, a target adjustment number of shards for the data stream object is generated. The index shards of the data stream object are adjusted based on the target adjustment number of shards to obtain the adjusted index shards, thereby reducing the management and maintenance costs of time-series data in the search engine. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1This is a schematic diagram illustrating a scenario of the data management method provided in an embodiment of this application;

[0067] Figure 2 This is a flowchart illustrating the data management method provided in an embodiment of this application;

[0068] Figure 3 This is a schematic diagram illustrating a scenario for generating a data stream object according to an embodiment of this application;

[0069] Figure 4 This is a schematic diagram of a data writing scenario provided in an embodiment of this application;

[0070] Figure 5 This is another schematic diagram of a data writing scenario provided in the embodiments of this application;

[0071] Figure 6 This is another schematic diagram of a data writing scenario provided in the embodiments of this application;

[0072] Figure 7 This is another schematic diagram of a data writing scenario provided in the embodiments of this application;

[0073] Figure 8 This is another flowchart illustrating the data management method provided in the embodiments of this application;

[0074] Figure 9 This is another flowchart illustrating the data management method provided in the embodiments of this application;

[0075] Figure 10 This is a schematic diagram of the structure of the data management device provided in the embodiments of this application;

[0076] Figure 11 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. However, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0078] This application proposes a data management method, which can be executed by a data management device integrated into a computer device. The computer device may include at least one of a terminal and a server. That is, the data management method proposed in this application can be executed by a terminal, a server, or jointly by a terminal and a server capable of communicating with each other.

[0079] The terminal may include, but is not limited to, smartphones, tablets, laptops, personal computers (PCs), smart home appliances, wearable electronic devices, VR / AR devices, in-vehicle terminals, intelligent voice interaction devices, etc.

[0080] A server can be an interconnecting server between multiple heterogeneous systems or a backend server. It can also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms, etc.

[0081] It should be noted that the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0082] In one embodiment, such as Figure 1 The data management device can be integrated into computer devices such as terminals or servers to implement the data management method proposed in this application embodiment. Specifically, server 11 can obtain management data containing data stream objects in the search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards; query the management data to obtain index description parameters of the data stream objects in multiple different dimensions; calculate the number of reference index shards of the data stream objects in different dimensions based on the index description parameters of different dimensions; generate the target adjustment shard number of the data stream objects based on the current number of index shards of the data stream objects and the number of reference index shards of the data stream objects in different dimensions; adjust the index shards of the data stream objects based on the target adjustment shard number to obtain the adjusted index shards.

[0083] In addition, when the search engine needs to update data, the server 11 can obtain the time-series data to be updated through the terminal 10; determine the writing mode of the time-series data to be updated based on the business information of the data to be updated; and write the time-series data to be updated into the target write index of the data stream object based on the writing mode.

[0084] The following will provide a detailed description of each example. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0085] This application embodiment will be described from the perspective of a data management device, which can be integrated into a computer device, such as a server or a terminal.

[0086] like Figure 2The present invention provides a data management method, the specific process of which includes:

[0087] 101. Obtain management data containing data stream objects in the search engine. The data stream objects are associated with multiple indexes of time-series data in the search engine. The data stream objects include multiple index shards.

[0088] A search engine is a retrieval technology that uses specific strategies and algorithms to retrieve information from the internet and provide it to users based on their needs. Search engines allow for centralized data storage and enable operations such as searching, indexing, and analyzing that data.

[0089] For example, search engines can include Lucene and Elasticsearch, among others. Elasticsearch is also a distributed search engine architecture that can easily scale to hundreds of service nodes and support petabyte (PB) level data queries, enabling the system to have high availability and high concurrency.

[0090] Time-series data can include data recorded in chronological order. It is typically retained for a specific duration and reflects the development of a business over a given period. For example, industrial enterprises install sensors at various key points to collect data and monitor the operational status of equipment, production lines, and the entire system. This data is generated periodically or quasi-periodically, with varying collection frequencies. This collected data is generally sent to a server for aggregation and real-time processing, enabling real-time monitoring and early warning of system operation.

[0091] In one embodiment, time-series data may include the following characteristics: (1) the time is sequential and has a timestamp. (2) time-series data focuses on trends over a period of time. (3) time-series data generally has a retention period. For example, when a search engine stores time-series data, outdated time-series data may be deleted after one month, so the utilization rate and reference value of outdated time-series data are not high. (4) the data volume of time-series data is generally large.

[0092] In one embodiment, in order to improve the convenience of data management and the efficiency of retrieval, some related time-series data are usually grouped together and represented by the same index.

[0093] In one embodiment, a data stream object can be associated with multiple indexes of time-series data in a search engine. The data stream object can be an abstract collection storing multiple indexes of time-series data. Essentially, the data stream object provides an external interface for the indexes; when an index is invoked, it can be accessed through the data stream object without needing to concern itself with the details of the index invocation, thus simplifying the process of accessing the index.

[0094] For example, the data stream object could be a DataStream. A DataStream is a time-series data model that provides read and write requests for a single object, internally associated with a list of indices consisting of multiple indices.

[0095] In one embodiment, a data stream object may include a name, a time field, and a list of associated indexes, etc. The list of associated indexes includes indexes that are associated with the data stream object. The time field can be used to describe the time range of the time-series data associated with the index.

[0096] In one embodiment, because a search engine can store and manage large amounts of data, and must also support operations such as index querying and writing, it needs to be supported by one or more servers. The servers supporting the search engine's operation can be called a cluster. The servers in the cluster can be called nodes.

[0097] In one embodiment, the volume of time-series data is generally quite large. Storing all the time-series data on a single server often leads to excessive server load, reducing the server's efficiency and effectiveness in processing the time-series data. To address this issue, when the volume of index-related time-series data is too large, it is typically horizontally partitioned. Each partitioned portion of the time-series data can be called an index shard. After splitting the index-related time-series data into multiple index shards, these shards can be distributed and stored across multiple different nodes in the search engine cluster, thereby reducing the load on each node and improving its efficiency and performance.

[0098] For example, the time-series data associated with the index is 1000 gigabytes (GB). Storing all 1000GB of time-series data on a single server would overload that server, affecting the search engine's response speed. To solve this problem, the 1000GB of time-series data can be split into 10 index shards, with each shard containing 100GB of time-series data. These 10 index shards can then be stored on different nodes in the cluster, thus resolving the issue of overloading a single server.

[0099] In one embodiment, when the index associated with the data stream object can be sharded, the data stream object can be sharded with multiple indexes.

[0100] In one embodiment, the method proposed in this application can automatically and intelligently adjust index shards to obtain adjusted index shards. Specifically, this method may adjust the index shards in the most recently created index within a data stream object to obtain adjusted index shards.

[0101] For example, if a data stream object is associated with 10 indexes, and index 10 is the most recently created index, then the method proposed in this application embodiment can be used to adjust the index shards in index 10.

[0102] In one embodiment, management data may include data that describes the data in the data stream object. For example, the management data may be metadata. For example, the management data may exist in the form of a table, including multiple fields, each with a specific descriptive meaning. For example, some fields describe the relationships between indexes, some fields describe the attributes of the indexes, and so on.

[0103] In one embodiment, in the prior art, generating a data stream object requires manual programming code from the user, which makes the process complex and cumbersome. If the user is unfamiliar with programming code, a significant amount of time will be wasted on adjusting the code. To simplify the process of generating data stream objects, this application provides a search engine configuration interface. Users can configure various index-related parameters through this interface and then directly generate data stream objects.

[0104] Specifically, before the step "obtaining management data containing data stream objects from the search engine", the following may be included:

[0105] In response to a configuration-triggered action, the search engine configuration interface is displayed;

[0106] Obtain configuration parameters through the search engine configuration interface;

[0107] Generate a data stream object based on the configuration parameters;

[0108] The data stream object is encapsulated to obtain the management data.

[0109] The search engine configuration interface can be a visual interface provided to users, allowing them to manage the search engine. For example, users can create, delete, and modify data stream objects through this interface. Another example is managing parameters stored in the search engine through this interface.

[0110] In one embodiment, a search engine configuration interface can be displayed in response to a configuration triggering operation. The configuration triggering operation includes actions that cause the data management device to display the search engine configuration interface. For example, the configuration triggering operation may include a click operation and a swipe operation, etc.

[0111] For example, a data management device can provide an interface launch control that displays a search engine configuration interface when clicked by a user.

[0112] In one embodiment, after displaying the search engine configuration interface, the user can configure time-series data and its indexes through the interface. For example, the user can configure the period for deleting time-series data. Another example is that the user can configure the relationships between indexes. Yet another example is that the user can configure how to shard the indexes, and so on. After the user configures the indexes, the data management device can obtain the configuration parameters and generate a data stream object based on them.

[0113] Specifically, when generating data stream objects, the data management device can first generate an index lifecycle management strategy and an index model, and then generate the data stream objects based on the index lifecycle management strategy and the index model. Specifically, the step "Generate data stream objects based on configuration parameters" can include:

[0114] Create an index lifecycle management strategy based on the configuration parameters. The index lifecycle management strategy includes a strategy name.

[0115] Generate a mapping relationship between the index and business content based on the configuration parameters, and generate an index template based on the mapping relationship and the policy name of the index lifecycle management strategy;

[0116] A data stream object is generated based on the configuration parameters and the template name of the index template.

[0117] In one embodiment, because time-series data is constantly updated, without a strategy to manage it, the amount of data corresponding to the index of the time-series data will grow increasingly large. When the amount of indexed data reaches a certain level, the search engine may experience problems. For example, when the amount of indexed data becomes large, the search engine's retrieval speed will be slow, and data writing and updates will also be affected to varying degrees. Furthermore, in some business scenarios, users are more concerned with business data from the last 3 days or the last 7 days; a large index would aggregate all historical data, which is detrimental to data queries in such scenarios. Therefore, it is necessary to manage the time-series data and its index to prevent the amount of indexed data from growing indefinitely.

[0118] Among them, Index Lifecycle Management (ILM) is a strategy used to manage the lifecycle of time-series data and its indexes.

[0119] In one embodiment, an ILM policy may include a policy name and policy content. The policy content may describe how to manage time-series data and its indexes.

[0120] For example, an ILM strategy might define the lifecycle of an index as having four phases: hot, warm, cold, and delete. Each phase of the index has different functions. For instance, when an index is in the hot phase, it indicates that the time-series data associated with the index is being utilized frequently, and is being written to and queried often; therefore, the index can support writing and querying in this phase. Conversely, when an index is in the cold phase, it indicates that the time-series data associated with the index is being utilized very slowly; in this case, the index cannot support writing, but it can still support querying, albeit at a much slower speed. Finally, when an index is in the delete phase, it means that the index can be safely deleted.

[0121] In one embodiment, to facilitate index management, the index settings and index mappings of an existing index can be saved as an index template. When creating a new index, a name for the index template can be specified, allowing direct management of the index using the pre-defined settings and mappings. The index mapping can include the mapping relationship between index field names and data types. The index settings can include the number of shards and replicas, etc.

[0122] In one embodiment, such as Figure 3 As shown, when generating data flow objects based on configuration parameters, an ILM policy can be created according to the configuration parameters. The ILM policy can include a policy name and policy content. Then, a mapping relationship between indexes and business content can be generated based on the configuration parameters, and an index template can be generated based on the mapping relationship and the policy name of the index lifecycle management policy. For example, the index settings in the index template can be specified as the policy name of the newly created ILM policy, thereby establishing a connection between the index template and the ILM. Newly created indexes can directly reuse the ILM policy through the index template.

[0123] Then, data stream objects can be generated based on configuration parameters and the template name of the index template. For example, a list of associated indexes, a time field, and the name of the data stream object can be generated based on configuration parameters, and so on. The name of the data stream object can be associated with the template name of the index template, allowing the indexes associated with the data stream object to directly reuse the index settings and index mappings in the index template, thus improving the efficiency of index management.

[0124] In one embodiment, after generating the data stream object, the data stream object can be encapsulated to obtain management data. This management data can then be published to other nodes in the cluster, reducing the frequency of management data publication.

[0125] In addition, data stream objects, index templates, and ILM policies can be encapsulated together in management data, and then the management data can be published to other nodes in the cluster, reducing the number of times management data can be published.

[0126] 102. Query the management data to obtain the index description parameters of the data flow object in multiple different dimensions.

[0127] The index description parameter is used to describe the indexes of the data stream object. For example, this index description parameter can be used to describe the most recently created index in the data stream object.

[0128] In one embodiment, index description parameters across multiple dimensions may include index write speed, index storage capacity, and the number of indexed documents, among others.

[0129] Among them, index write speed can refer to the speed at which data is written to the index or index shards.

[0130] Among them, index storage capacity can refer to how much data an index or index shard can be associated with.

[0131] The number of indexed documents can refer to the amount of data written to the index or index shards.

[0132] In one embodiment, this application proposes a diagnostic system that can be used to detect the clusters of various search engines, thereby determining whether the number of index shards in the cluster is reasonable. When the diagnostic system detects that the number of index shards in the cluster is unreasonable, it can dynamically adjust the number of time-series index shards, thereby improving the reliability of the cluster.

[0133] For example, when the search engine is Elasticsearch, the diagnostic system can periodically query the various Elasticsearch clusters it manages to obtain all DataStream objects within the cluster. For each DataStream object, it can obtain index description parameters across multiple dimensions, such as index write speed, index storage capacity, and the number of indexed documents.

[0134] In one embodiment, the index description parameters obtained by querying are generally the parameters of the index of the data stream object over a period of time. These parameters are affected by the traffic of accessing the search engine and may be noisy. Therefore, the initial index description parameters can be queried first, and then the initial index description parameters can be denoised to obtain the index description parameters.

[0135] Specifically, the step "querying the management data to obtain the index description parameters of the data flow object across multiple different dimensions" can include:

[0136] Query the management data to obtain the initial index description parameters of the data flow object in multiple different dimensions;

[0137] Identify the parameter variation patterns of the initial index description parameters;

[0138] Based on the parameter variation pattern, the initial index description parameters are denoised to obtain the index description parameters of the data stream object in multiple different dimensions.

[0139] The parameter variation pattern can be used to describe the change pattern of index description parameters over a period of time. For example, this variation pattern can include sharp increases, steady decreases, steady increases, and peak changes, etc. A sharp increase pattern refers to a parameter exhibiting a rapid rise within a short period. A peak change pattern refers to a parameter undergoing drastic changes over a period of time, resulting in a peak. For example, if a search engine experiences a sudden surge in traffic at a certain point in time, the index description parameters may exhibit peak changes.

[0140] In one embodiment, the index description parameters on different dimensions have different characteristics. Therefore, different query methods can be used to obtain the initial index description parameters on different dimensions.

[0141] For example, index write speed is time-sensitive; generally, only the latest index write speed is meaningful for reference. Therefore, the write speed of index shards in a data stream object can be collected in real time to obtain the initial index write speed. For instance, the write speed of index shards in a data stream object over the past hour can be collected to obtain the initial index write speed.

[0142] For example, besides recent data being relevant, historical data is also meaningful for index storage capacity. Therefore, we can view index storage capacity over a longer period. Thus, the initial index shard capacity can include the storage capacity of the index shard over a recent period, as well as its storage capacity over a past period. For instance, the initial index shard capacity could include the storage capacity of the index shard from the past week to the present.

[0143] Similarly, in addition to recent data being relevant, historical data on the number of indexed documents is also very meaningful. Therefore, we can look at the number of indexed documents over a longer period.

[0144] In one embodiment, since the initial index description parameters may be affected by noise during the generation process, thus affecting the accuracy of the calculated index description parameters, it is necessary to identify the parameter change pattern of the initial index description parameters, and then perform noise reduction processing on the initial index description parameters according to the parameter change pattern to obtain the index description parameters of the data stream object in multiple different dimensions.

[0145] For example, if the initial index description parameter exhibits a peak-like variation pattern, box plotting can be used to denoise the parameter and identify anomalous peaks. Alternatively, if the initial index description parameter shows a steady decrease or increase, least squares can be used for denoising. Furthermore, if the initial index description parameter shows a sharp increase, a change-point detection algorithm can be used for denoising. Finally, if the initial index description parameter exhibits a variation pattern other than peak variation, steady increase, steady decrease, or sharp increase, the maximum value within the period of the initial index description parameter can be used as the index description parameter.

[0146] 103. Calculate the number of reference index fragments of the data stream object in different dimensions based on the index description parameters of different dimensions.

[0147] In one embodiment, after obtaining the index description parameters of different dimensions, the number of reference index fragments of the data stream object in different dimensions can be estimated based on the index description parameters of each dimension.

[0148] For example, index description parameters across multiple dimensions can include index write speed, index storage capacity, and the number of indexed documents, etc.

[0149] In one embodiment, the number of first reference index shards can be estimated based on the index write speed. Similarly, the number of second reference index shards can be estimated based on the index storage capacity. Likewise, the number of third reference index shards can be estimated based on the number of indexed documents. Then, the number of first, second, and third reference index shards constitutes the number of reference index shards for the data stream object in different dimensions.

[0150] In one embodiment, the index description parameters of an index are generally capped, which also results in an upper limit for the index description parameters of an index shard.

[0151] For example, suppose an index creates only one index shard. An index shard can be viewed as a container storing time-series data associated with the index. Two dimensions can be used to measure whether an index shard is large enough. One dimension is the number of indexed documents; generally, the upper limit for the total number of documents in an index shard is 2.1 billion. The other dimension is the index storage capacity. If the index storage capacity of an index shard is too large, the resources required for that index shard will be excessive. Therefore, it's generally best to avoid excessively large index storage capacity for index shards. Typically, the upper limit for index storage capacity for an index shard is 30GB-50GB. Regarding index write speed, if the index write speed is too fast, the amount of data received by the index shard per unit of time will be too large. To avoid the index shard receiving too much data per unit of time, the index write speed also has an upper limit.

[0152] In one embodiment, the number of reference index fragments of a data stream object in different dimensions can be calculated based on the upper limit of the index description parameters in different dimensions.

[0153] The estimated number of index shards can be obtained by dividing the upper limit of the index storage capacity by the index storage capacity of the queried index shards.

[0154] Similarly, the estimated number of index shards can be obtained by dividing the upper limit of the number of indexed documents by the number of indexed documents in the queried index shards.

[0155] In one embodiment, the index write speed of an index shard can be converted into index storage capacity. Specifically, the unit of index write speed is documents per second. Then, the index write speed can be multiplied by the number of days in a day to obtain the number of index documents written to the index shard in a day. Then, the number of index storage documents can be multiplied by the storage capacity required for each document to obtain the index storage capacity of the index shard.

[0156] Then, the index write speed of each index shard in the index can be converted into index storage capacity, thus obtaining the total index storage capacity. Next, it can be determined whether the total index storage capacity exceeds the upper limit. If it does not exceed the upper limit, the number of index shards can be estimated based on this total index storage capacity.

[0157] Because the index write speed of different businesses is difficult to evaluate, converting the index write speed into index storage capacity can simplify the process of evaluating the number of index shards based on index write parameters.

[0158] In one embodiment, the number of index shards can also be calculated directly from the index write speed. For example, assuming the absolute value of the write speed of a single shard is 'a' and the write speed of the entire index is 'b', the estimated number of index shards can be calculated.

[0159] 104. Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, generate the target adjustment number of shards for the data stream object.

[0160] In one embodiment, after estimating the number of reference index shards of the data stream object in different dimensions, the target adjustment shard number of the data stream object can be generated by combining the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions.

[0161] In one embodiment, the target number of reference index shards can be determined from reference index shard data in different dimensions. Then, the target number of reference index shards is compared with the current number of index shards of the data stream object, and a target number of adjusted shards is generated based on the comparison result. Specifically, the step "generating the target number of adjusted shards for the data stream object based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions" may include:

[0162] Determine the target reference index fragment number from the reference index fragment number in different dimensions;

[0163] The target reference index shard number is compared with the current index shard number of the data stream object to obtain the comparison result;

[0164] Based on the comparison results, the target number of shards for generating the data stream object is adjusted.

[0165] In one embodiment, the target reference index shard number can be determined from the reference index shard numbers across different dimensions. For example, the reference index shard number with the largest value can be determined as the target reference index shard number. Alternatively, the average of the index shard numbers across different dimensions can be calculated, and this average can be used as the target reference index shard number, and so on.

[0166] For example, the number of reference index shards in different dimensions includes the number of first reference index shards (shard1), the number of second reference index shards (shard2), and the number of third reference index shards (shard3). Then, shard1, shard2, and shard3 can be compared, and the maximum value can be taken as the target number of reference index shards (target_shard).

[0167] In one embodiment, the target reference index shard number can be compared with the current index shard number of the data stream object to obtain a comparison result. Then, based on the comparison result, a target adjusted shard number for the data stream object is generated. Finally, the index shards of the data stream object are adjusted based on the target adjusted shard number to obtain the adjusted index shards.

[0168] Adjusting the index shards of a data stream object based on a target shard count increases data overhead. For example, increasing the target shard count requires adding new shards and changing the relationships between indexes and index shards, as well as the relationships between time-series data and index shards. Conversely, decreasing the target shard count requires reducing the number of shards and changing the relationships between indexes and index shards, as well as the relationships between time-series data and index shards.

[0169] Secondly, when adjusting the index shards of a data stream object based on the target number of shards, it's necessary to consider how the degree of change in the number of index shards will affect cluster performance. For example, if the target number of shards is twice the current number of index shards, although the data overhead will be greater, it may significantly improve cluster performance. Conversely, if the target number of shards is 9 and the current number of index shards is 8, while the data overhead will be smaller and cluster performance can still be improved, the improvement will be less significant. Therefore, when generating the target number of shards for a data stream object, the impact of the target number of shards on cluster performance must be considered to ensure that the adjusted index shards can significantly improve search engine performance.

[0170] Therefore, when comparing the target reference index shard count with the current index shard count of the data stream object, the current index shard count of the data stream object can be scaled up or down before comparison.

[0171] Specifically, the step "comparing the target reference index shard number with the current index shard number of the data stream object to obtain the comparison result" can include:

[0172] The current number of index shards is amplified to obtain the amplified number of index shards.

[0173] Compare the number of target reference index fragments with the number of magnified index fragments.

[0174] For example, the current number of index shards can be multiplied by a factor to obtain the increased number of index shards. For example, the current number of index shards can be multiplied by 2 or 3, and so on.

[0175] Then, the target reference index fragment number and the magnified index fragment number are compared, and the target adjusted fragment number of the data stream object is generated based on the comparison result.

[0176] In one embodiment, when the target reference index fragment number is greater than the magnified index fragment number, the target reference index fragment number can be determined as the target adjustment fragment number.

[0177] For example, the number of index shards after scaling up is twice the current number of index shards. When the target reference index shards are greater than twice the current number of index shards, the target reference index shards can be set as the target adjustment shards.

[0178] In one embodiment, when the target reference index fragment number is less than or equal to the amplified index fragment number, the current index fragment number can be reduced to obtain a reduced index fragment number. Then, the target reference index fragment number and the reduced index fragment number are compared, and a target adjustment fragment number is generated based on the comparison result.

[0179] Specifically, the method in the embodiments of this application may further include:

[0180] When the target reference index fragment number is less than or equal to the amplified index fragment number, the current index fragment number is reduced to obtain the reduced index fragment number.

[0181] Compare the target reference index fragment number with the reduced index fragment number;

[0182] When the target reference index fragment number is less than the reduced index fragment number, the reduced index fragment number will be determined as the target adjustment fragment number.

[0183] For example, when the target reference index fragment number is less than or equal to the amplified index fragment number, the current index fragment number can be reduced by a certain proportion to obtain the reduced index fragment number. For example, the reduced index fragment number could be 1 / 2 of the current index fragment number. Or, for example, the reduced index fragment number could be 1 / 4 of the current index fragment number, and so on.

[0184] Then, the target number of reference index shards and the reduced number of index shards are compared. When the target number of reference index shards is less than the reduced number of index shards, the reduced number of index shards is determined as the target number of adjusted shards.

[0185] For example, when the target reference index shard number is less than 1 / 4 of the current index shard number, the target shard number can be adjusted to 1 / 4 of the current index shard number.

[0186] For example, when the target reference index shard number is less than 1 / 4 of the current index shard number, the target shard number can be adjusted to 1 / 2 of the current index shard number.

[0187] In one embodiment, when the target reference index shard number is greater than or reduced index shard number, it indicates that adjusting the current index shard number of the data stream object is not very meaningful, so the index shard number can be left unchanged.

[0188] 105. Adjust the index shards of the data stream object based on the target adjustment shard number to obtain the adjusted index shards.

[0189] In one embodiment, when it is necessary to adjust the current number of index shards of a data stream object, all shards of the data stream object can be adjusted based on the target number of shards to obtain the adjusted index shards.

[0190] For example, if the target number of shards is to increase, new shards need to be added, and the relationships between indexes and index shards, as well as the relationships between time-series data and index shards, need to be changed. Conversely, if the target number of shards is to decrease, shards need to be reduced, and the relationships between indexes and index shards, as well as the relationships between time-series data and index shards, need to be changed.

[0191] In one embodiment, the data management device can automatically adjust index shards using the method proposed in this application, achieving dynamic adjustment of index shards. By dynamically adjusting index shards, the load on the search engine cluster can be adjusted, thereby improving search engine performance. For example, when adjusting index shards, the current number of index shards for the data stream object can be increased by a factor in advance, thus preventing excessive changes in write speed that could cause the search engine to malfunction. Moreover, users do not need to be aware of the write availability issues caused by the index shard adjustment process, reducing maintenance costs.

[0192] In one embodiment, by associating an index with a data stream object, users can write data to a target write index directly through the data stream object. This eliminates the need for users to concern themselves with the underlying implementation details of the index; they can write data to the target write index simply by calling the data stream object, thus improving write efficiency.

[0193] In one embodiment, this application provides different writing modes for data from different services, thereby meeting the application scenarios of different services. Specifically, this application can write the time-series data to be updated into the target write index of the data stream object according to the writing mode of the data to be updated. Specifically, the method proposed in this application may further include:

[0194] When updating data, retrieve the time series data to be updated;

[0195] Based on the business information of the data to be updated, determine the writing mode of the time-series data to be updated;

[0196] Write the time-series data to be updated to the target write index of the data stream object based on the write mode.

[0197] The time-series data to be updated may include the time-series data that needs to be written into the index.

[0198] The write model for the time-series data to be updated specifies how the data should be written to the index. For example, this write mode can include time partitioning mode and append write mode.

[0199] When the writing mode of the time series data to be updated is time partition mode, it means that the target write index for writing the time series data to be updated is determined based on the time information of the time series data to be updated.

[0200] When the write mode of the time series data to be updated is append write mode, it means that the time series data to be updated will be written to the latest index.

[0201] The business information of the data to be updated describes the characteristics of the time-series data. This business information can refer to the business scenario to which the data is applicable. For example, business information could include log analysis and data detection scenarios. For instance, when the business information of the data to be updated pertains to a log analysis scenario, the corresponding write mode could be append-only. Similarly, when the business information of the data to be updated pertains to a data detection scenario, the corresponding write mode could be time-partitioned.

[0202] In one embodiment, when the write mode of the time series data to be updated is time partitioning mode, the step "writing the time series data to be updated to the target write index of the data stream object based on the write mode" may include:

[0203] The time series data to be updated is parsed to obtain the time information of the time series data to be updated;

[0204] Identify the time range information corresponding to each index in the data stream object;

[0205] The time information of the time series data to be updated is matched with the time range information corresponding to the index to obtain the matching result;

[0206] Based on the matching results, the target write index corresponding to the time series data to be updated is determined in the index of the data stream object.

[0207] In one embodiment, the time series data to be updated can be parsed to obtain its time information. For example, the time field in the time series data to be updated can be queried to obtain its time information.

[0208] In one embodiment, the time range information for each index in the data stream object can be identified. This time range information can have multiple layers of meaning. For example, the time range information can refer to the time range within which the data associated with the index was written to the index. Or, for another example, the time range information can describe the time range within which the data associated with the index was generated.

[0209] In one embodiment, the time information of the time series data to be updated can be matched with the time range information corresponding to the index to obtain a matching result. Then, based on the matching result, the target write index corresponding to the time series data to be updated is determined in the index of the data stream object.

[0210] For example, such as Figure 4 As shown, the indexes associated with the data stream object include indices 1 to 8. If the time information of the time series data to be updated matches the time range information of index 6, then all indexes 6 can be identified as targets for writing to the indexes, and the data to be updated can be written to index 6.

[0211] In one embodiment, when the time information of the time-series data to be updated does not match the time range information corresponding to the index in the data stream object, it can be determined whether the time-series data to be updated is expired. Specifically, if the time information of the time-series data to be updated is less than the minimum time node of the index in the data stream object, it indicates that the time-series data to be updated is expired. Conversely, if the time information of the time-series data to be updated is greater than the maximum time node of the index in the data stream object, it indicates that the time-series data to be updated is not expired.

[0212] If the time-series data to be updated is expired, the write request for the time-series data to be updated will be rejected. For example, ... Figure 6 As shown, if the time series data to be updated is expired, the writing of the time series data to be updated will be rejected.

[0213] When the time-series data to be updated is not expired, a new index is created, and the time-series data to be updated is written to the newly created index. For example, ... Figure 5As shown, when the time series data to be updated is not expired, a new index 8 is created, and the time series data to be updated is written into the newly created index 8.

[0214] In one embodiment, when the write mode of the time series data to be updated is append write mode, the step "writing the time series data to be updated to the target write index of the data stream object based on the write mode" may include:

[0215] Identify the creation identifier corresponding to each index in the data stream object;

[0216] Based on the creation identifier, the index to be written is identified in the index of the data stream object;

[0217] Identify the status information of the index to be written, and determine the target index to be written based on the status information of the index to be written;

[0218] Write the time-series data to be updated to the target index.

[0219] The index creation identifier can be used to indicate whether the index is newly created, because in append-only write mode, the time-series data to be updated is generally written to the most recently created index. For example, the index creation identifier can be a numerical value; the larger the index creation identifier, the newer the index. Another example is that the index creation identifier can be the index creation time, and so on.

[0220] The index to be written can be the most recently created index among the indexes associated with the data stream object.

[0221] In one embodiment, a creation identifier corresponding to each index in a data stream object can be identified, and then, based on the creation identifier, the index to be written can be identified in the index of the data stream object.

[0222] For example, if the index's creation identifier is a number, the index with the largest creation identifier can be identified as the index to be written to. As another example, when the index's creation identifier is its creation time, the index to be written to can be directly identified through its creation time.

[0223] In one embodiment, once an index is created, it generally cannot be used immediately. This is because the index shards corresponding to the index have not yet been started, so data cannot be written to the index at this time. Therefore, after identifying the index to be written, its status information can also be identified. The status information of the index to be written can be used to indicate whether the index supports data writing.

[0224] In one embodiment, after identifying the status information of the index to be written, the target write index can be determined based on the status information. The target write index can refer to the index where the updated time-series data is actually written. For example, when the status information of the index to be written meets the requirements, the index to be written can be determined as the target write index. Conversely, when the status information of the index to be written does not meet the requirements, in order to improve writing efficiency and save writing time, the index associated with the index to be written can be determined as the target write index.

[0225] Specifically, the step "determine the target write index based on the status information of the index to be written" may include:

[0226] Match the status information to be written to the index with the preset status conditions;

[0227] When the status information of the index to be written meets the preset status conditions, the index to be written is determined as the target index to be written.

[0228] When the status information of the index to be written does not meet the preset status conditions, identify the associated indexes related to the index to be written;

[0229] The associated index is identified as the target index for writing.

[0230] The preset status conditions can be used to determine whether the index to be written can be written. For example, if the status information of the index to be written meets the preset status conditions, it means that the index to be written can be written. If the status information of the index to be written does not meet the preset status conditions, it means that the index to be written cannot be written.

[0231] In one embodiment, when the status information of the index to be written meets the preset status conditions, the index to be written is determined as the target index to be written.

[0232] In one embodiment, when the status information of the index to be written does not meet the preset status conditions, the associated index with the index to be written is identified, and then the associated index is determined as the target index to be written.

[0233] Among them, the associated index with the index to be written can be the second newest index created in the data stream object.

[0234] For example, such as Figure 7 As shown, the indexes associated with the data stream object include indices 1 to 8. Among them, index 8 is the most recently created index, so index 8 can be identified as the index to be written.

[0235] By matching the status information of index 8 with the preset status conditions, it is determined that index 8 does not yet support write operations. Therefore, to save write time and improve write efficiency, index 7 can be designated as the target write index to avoid write blocking and affecting write throughput.

[0236] This application proposes a data management method, which includes: acquiring management data containing data stream objects in a search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards; querying the management data to obtain index description parameters of the data stream objects in multiple different dimensions; calculating the number of reference index shards of the data stream objects in different dimensions based on the index description parameters of different dimensions; generating a target adjustment shard number for the data stream objects based on the current number of index shards and the number of reference index shards of the data stream objects in different dimensions; and adjusting the index shards of the data stream objects based on the target adjustment shard number to obtain the adjusted index shards. The method proposed in this application can achieve dynamic adjustment of the number of index shards. When there is a problem with the number of index shards of a data stream object, the data management system can automatically adjust the index shards without requiring manual adjustment by the user, thereby reducing the user's management and maintenance costs for time-series data in the search engine. Furthermore, users do not need to be aware of the index write availability issues caused by changes in index shards, further reducing maintenance costs.

[0237] Secondly, the embodiments of this application can also achieve the following: when data is updated, obtain the time-series data to be updated; determine the writing mode of the time-series data to be updated based on the business information of the data to be updated; and write the time-series data to be updated into the target write index of the data stream object based on the writing mode. The embodiments of this application can provide multiple writing modes for the time-series data to be updated, thereby meeting various application scenarios.

[0238] In addition, this application embodiment also provides a search engine configuration interface for users, through which users can configure data stream objects without manually programming code, thereby improving the convenience and efficiency of generating data stream objects.

[0239] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0240] This application will use the integration of a data management method on a server as an example to illustrate the method of this application.

[0241] In one embodiment, such as Figure 8 As shown, a data management method has the following specific process:

[0242] 201. The server obtains management data containing data stream objects from the search engine. The data stream objects are associated with multiple indexes of time-series data in the search engine. The data stream objects include multiple index shards.

[0243] For example, a server can retrieve metadata from Elasticsearch containing a DataStream, which is associated with multiple indexes of time-series data in Elasticsearch. The data stream object includes multiple index shards from the most recently created index.

[0244] In one embodiment, a search engine configuration interface can be provided to the user, through which the user can configure various index-related parameters and then directly generate a data stream object.

[0245] The search engine configuration interface can be a visual interface provided to users, allowing them to manage the search engine. For example, users can create, delete, and modify data stream objects through this interface. Another example is managing parameters stored in the search engine through this interface.

[0246] When generating data stream objects, the server can first generate an index lifecycle management strategy and an index model, and then generate data stream objects based on the index lifecycle management strategy and the index model.

[0247] For example, such as Figure 3 As shown, when a DataStream is created, it first generates an ILM strategy, then generates an index template, and the index settings within the index template specify the name of the newly created ILM strategy. Finally, a DataStream object is created. Then, the DataStream object, ILM strategy, and index template are merged into a single Elasticsearch metadata file, and this Elasticsearch metadata is published to other nodes in the cluster, reducing the frequency of metadata publication. The DataStream's time field is variable, and the metadata stores information such as the DataStream's write mode and expiration time.

[0248] 202. The server queries the management data to obtain the index description parameters of the data stream object in multiple different dimensions.

[0249] For example, a server can periodically poll the various Elasticsearch clusters it manages to obtain all DataStream objects for the cluster. For each DataStream object, it retrieves index description parameters such as the latest created index write speed, index storage capacity, and number of index documents, and combines this information with the current DataStream's index shard settings to determine whether the index shard count needs to be adjusted.

[0250] For example, such as Figure 9 As shown, the server can check whether the DataStream sharding adjustment switch is enabled. If the DataStream sharding adjustment switch is enabled, it means that Elasticsearch allows the number of shards to be adjusted; if the DataStream sharding adjustment switch is not enabled, it means that Elasticsearch does not allow the number of shards to be adjusted.

[0251] The server will only continue with subsequent steps if it detects that the DataStream sharding adjustment switch is enabled; otherwise, it will terminate the current task.

[0252] Once the server detects that the DataStream sharding adjustment switch is enabled, the server can query the index write speed, index storage capacity, and number of indexed documents.

[0253] 203. The server calculates the number of reference index fragments for the data stream object in different dimensions based on the index description parameters of different dimensions.

[0254] For example, such as Figure 9 As shown, the server can estimate the number of shards for the first reference index based on the index write speed. The server can estimate the number of shards for the second reference index based on the index storage capacity. The server can estimate the number of shards for the third reference index based on the index storage capacity.

[0255] 204. The server generates the target adjustment number of the data stream object based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions.

[0256] For example, such as Figure 9 As shown, shard1, shard2, and shard3 can be compared, and the maximum value among them can be taken as the target reference index shard number (target_shard).

[0257] Then, the server can determine if `target_shard` is greater than twice the current number of index shards in the `DataStream` object. If it is, the current number of index shards is adjusted to `target_shard`. If `target_shard` is not greater than twice the current number of index shards, it checks if `target_shard` is less than one-quarter of the current number of index shards. If `target_shard` is less than one-quarter of the current number of index shards, the number of index shards in the `DataStream` object is adjusted to half the current number of shards, slowing down the reduction process and preventing increased write traffic that could affect write availability.

[0258] 205. The server adjusts the index shards of the data stream object based on the target shard count to obtain the adjusted index shards.

[0259] In one embodiment, this application provides different write models for writing data to be updated into an index in a data stream object.

[0260] For example, the write model may include time partitioning mode and append write mode.

[0261] Specifically, when the write mode of the data to be updated is time-partitioned write mode, the time information of the time-series data to be updated can be matched with the time range information corresponding to the index to obtain a matching result. Then, based on the matching result, the target write index corresponding to the time-series data to be updated is determined in the index of the data stream object.

[0262] For example, such as Figure 4 As shown, the indexes associated with the data stream object include indices 1 to 8. If the time information of the time series data to be updated matches the time range information of index 6, then all indexes 6 can be identified as targets for writing to the indexes, and the data to be updated can be written to index 6.

[0263] Specifically, if the time information of the time-series data to be updated does not match the time range information corresponding to the index, and the time information of the time-series data to be updated is greater than the maximum time node of the index in the data stream object, it indicates that the time-series data to be updated is not expired. Then, a new index can be created, and the time-series data to be updated can be written into the newly created index.

[0264] For example, such as Figure 6 As shown, when the time series data to be updated is not expired, a new index 8 is created, and the time series data to be updated is written into the newly created index 8.

[0265] Specifically, when the time information of the time series data to be updated does not match the time range information corresponding to the index, and the time series data to be updated is expired, the writing of the time series data to be updated can be rejected.

[0266] Specifically, when the write mode for the data to be updated is append-only, the time-series data to be updated can be written to the latest index in the associated index list of the data stream object. If the latest index is not yet active, the time-series data to be updated can be written to a second new index.

[0267] For example, such as Figure 7 As shown, the indexes associated with the data stream object include indices 1 to 8. Among them, index 8 is the most recently created index, so index 8 can be identified as the index to be written.

[0268] By matching the status information of index 8 with the preset status conditions, it is determined that index 8 does not yet support write operations. Therefore, to save write time and improve write efficiency, index 7 can be designated as the target write index to avoid write blocking and affecting write throughput.

[0269] This application proposes a data management method, which includes: a server acquiring management data containing data stream objects in a search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards; the server querying the management data to obtain index description parameters of the data stream objects in multiple different dimensions; the server calculating the number of reference index shards of the data stream objects in different dimensions based on the index description parameters of different dimensions; the server generating a target adjustment shard number for the data stream objects based on the current number of index shards of the data stream objects and the number of reference index shards of the data stream objects in different dimensions; and the server adjusting the index shards of the data stream objects based on the target adjustment shard number to obtain the adjusted index shards. The method proposed in this application can achieve dynamic adjustment of the number of index shards. When there is a problem with the number of index shards of a data stream object, the data management system can automatically adjust the index shards without requiring manual adjustment by the user, thereby reducing the user's management and maintenance costs for time-series data in the search engine. Furthermore, users do not need to be aware of the index write availability issues caused by changes in index shards, further reducing maintenance costs.

[0270] Secondly, the embodiments of this application can also achieve the following: when data is updated, the server obtains the time-series data to be updated; the server determines the writing mode of the time-series data to be updated based on the business information of the data to be updated; the server writes the time-series data to be updated into the target write index of the data stream object based on the writing mode. The embodiments of this application can provide multiple writing modes for the time-series data to be updated, thereby meeting various application scenarios.

[0271] In addition, this application embodiment also provides a search engine configuration interface for users, through which users can configure data stream objects without manually programming code, thereby improving the convenience and efficiency of generating data stream objects.

[0272] To better implement the data management method provided in the embodiments of this application, one embodiment also provides a data management device that can be integrated into a computer device. The meanings of the terms used are the same as in the data management method described above, and specific implementation details can be found in the description of the method embodiments.

[0273] In one embodiment, a data management device is provided, which can be specifically integrated into a computer device, such as... Figure 10 As shown, the data management device includes: an acquisition unit 301, a query unit 302, a calculation unit 303, a generation unit 304, and an adjustment unit 305, as detailed below:

[0274] The acquisition unit 301 is used to acquire management data containing data stream objects in the search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards.

[0275] The query unit 302 is used to query the management data to obtain the index description parameters of the data flow object in multiple different dimensions;

[0276] The calculation unit 303 is used to calculate the number of reference index fragments of the data stream object in different dimensions according to the index description parameters of different dimensions;

[0277] The generation unit 304 is used to generate the target adjustment number of the data stream object based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions.

[0278] The adjustment unit 305 is used to adjust the index shards of the data stream object based on the target number of adjustment shards to obtain the adjusted index shards.

[0279] In one embodiment, the generation unit 304 may include:

[0280] A subunit is defined to determine the target number of reference index fragments from the number of reference index fragments in the different dimensions;

[0281] The comparison subunit is used to compare the target reference index shard number with the current index shard number of the data stream object to obtain a comparison result.

[0282] A generation subunit is used to generate the target adjusted number of shards for the data stream object based on the comparison result.

[0283] In one embodiment, the comparison subunit may include:

[0284] An amplification module is used to amplify the current number of index fragments to obtain an amplified number of index fragments.

[0285] The comparison module is used to compare the target reference index fragment number with the magnified index fragment number;

[0286] In one embodiment, the generating subunit may include:

[0287] The determination module is used to determine the target reference index fragment number as the target adjustment fragment number when the target reference index fragment number is greater than the magnified index fragment number.

[0288] In one embodiment, the generation unit 304 may further include:

[0289] The shrinking subunit is used to shrink the current number of index fragments when the target reference index fragment number is less than or equal to the amplified index fragment number, so as to obtain the shrunken index fragment number.

[0290] The narrowing comparison subunit is used to compare the target reference index fragment number with the narrowed index fragment number;

[0291] A subunit is defined to determine the number of reduced index fragments as the target adjusted fragment number when the target reference index fragment number is less than the reduced index fragment number.

[0292] In one embodiment, the data management device may further include:

[0293] The data acquisition unit is used to acquire the time-series data to be updated when data updates are performed.

[0294] The mode determination unit is used to determine the writing mode of the time-series data to be updated based on the service information of the data to be updated.

[0295] The writing unit is used to write the timing data to be updated into the target write index of the data stream object based on the writing mode.

[0296] In one embodiment, the writing unit may include:

[0297] The parsing subunit is used to parse the time series data to be updated to obtain the time information of the time series data to be updated;

[0298] The first identification subunit is used to identify the time range information corresponding to each index in the data stream object;

[0299] The matching subunit is used to match the time information of the time series data to be updated with the time range information corresponding to the index to obtain the matching result;

[0300] The first index determination subunit is used to determine the target write index corresponding to the time-series data to be updated in the index of the data stream object based on the matching result.

[0301] The first write subunit is used to write the timing data to be updated into the target write index.

[0302] In one embodiment, the writing unit may include:

[0303] The second identification subunit is used to identify the creation identifier corresponding to each index in the data stream object;

[0304] The third identification subunit is used to identify the index to be written in the index of the data stream object based on the creation identifier;

[0305] The second index determination subunit is used to identify the status information of the index to be written, and determine the target write index based on the status information of the index to be written;

[0306] The second write subunit is used to write the timing data to be updated into the target write index.

[0307] In one embodiment, the second index determining subunit may include:

[0308] The matching module is used to match the status information to be written to the index with preset status conditions;

[0309] An index determination module is used to determine the index to be written as the target write index when the status information of the index to be written meets the preset status conditions.

[0310] An index identification module is used to identify associated indexes related to the index to be written when the status information of the index to be written does not meet the preset status conditions.

[0311] A write module is used to determine the associated index as the target write index.

[0312] In one embodiment, the query unit 302 may include:

[0313] The query subunit is used to query the management data to obtain the initial index description parameters of the data flow object in multiple different dimensions;

[0314] The pattern recognition subunit is used to identify the pattern of parameter changes of the initial index description parameters;

[0315] The denoising subunit is used to denoise the initial index description parameters according to the parameter variation pattern, so as to obtain the index description parameters of the data stream object in multiple different dimensions.

[0316] In one embodiment, the data management device may further include:

[0317] The display unit is used to display the search engine configuration interface in response to configuration trigger operations;

[0318] The parameter acquisition unit is used to acquire configuration parameters through the search engine configuration interface;

[0319] An object generation unit is used to generate a data stream object according to the configuration parameters;

[0320] The encapsulation unit is used to encapsulate the data stream object to obtain the management data.

[0321] In one embodiment, the object generation unit may include:

[0322] The strategy creation subunit is used to create an index lifecycle management strategy based on the configuration parameters, wherein the index lifecycle management strategy includes a strategy name;

[0323] The template generation subunit is used to generate a mapping relationship between the index and the business content according to the configuration parameters, and to generate an index template according to the mapping relationship and the policy name of the index lifecycle management strategy.

[0324] The object generation subunit is used to generate the data stream object based on the configuration parameters and the template name of the index template.

[0325] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0326] The data management device described above can improve the utilization of memory resources, thereby improving the efficiency of data management.

[0327] This application also provides a computer device, which may include a terminal or a server. For example, the computer device may serve as a data management terminal, such as a mobile phone, tablet computer, etc.; or it may serve as a server, such as a data management server. Figure 11As shown, it illustrates the structural diagram of the terminal involved in the embodiments of this application, specifically:

[0328] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 11 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0329] The processor 401 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0330] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0331] The computer device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0332] The computer device may also include an input unit 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0333] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0334] The management data of the search engine containing data stream objects is obtained, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards;

[0335] The management data is queried to obtain the index description parameters of the data stream object in multiple different dimensions;

[0336] Based on the index description parameters of different dimensions, calculate the number of reference index fragments of the data stream object in different dimensions;

[0337] Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, generate the target adjustment number of shards for the data stream object.

[0338] The index shards of the data stream object are adjusted based on the target number of shards to obtain the adjusted index shards.

[0339] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0340] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

[0341] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0342] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the data management methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0343] The management data of the search engine containing data stream objects is obtained, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards;

[0344] The management data is queried to obtain the index description parameters of the data stream object in multiple different dimensions;

[0345] Based on the index description parameters of different dimensions, calculate the number of reference index fragments of the data stream object in different dimensions;

[0346] Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, generate the target adjustment number of shards for the data stream object.

[0347] The index shards of the data stream object are adjusted based on the target number of shards to obtain the adjusted index shards.

[0348] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0349] Since the computer program stored in the storage medium can execute the steps of any of the data management methods provided in the embodiments of this application, the beneficial effects that any of the data management methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0350] The data management method, apparatus, computer device, and storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data management method characterized by, include: The system retrieves management data containing data stream objects from a search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards. The management data is queried to obtain the index description parameters of the data stream object in multiple different dimensions; Based on the index description parameters of different dimensions, calculate the number of reference index fragments of the data stream object in different dimensions; Based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions, generate the target adjustment number of shards for the data stream object. The index shards of the data stream object are adjusted based on the target number of shards to obtain the adjusted index shards; When updating data, retrieve the time series data to be updated; Based on the business information of the time series data to be updated, the write mode of the time series data to be updated is determined, and the write mode includes an append write mode; Writing the time-series data to be updated to the target write index of the data stream object based on the write mode includes: identifying the creation identifier corresponding to each index in the data stream object; identifying the index to be written in the index of the data stream object based on the creation identifier; identifying the status information of the index to be written; determining the target write index based on the status information of the index to be written; and writing the time-series data to be updated to the target write index.

2. The method of claim 1, wherein, The step of generating the target adjusted number of shards for the data stream object based on the current number of index shards and the number of reference index shards for the data stream object in different dimensions includes: Determine the target number of reference index fragments from the reference index fragment numbers across the different dimensions; The target reference index shard number is compared with the current index shard number of the data stream object to obtain the comparison result; Based on the comparison results, the target number of adjusted shards for the data stream object is generated.

3. The method of claim 2, wherein, The step of comparing the target reference index shard number with the current index shard number of the data stream object to obtain the comparison result includes: The current number of index fragments is amplified to obtain the amplified number of index fragments; Compare the number of target reference index fragments with the number of magnified index fragments; The step of generating the target adjusted number of shards for the data stream object based on the comparison result includes: When the target reference index fragment number is greater than the magnified index fragment number, the target reference index fragment number is determined as the target adjustment fragment number.

4. The method of claim 3, wherein, The method further includes: When the target reference index fragment number is less than or equal to the amplified index fragment number, the current index fragment number is reduced to obtain the reduced index fragment number. Compare the target reference index fragment number with the reduced index fragment number; When the target reference index fragment number is less than the reduced index fragment number, the reduced index fragment number is determined as the target adjustment fragment number.

5. The method of claim 1, wherein, The write mode includes a time partitioning mode, and the step of writing the time-series data to be updated to the target write index of the data stream object based on the write mode includes: The time series data to be updated is parsed to obtain the time information of the time series data to be updated; Identify the time range information corresponding to each index in the data stream object; The time information of the time series data to be updated is matched with the time range information corresponding to the index to obtain the matching result; Based on the matching results, the target write index corresponding to the time series data to be updated is determined in the index of the data stream object; The time series data to be updated is written into the target write index.

6. The method of claim 1, wherein, Determining the target write index based on the status information of the index to be written includes: The status information to be written to the index is matched with the preset status conditions; When the status information of the index to be written meets the preset status conditions, the index to be written is determined as the target index to be written. When the status information of the index to be written does not meet the preset status conditions, the associated indexes related to the index to be written are identified; The associated index is determined as the target write index.

7. The method according to claim 1, characterized in that, The querying of the management data to obtain the index description parameters of the data stream object across multiple dimensions includes: The management data is queried to obtain the initial index description parameters of the data stream object in multiple different dimensions; Identify the parameter variation patterns of the initial index description parameters; Based on the parameter variation pattern, the initial index description parameters are denoised to obtain the index description parameters of the data stream object in multiple different dimensions.

8. The method according to claim 1, characterized in that, Before obtaining the management data of the data stream object contained in the search engine, the process includes: In response to a configuration-triggered action, the search engine configuration interface is displayed; Obtain configuration parameters through the search engine configuration interface; Generate a data stream object based on the configuration parameters; The data stream object is encapsulated to obtain the management data.

9. The method according to claim 8, characterized in that, The step of generating a data stream object based on the configuration parameters includes: Create an index lifecycle management strategy based on the configuration parameters, wherein the index lifecycle management strategy includes a strategy name; The mapping relationship between the index and the business content is generated based on the configuration parameters, and an index template is generated based on the mapping relationship and the policy name of the index lifecycle management strategy. The data stream object is generated based on the configuration parameters and the template name of the index template.

10. A data management device, characterized in that, include: The acquisition unit is used to acquire management data containing data stream objects in the search engine, wherein the data stream objects are associated with multiple indexes of time-series data in the search engine, and the data stream objects include multiple index shards; The query unit is used to query the management data to obtain the index description parameters of the data stream object in multiple different dimensions; The calculation unit is used to calculate the number of reference index fragments of the data stream object in different dimensions based on the index description parameters of different dimensions. The generation unit is used to generate the target adjustment number of the data stream object based on the current number of index shards of the data stream object and the number of reference index shards of the data stream object in different dimensions; An adjustment unit is used to adjust the index shards of the data stream object based on the target number of adjustment shards, so as to obtain the adjusted index shards; The data acquisition unit is used to acquire the time-series data to be updated when data updates are performed. The mode determination unit is used to determine the writing mode of the time series data to be updated based on the service information of the time series data to be updated, wherein the writing mode includes an append writing mode. A writing unit is used to write the timing data to be updated into the target write index of the data stream object based on the writing mode. The writing unit includes: The second identification subunit is used to identify the creation identifier corresponding to each index in the data stream object; The third identification subunit is used to identify the index to be written in the index of the data stream object based on the creation identifier; The second index determination subunit is used to identify the status information of the index to be written, and determine the target write index based on the status information of the index to be written; The second write subunit is used to write the timing data to be updated into the target write index.

11. The data management device according to claim 10, characterized in that, The generation unit includes: A subunit is defined to determine the target number of reference index fragments from the number of reference index fragments in the different dimensions; The comparison subunit is used to compare the target reference index shard number with the current index shard number of the data stream object to obtain a comparison result. A generation subunit is used to generate the target adjusted number of shards for the data stream object based on the comparison result.

12. The data management device according to claim 11, characterized in that, The comparison subunit includes: An amplification module is used to amplify the current number of index fragments to obtain an amplified number of index fragments. The comparison module is used to compare the target reference index fragment number with the magnified index fragment number; The generating subunit includes: The determination module is used to determine the target reference index fragment number as the target adjustment fragment number when the target reference index fragment number is greater than the magnified index fragment number.

13. The data management device according to claim 12, characterized in that, The generation unit further includes: The shrinking subunit is used to shrink the current number of index fragments when the target reference index fragment number is less than or equal to the amplified index fragment number, so as to obtain the shrunken index fragment number. The narrowing comparison subunit is used to compare the target reference index fragment number with the narrowed index fragment number; A subunit is defined to determine the number of reduced index fragments as the target adjusted fragment number when the target reference index fragment number is less than the reduced index fragment number.

14. The data management device according to claim 10, characterized in that, The write mode includes a time partition mode, and the write unit includes: The parsing subunit is used to parse the time series data to be updated to obtain the time information of the time series data to be updated; The first identification subunit is used to identify the time range information corresponding to each index in the data stream object; The matching subunit is used to match the time information of the time series data to be updated with the time range information corresponding to the index to obtain the matching result; The first index determination subunit is used to determine the target write index corresponding to the time-series data to be updated in the index of the data stream object based on the matching result. The first write subunit is used to write the timing data to be updated into the target write index.

15. The data management device according to claim 10, characterized in that, The second index determining subunit includes: The matching module is used to match the status information to be written to the index with preset status conditions; An index determination module is used to determine the index to be written as the target write index when the status information of the index to be written meets the preset status conditions. An index identification module is used to identify associated indexes related to the index to be written when the status information of the index to be written does not meet the preset status conditions. A write module is used to determine the associated index as the target write index.

16. The data management device according to claim 10, characterized in that, The query unit includes: The query subunit is used to query the management data to obtain the initial index description parameters of the data stream object in multiple different dimensions; The pattern recognition subunit is used to identify the pattern of parameter change of the initial index description parameters; The denoising subunit is used to denoise the initial index description parameters according to the parameter variation pattern, so as to obtain the index description parameters of the data stream object in multiple different dimensions.

17. The data management device according to claim 10, characterized in that, The data management device further includes: The display unit is used to display the search engine configuration interface in response to configuration trigger operations; The parameter acquisition unit is used to acquire configuration parameters through the search engine configuration interface; An object generation unit is used to generate a data stream object according to the configuration parameters; The encapsulation unit is used to encapsulate the data stream object to obtain the management data.

18. The data management device according to claim 17, characterized in that, The object generation unit includes: The strategy creation subunit is used to create an index lifecycle management strategy based on the configuration parameters, wherein the index lifecycle management strategy includes a strategy name; The template generation subunit is used to generate a mapping relationship between the index and the business content according to the configuration parameters, and to generate an index template according to the mapping relationship and the policy name of the index lifecycle management strategy. The object generation subunit is used to generate the data stream object based on the configuration parameters and the template name of the index template.

19. A computer device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the data management method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the data management method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the data management method according to any one of claims 1 to 9.

Images