Data management method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a data management method, which comprises the following steps: determining original data and a first index corresponding to the original data; filtering the original data to obtain sampling data, and determining a second index corresponding to the sampling data; in response to the first index and the original data being deleted, determining a third index associated with the second index based on the first index, the third index matching the first index, so that the sampling data corresponding to the third index is output when data is queried based on the first index. The application also discloses a data management device, an electronic device and a computer readable storage medium. In the foregoing manner, the application fills the data vacancy of the deleted original data with the sampling data, which is simple, convenient and fast, and can expand the time range of the retained monitoring data without affecting the normal query and use of the data.

Description

Technical Field

[0001] This application relates to the field of data storage, and in particular to a data management method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the advent of the big data era, monitoring relevant nodes and analyzing the monitoring data is an important means of judging and solving problems. However, monitoring data is generally collected in real time and continuously, resulting in a massive amount of data. Therefore, if no relevant operations are performed on the data stored for a period of time, the data will continue to be deleted to reserve storage space for new monitoring data.

[0003] However, if some monitoring data is deleted, it becomes unviewable later, hindering data retrieval. Therefore, how to store and manage monitoring data has become a hot research topic. Summary of the Invention

[0004] The main objective of this application is to provide a data management method, apparatus, electronic device, and computer-readable storage medium that can solve the technical problem of expanding the time range of retained monitoring data without affecting the normal query and use of the data.

[0005] This application provides a data management method, which includes: determining original data and a first index corresponding to the original data; filtering the original data to obtain sampled data, and determining a second index corresponding to the sampled data; in response to the deletion of the first index and the original data, determining a third index associated with the second index based on the first index, wherein the third index matches the first index, so as to output the sampled data corresponding to the third index when performing a data query based on the first index.

[0006] This application provides a data management device, which includes: an acquisition unit for determining original data and a first index corresponding to the original data; and a data processing unit for filtering the original data to obtain sampled data and determining a second index corresponding to the sampled data. In response to the deletion of the first index and the original data, the unit determines a third index associated with the second index based on the first index, wherein the third index matches the first index, so as to output the sampled data corresponding to the third index when performing a data query based on the first index.

[0007] This application provides an electronic device including a processor and a computer storage medium coupled to the processor, wherein a computer program is stored in the computer storage medium, and the processor is used to execute the computer program to implement the method described in the first technical solution.

[0008] This application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the data management method described above.

[0009] The beneficial effects of this application are as follows: Unlike existing technologies, this method first determines the original data, then filters and samples it to obtain corresponding sampled data. The original data corresponds to a first index, and the sampled data corresponds to a second index. After deleting the corresponding original data and the first index, a related third index is added as an alias to the second index. The third index matches the first index, allowing both the first and third indexes to be retrieved based on the same query operation, thus obtaining the corresponding data. When deleting the original data, only the sampled data needs to be aliased to replace the original data during data querying, filling the data gaps left by the deleted original data with sampled data. The operation is simple, convenient, and fast, without consuming excessive computing resources. While expanding the time range of the retained monitoring data, it does not affect the normal query and use of the data. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a data management solution;

[0012] Figure 2 This is a schematic diagram of another data management solution;

[0013] Figure 3 This is a schematic diagram of another data management method technical solution provided in this application;

[0014] Figure 4 This is a flowchart illustrating the first embodiment of the data management method of this application;

[0015] Figure 5 This is a flowchart illustrating the second embodiment of the data management method of this application;

[0016] Figure 6 This is a flowchart illustrating the third embodiment of the data management method of this application;

[0017] Figure 7 This is a schematic diagram of the structure of an embodiment of the data management device of this application;

[0018] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] Before introducing this application, a brief introduction to the relevant technologies will be given.

[0023] Elasticsearch (ES) is a search server built on Lucene. It's a distributed, highly scalable, and real-time search and data analytics engine. It can handle hundreds of service nodes and supports petabytes of structured or unstructured data. It easily enables the searching, analysis, and exploration of large amounts of data. Leveraging Elasticsearch's horizontal scalability makes data more valuable in production environments. ES identifies data through indexes. `_reindex` is an ES command used to copy documents from one index to another, often used for data migration. `_aliases` is also an ES command used to set aliases for an index; ES automatically converts the aliases to the index names during searches to retrieve the corresponding data.

[0024] Currently, considering the limited storage space, in order to maximize the monitoring time range of monitoring data, that is, to store monitoring data for a longer period of time, the following two technical solutions are usually adopted: one is to sample the monitoring data when it is stored and retain the sampled data separately; the other is to create an index for the stored monitoring data according to the period, that is, to process the deleted monitoring data, retain the sampled part and delete the non-sampled part to reduce the amount of stored data.

[0025] like Figure 1 As shown, in the first technical solution, the monitoring data needs to be analyzed during acquisition to determine whether it needs to be sampled and retained. This requires changes to the system architecture of the storage mechanism, increasing the complexity of the technical architecture. When data is written, the original data corresponds to the original data index, while the sampled data is stored separately according to the sampled index, thus distinguishing it from the original data. Deleting the original data will not affect the sampled data. However, after data is written, if the original data is not deleted, parts of the original data and the sampled data are the same, which increases the storage burden on the storage device and wastes storage space.

[0026] like Figure 2 As shown, in the second technical solution, an index is created for the original data when writing data to the storage device. Then, the index is judged based on time to determine the original data that needs to be deleted. After determining the original data to be deleted, the original data is sampled, and unnecessary data is deleted. The retained sampled data is then matched with the index of the original data. This solution operates directly on the original index during deletion and consumes a large amount of CPU (central processing unit) and I / O (input / output) resources. This may cause the storage device to experience lag, query failures, slow query speeds, etc., when monitoring data deletion, thus affecting the user experience.

[0027] To address the aforementioned problems, this application proposes a data management method, such as... Figure 3As shown, in this technical solution, after the data is written to the storage device, an index corresponding to the original data is used. Then, in step 1, the original data is sampled to obtain sampled data, which is then stored using the reindex command. In step 2, after deleting the original data that exceeds the retention period, aliases are added to the corresponding sampled data so that when the original data is queried through the index, the sampled data corresponding to the previously deleted original data can also appear. By sampling before the original data is deleted and adding aliases to the sampled data after the original data is deleted using the aliases command, duplicate sampled data will not occupy storage space for a long time. Moreover, adding aliases can be easily implemented using the alias command, which expands the scope of monitoring data query without affecting the normal query and use of data. Specific implementation can be found in the following embodiments.

[0028] Reference Figure 4 , Figure 4 This is a flowchart illustrating the first embodiment of the data management method of this application. The data management method proposed in this application can be executed by an electronic device, which can be a terminal device, such as a mobile phone, computer, or smart interactive device; or, the electronic device can be a server, such as a standalone physical server or a cloud server performing cloud computing. The electronic device can be deployed in a storage device. Figure 4 The data management method shown includes the following steps:

[0029] S11: Determine the original data and the first index corresponding to the original data.

[0030] Raw data can be any piece of raw data stored in a storage device. Raw data refers to the most authentic monitoring data that has not undergone sampling processing. To distinguish the raw data stored in the storage device, an index is generated for each piece of raw data. As the name suggests, the index has a guiding and indicative function. When the index of a piece of raw data is known, the unique raw data can be found based on the index. Similarly, when a piece of raw data is known, the unique index corresponding to that piece of raw data can be found.

[0031] Assume the index corresponding to the original data is set as the first index. The first index can be determined by the target class index tag and the corresponding original data storage time. The target class index can also be an index name prefix. For example, assuming the target class index tag is 'index' and the original data storage time is March 2, 2022, then the first index corresponding to this original data can be represented as index_20220302.

[0032] S12: Filter the original data to obtain sampled data and determine the second index corresponding to the sampled data.

[0033] Once the original data is determined, if it is data that needs to be sampled, it is filtered to obtain sampled data. A corresponding second index is determined for the sampled data. The second index differs from the first index and further includes a specific identifier. The specific identifier is used to indicate the sampled data so that the data can be distinguished by the index when searching for data. For example, if the specific identifier is "sample", and the original data stored on March 2, 2022 is sampled, retaining the data for the first hour, combined with the data time, the second index can be represented as "sample1h_index_20220303".

[0034] S13: In response to the deletion of the first index and the original data, determine the third index associated with the second index based on the first index.

[0035] In general, electronic devices set a data retention period for monitoring data. Data exceeding this period is deleted to free up storage space. When original data is deleted, its corresponding first index is also deleted. To broaden the monitoring time range, original data whose storage time is about to exceed the data retention period is sampled first, so that the sampled data can be used to replace the original data after deletion. After deleting original data with excessively long retention periods, to ensure that data information for the corresponding time period can still be retrieved during regular queries, an alias is added to the second index of the sampled data corresponding to the original data. The `_aliases` command can be used here. This command associates the third index with the second index, so that querying the third index will jump to the sampled data corresponding to the second index.

[0036] The third index includes a target class index marker and a specific identifier indicating the sampled data. The third index matches the first index, ensuring that the desired data can be retrieved using the same query pattern. There are many ways to match the third index with the first index; for example, both can use the same target class index marker (index) as a prefix. For instance, the first index corresponding to the original data might be index_20220303, and the index of the sampled data might be sample1h_index_20220303. After deleting the original data, an alias is added to the second index, matching the first index, so the third index could be index_20220303_sample1h. Thus, when querying using "index" as the prefix, both the first and third indexes, along with their corresponding data, will be retrieved. The target class index marker can also be placed elsewhere, as long as the third index can also be retrieved when the first index is queried according to the corresponding rules.

[0037] In this embodiment, the original data is first determined, and then filtered and sampled to obtain corresponding sampled data. The original data corresponds to a first index, and the sampled data corresponds to a second index. After the corresponding original data and the first index are deleted, an associated third index is added to the second index as an alias. The third index matches the first index so that the first index and the third index can be obtained based on the same query operation, thereby retrieving the corresponding data. When deleting the original data, only the sampled data needs to be aliased to replace the original data during data query, filling the data gaps of the deleted original data with sampled data. The operation is simple, convenient, and fast. While expanding the monitoring time range of the monitoring data in the storage device, it avoids the problem of consuming a lot of device performance and affecting the normal use of data query functions.

[0038] Reference Figure 5 , Figure 5 This is a flowchart illustrating a second embodiment of the data management method of this application. The data management method can be executed by an electronic device, which can be a terminal device such as a mobile phone, computer, or smart interactive device; alternatively, the electronic device can be a server, such as a standalone physical server or a cloud server performing cloud computing. The electronic device can be deployed in a storage system. It includes the following steps:

[0039] S21: Determine the original data and the first index corresponding to the original data.

[0040] Raw data can be any piece of raw data stored in a storage device. Raw data is the most authentic monitoring data before sampling processing. To distinguish data, each piece of raw data corresponds to a unique data index. The first index can be determined by the target class index tag and the corresponding raw data storage time. The target class index can also be a prefix of the index name. For example, assuming the target class index tag is 'index' and the raw data storage time is March 2, 2022, then the first index corresponding to this raw data can be represented as index_20220302.

[0041] S22: Determine whether the original data meets the conditions for deletion.

[0042] Once the original data is determined, in order to still be able to query relevant data for the corresponding time period after the original data is deleted, it is necessary to sample it before deletion. That is, the original data will only be sampled if it meets the deletion criteria. In this way, instead of sampling all original data indiscriminately, sampling resources can be saved.

[0043] Based on this, after determining the original data in step S21, it is first determined whether the original data meets the deletion criteria. The deletion criteria include any one or more of the following: the storage time of the original data exceeds a preset first time length, or a deletion command for the original data is received.

[0044] The preset first time length is a set length of time used to indicate the storage duration of raw data in the storage device. This preset first time length is shorter than the preset raw data retention time. For example, if the preset raw data retention time is ten days (the time calculated from when the raw data is stored in the storage device), then the preset first time length is set to nine days. When the storage duration of raw data exceeds the preset first time length, it is considered raw data that is about to be deleted. During the interval between the preset first time length and the raw data retention time, the raw data to be deleted can be sampled. The interval is the processing time reserved for the sampling process.

[0045] If a deletion command for the original data is received, the deletion can be temporarily suspended to allow time for the sampling process. The original data can be deleted after the sampling is completed.

[0046] By setting deletion conditions for the original data and reserving processing time for the sampling process, the data sampling process is set just before the original data is deleted. This avoids the sampling data occupying storage space for a long time and also avoids the situation where the sampling is not completed when the data is deleted. Compared with conventional technical solutions that extend the monitoring time range of monitoring data, this solves the technical problem that the sampled data occupies too much storage space as duplicate data when the original data is not deleted, thus reducing the storage pressure on the storage device.

[0047] S23: Determine whether the original data is data that needs to be sampled.

[0048] Once the deletion criteria are met, it is further determined whether the original data needs to be sampled. If the original data does not need to be sampled or processed, it can be deleted directly when the storage time of the original data exceeds the preset original data retention time.

[0049] The order of judgment in steps S22 and S23 is only illustrative and is not limited in this embodiment.

[0050] S24: If the original data is full of data that needs to be sampled, then filter the original data to obtain sampled data and determine the second index corresponding to the sampled data.

[0051] After determining the original data to be sampled, it is filtered to obtain sampled data, and a corresponding second index is assigned to it. The second index includes a target class index tag and a specific identifier. The specific identifier is used to indicate the sampled data.

[0052] Regardless of whether the original data needs to be sampled, the electronic device will regularly perform data deletion to ensure that there is some free storage space available for storing new monitoring data.

[0053] S25: Obtain multiple indices including the target class index marker, and iterate through the obtained multiple indices in this way.

[0054] The first index includes target class index tags, while the second and third indexes include target class-specific tags and specific identifiers. By querying the tags or identifiers, the corresponding indexes can be retrieved. The first, second, and third indexes can be found using the target class index. To locate the original data to be deleted, these indexes need to be distinguished.

[0055] S26: Determine whether the index includes a specific identifier.

[0056] The system then checks if any of the indexes containing the target class index tag also contain an index with a specific identifier. These indexes correspond to sampled data and cannot be used for subsequent deletion operations.

[0057] If the currently traversed index includes a specific identifier, then the currently traversed index is determined to be the second index. Before any data in the storage device has been used to fill the gaps in the deleted original data with sampled data, if the index includes a specific identifier, it is determined to be the second index. The second index corresponds to the processed sampled data and cannot be deleted.

[0058] Furthermore, when the original data has completed the operation of filling the deleted original data with data, if the judgment of this step is yes, these indices are the second index and the third index corresponding to other sampled data. These indices all correspond to the processed sampled data and cannot be deleted.

[0059] If the current traversal index does not include a specific identifier, then the current traversal index is determined to be the first index. The first index corresponds to the original data, and the original data that meets the conditions will be deleted later.

[0060] S27: Determine whether the storage time of the original data exceeds the preset second time length.

[0061] The system determines whether the storage time of the original data exceeds a preset second time. The preset second time is used to determine whether to delete the original data. The preset second time can be a pre-defined time length from when the original data is determined to be deleted to when it is actually deleted. The preset second time is longer than the preset first time. For example, if the preset original data retention time is ten days, then the second preset time is set to ten days. The preset first time is shorter than the preset second time to allow time for data sampling. When the storage time of the original data exceeds the preset second time, the data is deleted.

[0062] S28: Delete the original data.

[0063] The original data is deleted, and its first index is also deleted. After deletion, the third index is added to replace the original data with the sampled data.

[0064] In another embodiment, if a deletion command for the original data is received, the original data is determined to need to be sampled. After the original data is sampled, it can be deleted directly without determining whether the storage time of the original data exceeds a preset second time.

[0065] S29: Determine the third index associated with the second index based on the first index.

[0066] After deleting some of the sampled original data, a third index, acting as an alias, is added to the sampled data corresponding to the second index to ensure that relevant data for the same time period can be retrieved during subsequent data queries. The third index matches the first index, allowing it to be retrieved using the same query pattern. There are many matching methods; for example, both can use the same target index class marker (index) as a prefix. For instance, the first index corresponding to the original data might be index_20220303, and the index of the sampled data might be sample1h_index_20220303. After deleting the original data, an alias is added to the second index, matching the first index, so the third index could be index_20220303_sample1h. This way, when querying using "index" as the starting point, both the first and third indices, along with their corresponding data, will appear. The target index class marker can also be placed elsewhere, as long as the third index also appears when the first index is queried according to the corresponding rules.

[0067] By adding aliases, when querying the original data corresponding to the first index, it's also possible to retrieve the sampled data of the deleted original data corresponding to the second index. Because the third index, acting as an alias, is associated with the second index, querying the third index will redirect to the data in the second index. Adding aliases expands the monitoring time range of the data being monitored during queries without modifying the existing indexing model. Furthermore, when deleting original data, only the sampled data is renamed, while the original data is directly deleted. The operation is simple, convenient, and fast, without consuming significant computing resources or affecting normal data querying and usage.

[0068] Reference Figure 6 , Figure 6 This is a flowchart illustrating the third embodiment of the data management method of this application. This embodiment further extends the sampling process of the original data. The data management method can be executed by an electronic device, which can be a terminal device, such as a mobile phone, computer, or smart interactive device; or, the electronic device can be a server, such as a standalone physical server or a cloud server performing cloud computing. The electronic device can be deployed in a storage system. It includes the following steps:

[0069] S31: Determine the execution conditions of the sampling algorithm for filtering the original data, and obtain the sampling quantity based on the execution conditions.

[0070] Downsampling algorithms can be used to filter raw data. For example, for a day's monitoring data, samples can be taken from 24 hourly intervals. Further limitations can be made on the sampling time and data size. For instance, when sampling from 24 hourly intervals, if the sampling time is set to one minute, and assuming a data volume of three million records per minute, then the total sampled data for the day would be seventy-two million records.

[0071] S32: Set the sampling rate based on the sampling volume and system operation status.

[0072] After determining the size of the sampled data, the sampling process is performed based on the storage device's operational status. This operational status can refer to the storage system's performance during the sampling of monitoring data, or other usage scenarios affecting the storage device. Operational status can include the size and rate of data input and output. The sampling rate includes the query and write rates of the sampled data. If the sampled data size is 72 million records, and the sampling rate is set to 1,000 queries and writes per second, it will take approximately 20 hours to complete the writing of all sampled data and corresponding indexes. If the second preset time and the first preset time of the original data differ by one day, it is sufficient to complete the sampling of the original data the previous day. The sampled data can then be replaced after deleting the original data the next day. The 1,000 queries and writes per second have a negligible impact on the overall storage system and ensure the normal operation of the data query function.

[0073] Furthermore, different write and query rates can be set based on the time periods during which the storage device is used. For example, the storage device is used less between midnight and 3 AM, leaving more performance available, while usage is higher between 9 AM and 12 PM, with most of its performance being utilized. In this case, a higher sampling write rate can be set between midnight and 3 AM, and a lower sampling write rate can be set between 9 AM and 12 PM. This ensures normal data storage while maximizing the use of device performance and completing data sampling as quickly as possible.

[0074] By determining the size of the sampled data and taking into account the performance and actual operation of the storage device, the sampling rate of the sampled data can be reasonably arranged to ensure the normal data storage and use of the storage device while completing the data sampling process as quickly as possible.

[0075] S33: The original data is filtered based on the sampling rate to obtain the sampled data.

[0076] The raw data to be deleted is processed by sampling at a set sampling rate. The sampling rate should typically be set such that data sampling can be completed within the time interval between a preset first time length and a preset second time length. This prevents the storage device from being occupied by unnecessary sampled data most of the time, and also ensures that sampling is completed before the data to be deleted is ready.

[0077] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an embodiment of the data management device of this application.

[0078] The data management device includes an acquisition unit 110 and a data processing unit 120.

[0079] The acquisition unit 110 is used to query data and can determine the original data and the first index corresponding to the original data.

[0080] The data processing unit 120 is used to perform a sampling step, filtering the original data to obtain sampled data, and determining a second index for the sampled data. In response to the deletion of the first index and the corresponding original data, a third index associated with the second index can be determined based on the first index. The third index matches the first index, so that when performing a data query based on the first index, the sampled data corresponding to the third index can be output.

[0081] In one embodiment, the data processing unit 120 is further configured to determine whether the original data meets the deletion conditions when filtering the original data to obtain sampled data. The deletion conditions include one or more of the following: the storage time of the original data exceeds a preset first time length or a deletion command for the original data is received. It is also configured to determine whether the original data is data that needs to be sampled. If the original data meets the deletion conditions and needs to be sampled, the original data is filtered to obtain sampled data.

[0082] In one embodiment, the data processing unit 120 is further configured to delete the first index and the original data corresponding to the first index before determining the third index associated with the second index based on the first index. This deletion of the first index and the original data corresponding to the first index includes obtaining multiple indexes including a target class index marker, and sequentially traversing the obtained multiple indexes; if the currently traversed index includes a specific identifier, then the currently traversed index is determined to be the second index, and no deletion operation is performed; if the currently traversed index does not include the specific identifier, then the currently traversed index is determined to be the first index, and after the storage time of the original data corresponding to the first index exceeds a preset second time length, the first index and the original data corresponding to the first index are deleted.

[0083] In one embodiment, the data processing unit 120 is further configured to, during the process of filtering the original data to obtain sampled data, determine the execution conditions of the sampling algorithm for filtering the original data, and obtain the sampling quantity based on the execution conditions; set the sampling rate based on the sampling quantity and system operation status; and filter the original data based on the sampling rate to obtain sampled data.

[0084] In the above embodiments, the first index includes a target class index marker, and both the second and third indices include a target class index marker and a specific identifier. The specific identifier is used to indicate the sampled data. The preset second time length is greater than the preset first time length. In the data management device embodiments, the same or corresponding specific technical features and method steps used can be referred to the description of the above embodiments, and will not be repeated here. The data processing unit 120 can implement any embodiment of the above data management method and possible combinations thereof.

[0085] The data management device of this application can implement the above-described data management method. It first determines the original data, then filters and samples the original data to obtain corresponding sampled data. The original data corresponds to a first index, and the sampled data corresponds to a second index. After deleting the corresponding original data and the first index, an associated third index is added as an alias to the second index. The third index matches the first index, so that the first and third indexes can be obtained based on the same query operation, thereby retrieving the corresponding data. When deleting original data, only the sampled data needs to be aliased to replace the original data during data querying, filling the data gaps left by the deleted original data with sampled data. The operation is simple, convenient, and fast. While expanding the monitoring time range of the monitoring data, it avoids the problem of consuming a large amount of computing power and affecting the normal use of data query functions.

[0086] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application.

[0087] The electronic device includes a processor 210 and a computer storage medium 220.

[0088] Processor 210 controls the operation of electronic devices. Processor 210 can also be called a CPU (Central Processing Unit). Processor 210 may be an integrated circuit chip with signal sequence processing capabilities. Processor 210 can also be a general-purpose processor, a digital signal sequence processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0089] The computer storage medium 220 stores the instructions and computer programs required for the operation of the processor 210.

[0090] The processor 210 is used to execute instructions to implement the methods provided by any of the above embodiments and possible combinations of the data management methods described in this application.

[0091] This application provides a computer-readable storage medium storing a computer program that, when executed, implements the data management method provided in any of the above embodiments and possible combinations thereof.

[0092] The storage device may include a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or it may be a server that stores the program instructions. The server can send the stored program instructions to other devices for execution, or it can execute the stored program instructions itself.

[0093] In summary, this application first determines the original data, then filters and samples it to obtain corresponding sampled data. The original data corresponds to a first index, and the sampled data corresponds to a second index. After the corresponding original data and the first index are deleted, an associated third index is added to the second index as an alias. The third index matches the first index, so that the first and third indexes can be obtained based on the same query operation, thereby retrieving the corresponding data. When deleting the original data, only the sampled data needs to be aliased to replace the original data during data query, filling the data gaps left by the deleted original data with the sampled data. The operation is simple, convenient, and fast. While expanding the monitoring time range of the monitoring data, it avoids the problem of consuming a large amount of computing power and affecting the normal use of data query functions. Furthermore, this application sets the data sampling process just before the original data is deleted, avoiding the long-term occupation of storage space by the sampled data. Compared with conventional technical solutions that expand the monitoring time range of the monitoring data, it solves the problem of duplicate storage of the same data and reduces the storage pressure on the storage device.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated units in the other embodiments described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data management method characterized by, The method comprises: determining original data and a first index corresponding to the original data; filtering the original data to obtain sampling data and determining a second index corresponding to the sampling data; in response to the first index and the original data being deleted, determining a third index associated with the second index based on the first index, the third index matching the first index, so that when data is queried based on the first index, the sampling data corresponding to the third index is output; the first index and the second index both comprise a target class index mark, and the second index further comprises a specific identifier; further comprising: sequentially traversing a plurality of indexes comprising the target class index mark obtained; if the currently traversed index comprises the specific identifier, it is determined that the currently traversed index is the second index, and no deletion operation is performed; if the currently traversed index does not comprise the specific identifier, it is determined that the currently traversed index is the first index, and the first index and the original data corresponding to the first index are deleted.

2. The method of claim 1, wherein, The filtering of the original data to obtain sampling data comprises: determining whether the original data meets a deletion condition; the deletion condition comprises any one or more of the following conditions: the storage time length of the original data exceeds a preset first time length; or a deletion command for the original data is received; if the original data meets the deletion condition, the filtering of the original data to obtain the sampling data is performed.

3. The method of claim 2, wherein, The method further comprises: determining whether the original data is data that needs to be sampled; if yes, filtering the original data to obtain the sampling data; if no, deleting the original data and the first index corresponding to the original data.

4. The method of claim 2, wherein, The third index comprises the target class index mark and a specific identifier; wherein the data corresponding to the index comprising the specific identifier is sampling data.

5. The method of claim 4, wherein, Further comprising: obtaining a plurality of indexes comprising the target class index mark.

6. The method of claim 5, wherein, Before the deletion of the first index and the original data corresponding to the first index, further comprising: determining whether the storage time length of the original data exceeds a preset second time length, the preset second time length being greater than the preset first time length; if yes, deleting the original data and the corresponding first index.

7. The method of claim 1, wherein, The filtering of the original data to obtain sampling data comprises: determining an execution condition of a sampling algorithm for filtering the original data, and obtaining a sampling amount according to the execution condition; setting a sampling rate based on the sampling amount and system running conditions; filtering the original data based on the sampling rate to obtain the sampling data.

8. A data management apparatus characterized by comprising: The data management device comprises: an acquisition unit configured to determine original data and a first index corresponding to the original data; The data processing unit is configured to filter the original data to obtain sample data, determine a second index corresponding to the sample data, and determine a third index associated with the second index based on the first index in response to the first index and the original data being deleted, the third index matching the first index, so that the sample data corresponding to the third index is output when data is queried based on the first index. The first index and the second index each include a target class index mark, and the second index further includes a specific identifier. The data processing unit is further configured to sequentially traverse the obtained multiple indexes including the target class index mark. If the currently traversed index includes the specific identifier, the currently traversed index is determined as the second index, and the deletion operation is not performed. If the currently traversed index does not include the specific identifier, the currently traversed index is determined as the first index, and the first index and the original data corresponding to the first index are deleted.

9. An electronic device, comprising: The electronic device includes a processor and a computer storage medium coupled to the processor, the computer storage medium stores a computer program, and the processor is configured to execute the computer program to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

Images