Screening method and system for log data units to be optimized

Abstract

The invention discloses a method and a system for screening a to-be-optimized log data unit. The method comprises the following steps: acquiring a storage overhead index and a use heat index of the to-be-evaluated log data unit within a preset evaluation time range; an optimization priority score of the log data unit to be evaluated is determined, the optimization priority score is in direct proportion to a storage overhead index and is in inverse proportion to a use heat index, and the higher the optimization priority score is, the higher the priority of the log data unit to be evaluated into the log data unit to be optimized is; and under the condition that the optimization priority score of the to-be-evaluated log data unit meets a preset optimization condition, determining that the to-be-evaluated log data unit is a to-be-optimized log data unit needing to be subjected to storage optimization operation. Therefore, the log data units with high storage overhead and low utilization rate are accurately screened, and the method is suitable for changes of business requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log storage optimization, and in particular to a screening method and system for log data units to be optimized. BACKGROUND

[0002] In a log management and analysis platform, in order to support operation and maintenance monitoring, fault troubleshooting and business analysis, it is usually necessary to collect and store massive structured or semi-structured log data. These log data contain numerous fields, such as timestamp, error code, user identifier, time consumption, CPU usage, memory occupation, etc. With the passage of time, log data accumulates continuously, occupying huge storage resources, especially under some regulatory or business requirements, logs need to be saved for months or even years.

[0003] However, not all fields have equal query and analysis value throughout the storage period. For example, time consumption, CPU usage and other indicator fields are usually frequently used for performance analysis and anomaly detection in the near future (such as within a few days) after the log is stored; but after a few weeks or months, the probability of these detailed indicators being queried again is extremely low, and more often, only the basic information fields of the log (such as error type, occurrence time) are concerned. Common storage optimization methods rely on field deletion based on fixed time rules, for example, some fields are uniformly set to have a retention period through index lifecycle management strategies. This static rule method may mistakenly delete fields that still have query requirements.

[0004] Therefore, how to provide a screening method for log data units to be optimized to improve the accuracy of log storage optimization is an important research topic for those skilled in the art. SUMMARY

[0005] In a first aspect, the present application provides a screening method for log data units to be optimized, the method comprising: obtaining a storage overhead indicator and a usage heat indicator of a log data unit to be evaluated within a preset evaluation time range; determining an optimization priority score of the log data unit to be evaluated based on the storage overhead indicator and the usage heat indicator, the optimization priority score being directly proportional to the storage overhead indicator and inversely proportional to the usage heat indicator, the higher the optimization priority score, the higher the priority of the log data unit to be evaluated as a log data unit to be optimized; and in the case that the optimization priority score of the log data unit to be evaluated meets a preset optimization condition, determining that the log data unit to be evaluated is a log data unit to be optimized that needs to be subjected to a storage optimization operation.

[0006] The method for selecting log data units to be optimized provided in this application determines whether a log data unit is marked as requiring optimization based on the latest "storage cost-use value" status. By accurately selecting log data units with high storage costs and low utilization rates based on the cost-effectiveness of storage overhead and use value, this method adapts to changes in business needs, avoids blind deletion based on time, and improves the accuracy of log storage optimization. Furthermore, by clearly defining the semantics of optimization priority scores, the system can prioritize processing the log data units with the highest scores when resources are limited, maximizing optimization benefits.

[0007] In some possible implementations, the storage overhead metric is related to disk space usage, and the usage popularity metric includes at least two of the following: query frequency within a first preset statistical period, indicating the number of times it is used as a query condition or returned result within the preset statistical period; aggregation operation frequency within a second preset statistical period, indicating the number of times it is used for aggregation operations; and query recentity, indicating the freshness of the query.

[0008] This approach incorporates at least two dimensions of popularity metrics, including query frequency, aggregation operation frequency, and query recentity. This multi-metric fusion strategy, compared to using only a single metric (such as total access count), more accurately reflects the actual value of log data units in the business and avoids biased evaluation.

[0009] Furthermore, since aggregation operations typically represent log data units used in deep-value scenarios such as report generation and business analysis—for example, field units may rarely be queried individually, but they are core dimensions in monthly analytical reports—separating and listing the frequency of ordinary queries from the frequency of aggregation operations, and separately tracking the frequency of aggregation operations, can effectively identify and protect key types of log data units with high analytical value but low retrieval frequency, preventing unintended optimizations.

[0010] In some possible implementations, the optimization priority score is calculated using the following formula: Where field represents the log data unit being evaluated, and T represents the preset evaluation time range. This indicates the optimization priority score of the evaluated log data unit. This represents the storage cost of the evaluated log data unit within a preset evaluation time range T. The heat score represents the popularity score of the evaluated log data unit within a preset evaluation time range T, where W_s and W_h are weighting coefficients, and ε is a given real number used to prevent division by zero errors; the heat score The result is calculated by weighting at least two of the following sub-indicators: normalized query frequency, normalized aggregation operation frequency, and time decay factor based on the most recent access time.

[0011] In this way, the weight coefficients W_s and W_h are introduced, which provide key policy adjustment parameters for system administrators. By adjusting these two weights, a flexible balance can be achieved between the tendency to save storage costs and the tendency to ensure data availability, so that the same system can adapt to different business departments or different management goals at different times. In addition, the hotness score is obtained by weighting multiple sub-indicators, including at least query frequency, aggregation frequency, and time decay factor, which can improve the comprehensiveness and representativeness of the optimization priority score.

[0012] In some possible implementation manners, the hotness score is calculated by the following formula: wherein F_query is the normalized query frequency, F_recency is the time decay factor, and F_aggregation is the normalized aggregation operation frequency, 、 、 is a weight coefficient.

[0013] In this way, the weight coefficients 、 、 are another set of key policy adjustment parameters after W_s and W_h. For example, for a business that emphasizes real-time monitoring, the weight of F_recency can be adjusted to be higher; for a scenario that emphasizes historical analysis, the weight of F_aggregation can be adjusted to be higher. This can enable the same system to accurately match different definitions of data value by different business departments through configuration.

[0014] In some possible implementation manners, after determining that the to-be-evaluated log data unit is a to-be-optimized log data unit that needs to be subjected to a storage optimization operation, the method further includes: matching at least one storage optimization operation for the to-be-optimized log data unit based on a data type and a business rule, the storage optimization operation including: deleting storage data of the to-be-optimized log data unit within a preset storage time range, modifying an index mapping to close an index function of the to-be-optimized log data unit, and migrating storage data of the to-be-optimized log data unit within the preset storage time range to a preset lower-cost storage medium.

[0015] In this way, the storage optimization operation is based on the data type (for example, numerical type, identification type, and analysis type) and the business attribute (for example, whether it is a core business field) of the log data unit, and the unified to-be-optimized state is disassembled into specific and safe operation paths such as deletion, index mapping degradation, and migration.

[0016] In one aspect, it can be ensured that the automation operation does not violate the business continuity and compliance bottom line while pursuing efficiency, avoiding data islands and trace fault caused by optimization from the data integrity level, and improving the practicality and acceptability of the scheme.

[0017] In another aspect, different optimization operations are selected based on the data type, so that the most economic balance between storage cost and potential value of data can be achieved under the given business rules, rather than blindly pursuing the minimization of storage space. For example, ordinary text / numeric fields with strong independence and irrelevant value are directly deleted to maximize cost savings; for analysis critical fields, migration instead of deletion can be used to retain their potential value for future historical analysis or audit at a lower long-term storage cost; and index degradation is performed on identification type fields to sacrifice a small amount of query performance in exchange for a significant reduction in storage space and index maintenance cost.

[0018] In some possible implementation ways, after determining that the to-be-evaluated log data unit is a to-be-optimized log data unit that needs to be implemented with a storage optimization operation, the method further includes: in a case where the optimization priority score of the to-be-evaluated log data unit does not meet the preset optimization condition, continuing to evaluate the optimization priority score of the to-be-evaluated log data unit in a next preset evaluation time range, and determining whether the to-be-evaluated log data unit needs to be implemented with a storage optimization operation based on a change; for the to-be-optimized log data unit that has been implemented with a corresponding storage optimization operation and the storage optimization operation is not a deletion operation, regarding the to-be-optimized log data unit as an optimized log data unit, and continuing to count the optimization priority score of the optimized log data unit; if it is determined, based on the current optimization priority score of the optimized log data unit, that the optimized log data unit does not need to be implemented with a storage optimization operation any more, stopping the corresponding storage optimization operation about the optimized log data unit, resuming the corresponding function, and re-labeling the optimized log data unit as a to-be-evaluated log data unit.

[0019] In this way, the data unit that does not reach the optimization threshold is tracked and re-evaluated continuously, forming a dynamic optimization cycle to adapt to business changes. In addition, the current cold data may become hot in the future due to new query requirements. The heat recovery is sensed in real time, and after the optimization priority score does not meet the optimization condition, the corresponding storage optimization operation is automatically stopped, the corresponding function is resumed, the accuracy of the optimization operation is further improved, and the reliability of the scheme is improved.

[0020] In some possible implementation ways, the storage duration of the to-be-evaluated log data unit exceeds a preset storage duration.

[0021] This approach focuses on older log data units that have passed their peak business window and are most likely to result in storage waste, thus avoiding resource consumption caused by ineffective calculations on fresh and active data.

[0022] In some possible implementations, the log data unit to be evaluated is a log field.

[0023] This approach allows for independent retention or cleanup decisions for different fields within the same log record, achieving the smallest granularity for storage resource optimization. Different fields vary significantly in size and value; optimizing certain fields that are "volume-intensive but rapidly depreciating in value" directly corresponds to the greatest potential for cost savings.

[0024] Secondly, this application also provides a filtering device for log data units to be optimized, including a unit for performing any of the filtering methods for log data units to be optimized in the first aspect.

[0025] Thirdly, this application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded by a processor and executed by any of the filtering methods for log data units to be optimized in the first aspect.

[0026] Fourthly, embodiments of this application also provide a computer program product containing instructions, which, when run on an electronic device, causes the electronic device to execute any of the filtering methods for log data units to be optimized in the first aspect.

[0027] Fifthly, embodiments of this application also provide a chip module, including a transceiver component and a chip, wherein the chip is used to execute any of the filtering methods for log data units to be optimized in the first aspect.

[0028] It is understood that the filtering device for the log data units to be optimized, the computer storage medium, the computer program, the computer program product, and the chip system provided above are all used to execute the method shown in any implementation of the first aspect of the embodiments of this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a method for filtering log data units to be optimized, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of a filtering system for log data units to be optimized, provided in an embodiment of this application; Figure 3 This is a schematic diagram of a filtering device for log data units to be optimized, provided in an embodiment of this application. DETAILED DESCRIPTION

[0030] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described below with reference to the drawings.

[0031] It should be noted that the electronic device is taken as an example to illustrate the execution subject of the method for screening the log data unit to be optimized provided by the present application, and the electronic device can also be understood as the screening device of the log data unit to be optimized shown in the present application. In the present application, the electronic device can be a microprocessor or a computer for executing program code, and the electronic device for executing the method provided by the present application belongs to the protection scope of the present application, and the present application does not make any limitation. For example, the electronic device can be a desktop computer, a portable notebook, a mobile terminal, a 32-bit microprocessor or a 64-bit microprocessor, and the present application does not make any limitation.

[0032] Please refer to Figure 1 , Figure 1 A flow chart of the method for screening the log data unit to be optimized provided by the present application is shown in FIG. 1. As shown in FIG. 1, the method for screening the log data unit to be optimized includes the following steps: Figure 1 S101, the electronic device acquires the storage overhead index and the usage heat index of the log data unit to be evaluated within a preset evaluation time range.

[0033] For example, the preset evaluation time range can be 90 days, or half a year, or other required time range, which is not limited herein.

[0034] In some possible implementation manners, the above-mentioned log data unit to be evaluated can refer to one log field in a log index, for example, can be a timestamp, an error code, a user identifier, a time consumption, a CPU usage, or a memory occupation, etc. For example, in an app_error_log_2025 index, each log has a timestamp, an error code, a message, a stack trace, etc. The above-mentioned log data unit to be evaluated can be any one field in the app_error_log_2025 index.

[0035] In some possible implementation manners, the above-mentioned log data unit to be evaluated can refer to a complete log data record, for example, can refer to a single log file or a single log item, which is based on specific requirements.

[0036] ​In some possible implementation manners, the storage duration of the to-be-evaluated log data unit exceeds a preset storage duration, which can be 60 days, 90 days, or any other suitable duration, and the value evaluation is performed only on the part of data with a storage duration exceeding the preset storage duration to determine whether optimization is needed.

[0037] In some possible implementation manners, the storage overhead indicator is related to the size of the occupied disk space, and can be the proportion of the storage space occupied by the to-be-evaluated log data unit in the total index storage space, or the absolute size of the occupied storage space, in at least one storage structure, including at least one of the inverted index, the forward storage, and the columnar storage.

[0038] The usage heat indicator includes at least two of the following: the query frequency in a first preset statistical period, the aggregation operation frequency in a second preset statistical period, and the query recency.

[0039] The query frequency indicator indicates the number of times (which can be the total number or the average number) that the to-be-evaluated log data unit is used as a query condition or returned result in the first preset statistical period. For example, the first preset statistical period can be the last 7 days, the last 30 days, or the last 60 days, or any other suitable statistical period.

[0040] The aggregation operation frequency indicator indicates the number of times (which can be the total number or the average number) that the to-be-evaluated log data unit is used for aggregation operation in the second preset statistical period. The aggregation operation can include SUM, AVG, MAX / MIN, COUNT, GROUP BY, PERCENTILE, or any other aggregation operation. In the embodiments of the present application, the second preset statistical period can be the same as, partially the same as, or completely different from the first preset statistical period, and the present application does not limit the same.

[0041] It should be noted that, in the embodiments of the present application, the preset statistical period (including the first preset statistical period and the second preset statistical period) is included in a preset evaluation time range, which is used to determine the to-be-evaluated log data unit from the storage time dimension, for example, the log data unit with a storage time exceeding 90 days. The statistical period of the query frequency or the aggregation operation frequency is an independent time window, which is used to count the access request to the above-mentioned data subset occurring in the window, for example, the last 30 days. By combining the two, the system can accurately evaluate the access heat of the data of a specific ‘age’ in the recent period.

[0042] The query recency indicator is used to represent the freshness of the log data unit to be evaluated. For a log data unit to be evaluated, the closer the access time to the current time, the more current the hot value. Conversely, the access time long ago will decay the current value.

[0043] As an example, the query recency indicator is the time point of the last query of the log data unit to be evaluated (which can also be understood as the last access timestamp).

[0044] As another example, the query recency indicator can also be a time decay factor calculated based on the time difference between the last query timestamp of the log data unit to be evaluated and the current timestamp. The time decay factor decreases as the time difference parameter increases. Specifically, the time decay factor can be determined based on the time difference parameter using Newton's cooling law, a linear decreasing function, or an exponential decreasing function, which is not limited herein.

[0045] It can be understood that the combination of query frequency, aggregation operation frequency, and query recency indicator can more accurately distinguish between continuously active log data units and log data units that are only popular at a certain historical time period, thereby providing a key time dimension basis for intelligent optimization decisions.

[0046] S102, the electronic device determines the optimization priority score of the log data unit to be evaluated based on the storage overhead indicator and the usage heat indicator.

[0047] In the embodiments of the present application, the optimization priority score is proportional to the storage overhead and inversely proportional to the usage heat of the log data unit to be evaluated. The higher the optimization priority score, the higher the priority of the log data unit to be evaluated as the log data unit to be optimized. The optimization priority score is used to quantify the mismatch between the storage overhead and the usage heat of the log data unit to be evaluated within the preset evaluation time range. The higher the storage overhead and the lower the usage heat, the higher the priority (urgency) of the storage optimization operation.

[0048] In some possible implementations, the optimization priority score of the log data unit to be evaluated is calculated by the following formula 1: Wherein, field represents the log data unit to be evaluated, T represents the preset evaluation time range, represents the optimization priority score of the log data unit to be evaluated field , The higher the value, the higher the storage overhead, the lower the usage value, the more significant the resource mismatch, and the lower the overall benefit. Therefore, the priority of optimization (such as cleaning) is also higher.

[0049] represents the storage cost of the evaluated log data unit within the preset evaluation time range T; represents the hotness score of the evaluated log data unit within the preset evaluation time range T; W_s and W_h are global weight coefficients of the storage cost and the hotness score, used to adjust whether the system is biased towards "saving storage" or "ensuring query", for example, W_s and W_h can be 0.5 respectively, or other suitable values based on specific needs; ε is a given real number, used to prevent division by zero errors, for example, ε is a very small positive number, such as 0.001, or other suitable values; wherein the hotness score is calculated by weighting based on at least two sub-indicators: query frequency, representing the number of times being used as query conditions or returned results; time decay factor, representing the recency of being used as query conditions or returned results; aggregation operation frequency, representing the number of times being used for aggregation operations.

[0050] As an example, the hotness score is calculated based on the following formula 2.

[0051] wherein F_query is the normalized query frequency (i.e., the value after normalization of the query frequency within the first preset statistical period described in step S101 above), F_recency is the decay factor based on the most recent access time (which can refer to the description of the time decay factor in S101 above, and will not be described in detail here), and F_aggregation is the normalized aggregation operation frequency (i.e., the value after normalization of the aggregation operation frequency within the second preset statistical period described in step S101 above), 、 、 are weight coefficients, for example, can be 0.3, 0.35, 0.35 respectively, or other suitable values.

[0052] For example, the normalization calculation method for the query frequency sub-indicator of the log data unit to be evaluated can be calculated using the following formula 3 (the normalization calculation method for the aggregation frequency sub-indicator is similar).

[0053] wherein 、 are the maximum and minimum values of the query frequency sub-indicator of all log data units to be evaluated within the preset evaluation time range, a normalized query frequency of the i-th to-be-evaluated log data unit. a normalized query frequency of the i-th to-be-evaluated log data unit.

[0054] It can be understood that, on one hand, the magnitudes of the original query frequency indicators (or the magnitudes of the aggregated frequency indicators) of different to-be-evaluated log data units (for example, different fields) can be quite different, for example, the aggregation times of core business fields can reach millions of times, while the debugging fields can only have a few times. After normalization, the indicators are compared, which can improve the representativeness and fairness of the indicators. On the other hand, if the numerical ranges of the query frequency and the aggregation frequency differ by several orders of magnitude, normalization processing is performed to map the indicator values to a unified and comparable scale (usually the [0, 1] interval), which can avoid the situation that a large numerical sub-indicator completely dominates the result, resulting in the loss of the adjustment significance of the weight and .

[0055] It should be noted that the normalization calculation method can also be other suitable calculation methods, which are not limited in the present application. For example, the denominator in the above formula 3 ( ) can also consider using a reference value, for example, the average value or the percentile (such as the 95th percentile) of the query frequencies of all to-be-evaluated log data units.

[0056] It should be noted that the present application considers that the higher the optimization priority score is, the more urgent the storage optimization operation needs to be implemented, but based on specific design, the lower the optimization priority score is, the more urgent the storage optimization operation needs to be implemented (at this time, the optimization priority score is inversely proportional to the storage overhead, and is proportional to the use heat of the to-be-evaluated log data unit), which is an equivalent design.

[0057] S103, in the case where the optimization priority score of the to-be-evaluated log data unit meets a preset optimization condition, the electronic device determines that the to-be-evaluated log data unit is a to-be-optimized log data unit that needs to be implemented with a storage optimization operation.

[0058] For example, the preset optimization condition can be that the optimization priority score is greater than or equal to a preset score threshold. For example, the preset score threshold can be a custom numerical value, or can be the average score of all optimization priority scores, or twice the standard deviation of all optimization priority scores relative to the average score, which is not limited in the present application. Alternatively, the preset optimization condition can also be that the optimization priority score ranks the top n from large to small, and n is an integer greater than or equal to 1, for example, n is 2 or other suitable values.

[0059] Common storage optimization method techniques include: 1) field deletion or index rollover based on fixed time rules: this is a common storage optimization method. System administrators pre-set a retention period for certain fields considered "unimportant" based on experience (for example, in addition to core fields, other fields are only retained for 30 days). After the deadline, through a timed task or index lifecycle management strategy, the data of the field that exceeds the deadline is physically deleted or migrated to a lower-cost storage medium. The problem with this method is that the decision is based on static, coarse-grained manual experience, and cannot dynamically and accurately reflect the actual usage of the field. It may delete fields that still have query needs in the future, or retain a large number of "cold fields" that are not accessed for a long time, resulting in inaccurate optimization and high configuration maintenance costs. 2) Field filtering based on manual configuration: during log collection or index construction, pre-defined rules are used to selectively exclude certain fields considered unnecessary, so that they do not enter storage. This method controls the storage content from the source, but the disadvantage is also obvious: the decision is static and front-loaded, and lacks continuous evaluation of the actual value of the field.

[0060] However, by using the screening method for the log data unit to be optimized provided in the present application, the "storage cost" (disk space ratio) and "usage heat" (query frequency and recency) of the log data unit are considered for correlation calculation to generate a comparable priority score, and the log data unit to be optimized with high storage cost but low usage heat is identified. Unlike the extensive mode of fixed time rules, the strategy of the present application is dynamically adjusted, and the retention value of a specific log data unit in a specific time period is dynamically measured through mathematical modeling, which can adapt to changes in business query patterns and accurately locate the log data units that should be optimized at different data life cycle stages.

[0061] In some possible implementations, after the electronic device determines that the log data unit to be evaluated is a log data unit to be optimized that needs to be implemented with a storage optimization operation, the electronic device also matches at least one optimization operation for the log data unit to be optimized based on the data type and the business rule, and the optimization operation includes: 1) Data cleaning: deleting the storage data of the log data unit to be optimized within a pre-set storage time range; applicable to numerical, textual and other independent cleanable fields; specifically, deleting the data of the log data unit to be optimized within a pre-set storage time range (such as a storage duration of more than 90 days) from the physical storage.

[0062] 2) Index mapping degradation: modifying the index mapping to turn off the index function of the log data unit to be optimized; specifically, changing the index type of the field from "searchable" (such as index: true) to "not indexed" (index: false), and only keeping the original value in _source; this can release the space occupied by the index structure, but sacrifices the fast retrieval ability of the field.

[0063] 3) Data migration: migrating the storage data of the log data unit to be optimized within a preset storage time range to a preset lower-cost storage medium; for example, migrating the log data to be optimized (cold data) to a cheaper storage medium (such as object storage), and updating the metadata pointer.

[0064] For example, the log data unit to be evaluated is a log field (i.e., the log data unit to be optimized is a log field), and the electronic device matches at least one optimization operation for the log data unit to be optimized based on the data type and business rules, specifically including: 1) The log data unit to be optimized marked as "core business field" is prohibited from being deleted (prohibited from taking data cleaning operation) even if the preset optimization condition is met, even if the data is cold, but is preferentially selected for data migration operation, and is carefully taken for index mapping degradation operation.

[0065] 2) For the log data unit to be optimized that is not marked as "core business field", is numerical (including long, double, date) or ordinary text (text), the data independence is strong, and the deletion does not affect other fields, and the data cleaning (deletion operation) is taken; 3) For the log data unit to be optimized that is not marked as "core business field", is high-cardinality identification type (including keyword fields such as user identification user_id, order number order_id, etc.), is the key of data association, and deletion will cause the record to be unable to be associated and tracked, but can be degraded from "indexable" to "only storage", and the index mapping degradation or data migration operation is taken; 4) For the log data unit to be optimized that is an analysis key type (fields commonly used for aggregation, such as amount and duration), historical data analysis may still require the field, which is suitable for migration to low-cost storage for occasional queries, and the data migration operation is taken.

[0066] In some possible implementation manners, the electronic device can also select different storage optimization operations for the log data unit to be optimized based on the optimization priority score of the log data unit to be optimized. For example, if the optimization priority score is greater than or equal to a first preset score, the electronic device selects an index mapping degradation or data migration operation based on the data type and the business rule; if the optimization priority score is greater than or equal to a second preset score and less than the first preset score, the electronic device selects a deletion operation. For example, in the formula 1 above, the first preset score can be 0.9, and the second preset score can be 1.5, or other appropriate values based on specific design, which are not limited herein. The first preset score can be 0.9, and the second preset score can be 1.5, or other appropriate values based on specific design, which are not limited herein.

[0067] In some possible implementation manners, in a case where the optimization priority score of the log data unit to be evaluated does not satisfy the preset optimization condition, the electronic device continues to evaluate the optimization priority score of the log data unit to be evaluated in a next preset evaluation time range, and determines, based on the change, whether the log data unit to be evaluated needs to be implemented with a storage optimization operation, and if so, determines a specific storage optimization operation to be implemented.

[0068] In some possible implementation manners, for the optimized log data unit that has been implemented with a storage optimization operation and the storage optimization operation is not a deletion operation, the electronic device continues to calculate the optimization priority score of the optimized log data unit. If it is determined, based on the current optimization priority score of the optimized log data unit, that the optimized log data unit does not need to be implemented with a storage optimization operation, the electronic device stops the corresponding storage optimization operation, restores the corresponding function (for example, stops index degradation and restores the index function, or migrates it from the low-cost storage medium back to the original specific storage medium), and re-labels the optimized log data unit as a log data unit to be evaluated.

[0069] In some possible implementation manners, after the electronic device matches at least one optimization operation for the log data unit to be optimized based on the data type and the business rule, the electronic device further generates a strategy suggestion, the strategy suggestion being a structured object and containing: a target field (the log data unit to be optimized), an index of the target field, an optimization operation type, an operation range (such as a time range doc.create_time < "now-90d"), a predicted space release, a potential impact analysis (such as "this operation will slow down the historical range query for this field"), and the like. The strategy suggestion is pushed to a management console, and after being audited by an administrator, the optimization operation for the log data unit to be optimized is manually triggered, or the optimization operation for the log data unit to be optimized is automatically triggered based on a preset rule (such as an OPS exceeding a higher threshold value and an impact evaluation being "low risk").

[0070] In some possible implementation manners, the electronic device can further support a more flexible retention policy, for example, for some fields, a retention policy of "after being stored for more than 30 days, if there is no access for 60 consecutive days, the data is automatically cleaned up", so as to achieve a better balance between storage efficiency and data availability.

[0071] The embodiment of the present application further provides a screening system of a log data unit to be optimized, including a module for executing any one of the screening methods of the log data unit to be optimized in the above method embodiments.

[0072] Please refer to Figure 2 The embodiment of the present application provides an architecture diagram of a screening system of a log data unit to be optimized. As Figure 2 shown, the screening system of the log data unit to be optimized in the embodiment of the present application can include: a dynamic monitoring and data collection module 201, an intelligent analysis and strategy generation module 202. In addition, the system interacts with a lower-layer log storage platform and an upper-layer management console.

[0073] The dynamic monitoring and data collection module 201 is configured to collect a storage overhead index and a usage heat index of the log data unit to be evaluated in a preset evaluation time range. For example, if the log data unit to be evaluated is a log field, the dynamic monitoring and data collection module 201 is configured to collect index-level metadata and field-level metadata, the index-level metadata including an index name, a creation time, a total number of documents, a total storage size (main shard + replica), and a daily data volume. The field-level metadata includes a storage cost index, a usage heat index, and a field attribute of each field, the field attribute including a field name, a data type (such as keyword, long, text, and the like), and a flag indicating whether the field is indexed.

[0074] Exemplarily, the dynamic monitoring and data collection module 201 specifically comprises: a scheduler, which triggers a collection task periodically based on a timing task window mechanism; a platform adapter, which encapsulates specific management application programming interface calls for different log storage platforms (such as Loggly). For example, the application programming interfaces such as beaver's _stats and _field_usage_stats of Loggly are used to obtain the usage statistics of indexes and fields. A data preprocessing and calculation unit, which cleans, aggregates, and calculates the data returned by the original application programming interface. For example, the original access times are split by fields, the daily query frequency of a field in the last N days is calculated, the percentage of the storage size of a field in the total size of an index is calculated, and the like. A metadata storage, which stores the processed structured metadata in an internal database (such as MySQL) for use by the intelligent analysis and strategy generation module 202.

[0075] The intelligent analysis and strategy generation module 202 is configured to determine an optimization priority score of the to-be-evaluated log data unit based on the storage overhead indicator and the usage heat indicator, and determine that the to-be-evaluated log data unit is a to-be-optimized log data unit that needs to be subjected to a storage optimization operation if the optimization priority score of the to-be-evaluated log data unit meets a preset optimization condition.

[0076] In some possible implementation manners, the intelligent analysis and strategy generation module 202 is further configured to match at least one storage optimization operation for the to-be-optimized log data unit based on a data type and a business rule, and the storage optimization operation comprises: deleting storage data of the to-be-optimized log data unit within a preset storage time range, modifying an index mapping to close an index function of the to-be-optimized log data unit, and migrating the storage data of the to-be-optimized log data unit within the preset storage time range to a preset lower-cost storage medium.

[0077] In some possible implementation manners, the intelligent analysis and strategy generation module 202 is further configured to, in a case where the optimization priority score of the to-be-evaluated log data unit does not meet the preset optimization condition, continue to evaluate the optimization priority score of the to-be-evaluated log data unit in a next preset evaluation time range, and determine whether the to-be-evaluated log data unit needs to be implemented with a storage optimization operation based on a change; and for the to-be-optimized log data unit that has been implemented with a corresponding storage optimization operation and the storage optimization operation is not a deletion operation, taking the to-be-optimized log data unit as an optimized log data unit, and continuously counting the optimization priority score of the optimized log data unit; if it is determined that the optimized log data unit does not need to continue to be implemented with a storage optimization operation (the optimization priority score does not meet the preset optimization condition) based on the current optimization priority score of the optimized log data unit, stopping the corresponding storage optimization operation of the optimized log data unit, resuming the corresponding function, and re-labeling the optimized log data unit as a to-be-evaluated log data unit.

[0078] In some possible implementation manners, the intelligent analysis and strategy generation module 202 is further configured to generate a strategy suggestion, where the strategy suggestion is a structured object, and includes a target index, a target field, an optimization operation type, an operation range, a predicted released space, and potential impact analysis. The strategy suggestion can be pushed to a management console, and manually triggered after being audited by an administrator; or can be automatically entered into an execution queue according to a preset rule (for example, an OPS exceeds a higher threshold value, and an impact assessment is "low risk").

[0079] In some possible implementation manners, the system further includes a strategy execution and management module 203, which specifically includes an executor configured to convert the strategy suggestion into a specific operation command for the log storage platform. For example, for "field data cleaning", the _update_by_query request of Beaver can be executed to set the field value of the document in a specific time range to null, and space is recovered by cooperating with mapping update and forced merge. The execution process needs to consider batch processing, retry mechanism, failure rollback and concurrency control to avoid performance impact on online queries. A state manager is configured to maintain the life cycle state (for example, "to be executed", "in execution", "success", "failure" and "rolled back") of all strategies. Detailed information (start / end time, operation object, number of affected rows and released space) of each strategy execution is recorded. The state of the log storage platform is kept synchronized to ensure consistency of the system view.

[0080] In the embodiments of the present application, any implementation manner and corresponding term meaning mentioned in the method embodiments are also applicable to the to-be-optimized log data unit screening system provided in the present application, and the specific implementation steps can be referred to the description of the foregoing method embodiments, which will not be described in detail here.

[0081] The embodiment of the present application further provides a screening device for a log data unit to be optimized, which comprises a unit for executing any one of the screening methods for the log data unit to be optimized in the above method embodiments.

[0082] The embodiment of the present application further provides a screening device for a log data unit to be optimized, which comprises a processor for executing any one of the screening methods for the log data unit to be optimized in the above method embodiments.

[0083] Please refer to Figure 3 Another structure diagram of the screening device for a log data unit to be optimized provided by the embodiment of the present application is shown in Figure 3 The screening device 300 for a log data unit to be optimized can comprise at least one processor 301, such as a CPU, at least one communication interface 303, a memory 304 and at least one communication bus 302. The communication bus 302 is used to realize the connection and communication among the components. The communication interface 303 can optionally comprise a standard wired interface, a wireless interface (such as a WI-FI interface or a Bluetooth interface, etc.). The memory 304 can be a high-speed RAM memory or a non-volatile memory (such as at least one disk memory). The memory 304 can optionally be at least one storage device located away from the aforementioned processor 301. As shown in Figure 3 The memory 304 as a computer storage medium can comprise an operating system, a network communication module and program instructions.

[0084] In the screening device 300 for a log data unit to be optimized shown in Figure 3 The processor 301 can be used to load the program instructions stored in the memory 304 and specifically execute any one of the screening methods for the log data unit to be optimized in the above method embodiments.

[0085] It should be noted that the specific execution process can refer to the specific description of the above method embodiments, which will not be described in detail here.

[0086] The specific execution steps can refer to the description of the aforementioned method embodiments, which will not be described in detail here.

[0087] The embodiment of the present application further provides a computer storage medium, which can store a plurality of instructions. The instructions are suitable for being loaded by a processor and executing the screening method for a log data unit to be optimized provided by the embodiment of the present application. The specific execution process can refer to the specific description of the above method embodiments, which will not be described in detail here.

[0088] The embodiment of the present application further provides a computer program product comprising instructions, which, when executed on an electronic device, cause the electronic device to perform the method steps of the method embodiment shown above.

[0089] The embodiment of the present application further provides a chip module, comprising a transceiver assembly and a chip, the chip being configured to perform the method steps of the method embodiment shown above.

[0090] It can be understood that the screening system of the to-be-optimized log data unit, the screening device of the to-be-optimized log data unit, the computer storage medium, the computer program, the computer program product, and the chip provided above are all configured to perform the method shown in any implementation manner in the corresponding aspect of the embodiment of the present application. Therefore, the beneficial effects that can be achieved are referable to the beneficial effects in the corresponding method, which will not be described in detail here.

[0091] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments are included.

[0092] At least one of the present application relates to one or more of the (item), indicating one or more (item). More than two (item), refers to two (item) or more than two (item). "And / or", describes the relationship between the associated objects, indicating that there may be three kinds of relationship, for example, A and / or B, can represent: A alone, A and B exist at the same time, B alone, these three cases. The character " / " generally represents the relationship between the front and rear associated objects is a "or" relationship. In addition, it should be understood that although the present application may adopt the terms first, second and the like to describe various objects, these objects should not be limited to these terms. These terms are only used to distinguish each object from each other.

[0093] The terms "include" and "have" mentioned above and any variations thereof are intended to cover non-exclusive inclusion.

[0094] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for filtering log data units to be optimized, characterized in that, The method includes: Obtain the storage overhead and usage frequency metrics of the log data unit to be evaluated within the preset evaluation time range; Based on the storage overhead metric and the usage popularity metric, the optimization priority score of the log data unit to be evaluated is determined. The optimization priority score is directly proportional to the storage overhead metric and inversely proportional to the usage popularity metric. The higher the optimization priority score, the higher the priority of the log data unit to be evaluated as a log data unit to be optimized. If the optimization priority score of the log data unit to be evaluated meets the preset optimization conditions, the log data unit to be evaluated is determined to be a log data unit that needs to be optimized by storage optimization operation.

2. The method as described in claim 1, characterized in that, The storage overhead metric is related to disk space usage, and the usage popularity metric includes at least two of the following: The query frequency within the first preset statistical period indicates the number of times the query is used as a query condition or returned result within the preset statistical period. The aggregation operation frequency within the second preset statistical period indicates the number of times aggregation operations were performed. The query relevance indicates the freshness of the query.

3. The method as described in claim 1 or 2, characterized in that, The optimization priority score is calculated using the following formula: Where field represents the log data unit being evaluated, and T represents the preset evaluation time range. This indicates the optimization priority score of the evaluated log data unit. This represents the storage cost of the evaluated log data unit within a preset evaluation time range T. This represents the popularity score of the evaluated log data unit within the preset evaluation time range T. W_s and W_h are weighting coefficients, and ε is a given real number used to prevent division by zero errors. The heat score The result is calculated by weighting at least two of the following sub-indicators: normalized query frequency, normalized aggregation operation frequency, and time decay factor based on the most recent access time.

4. The method as described in claim 3, characterized in that, The heat score Calculated using the following formula: Where F_query is the normalized query frequency, F_recency is the time decay factor, and F_aggregation is the normalized aggregation operation frequency. , , These are the weighting coefficients.

5. The method according to any one of claims 1 to 4, characterized in that, After determining that the log data unit to be evaluated is a log data unit that needs to be optimized through storage optimization operations, the method further includes: Based on data type and business rules, at least one storage optimization operation is matched for the log data unit to be optimized. The storage optimization operation includes: deleting the stored data of the log data unit to be optimized within a preset storage time range, modifying the index mapping to disable the index function of the log data unit to be optimized, and migrating the stored data of the log data unit to be optimized within the preset storage time range to a preset lower cost storage medium.

6. The method according to any one of claims 1 to 5, characterized in that, After determining that the log data unit to be evaluated is a log data unit that needs to be optimized through storage optimization operations, the method further includes: If the optimization priority score of the log data unit to be evaluated does not meet the preset optimization conditions, the optimization priority score of the log data unit to be evaluated will continue to be evaluated within the next preset evaluation time range, and based on the changes, it will be determined whether the log data unit to be evaluated needs to be subject to storage optimization operations. For the log data unit to be optimized that has already undergone corresponding storage optimization operations and whose storage optimization operations are not deletion operations, the log data unit to be optimized is designated as an optimized log data unit, and the optimization priority score of the optimized log data unit is continuously calculated. If it is determined based on the current optimization priority score of the optimized log data unit that the optimized log data unit no longer needs to undergo storage optimization operations, then the corresponding storage optimization operations for the optimized log data unit are stopped, the corresponding functions are restored, and the optimized log data unit is remarked as a log data unit to be evaluated.

7. The method according to any one of claims 1 to 6, characterized in that, The storage duration of the log data unit to be evaluated exceeds the preset storage duration.

8. The method according to any one of claims 1 to 7, characterized in that, The log data unit to be evaluated is the log field.

9. A filtering system for log data units to be optimized, characterized in that, Includes a module for performing the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed, performs the method according to any one of claims 1 to 8.

Images