Log management method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a log management method, device, electronic device and storage medium. The method includes: dividing the product log into multiple indexes according to the log type, forming the information in the index storage area into aggregate information according to the preset aggregation conditions under each index, and storing the aggregate information in the storage data area. The embodiment of the present invention divides the product log into multiple indexes according to the log type of the product, so that the index category is clear and easy to manage; by forming the information in the index storage area into aggregate information according to the preset aggregation conditions under different indexes, and storing the aggregate information in the storage data area, the problem of the service quality control demand for space occupancy is solved, the full index aggregation operation of the index storage area itself is reduced, and the cluster performance and stability of the space occupied in the index storage area are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log management method, device, electronic equipment and storage medium. Background Art

[0002] Elasticsearch (ES for short) is a search server based on Lucene. It provides a distributed multi-user full-text search engine cloud platform product. ES can provide users with log storage services and record billions of logs.

[0003] The log service needs to do storage quality control to constrain the user's usage space. The log volume cannot grow without limit, and the query of the size of the log storage space usage cannot affect the cluster performance and stability of the space occupied in the index storage area. In the prior art, there are two main types of log storage management methods. One is to create indexes according to user dimensions, and each index belongs to only one user. In this way, you can know the space occupancy by directly querying the index information, and then delete the old logs through delete_by_query. However, this approach has a large number of indexes, which can easily cause the ES cluster to eat up a lot of memory, and it is very easy to cause the cluster performance and stability of the space occupied in the index storage area to be substandard. There are also many different types of log formats on the product side. If one index is used to store different types of log documents, it will increase the complexity of management and the degree of redundancy will also increase; if multiple indexes (such as user1-ips, user1-policy) are used, the cluster performance and stability problems of the space occupied in the index storage area will be more serious, and the efficiency will be extremely low. Another method is to create an index based on the time dimension. The document attributes in the index identify the user identity by adding fields. Although this approach solves the deletion efficiency problem, it cannot solve the service quality control requirements for space occupancy. Summary of the invention

[0004] In view of this, the present invention provides a log management method, device, electronic device and storage medium, which can solve the problem of service quality control demand for space occupancy, improve the cluster performance and stability of occupied space in the index storage area, and reduce the full index aggregation operation of the index storage area itself.

[0005] According to one aspect of the present invention, an embodiment of the present invention provides a log management method, the method comprising:

[0006] Divide product logs into multiple indexes according to log types;

[0007] Under each index, the information in the index storage area is grouped into aggregate information according to a preset aggregation condition;

[0008] The aggregate information is stored in a data storage area.

[0009] According to another aspect of the present invention, an embodiment of the present invention further provides a log management device, the device comprising:

[0010] Index partitioning module, used to divide product logs into multiple indexes according to log types;

[0011] An information composition module, used to compose the information in the index storage area into aggregated information according to preset aggregation conditions under each index;

[0012] The information storage module is used to store the aggregate information into a storage data area.

[0013] According to another aspect of the present invention, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log management method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log management method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention divides the product log into multiple indexes according to the log type of the product, and under each index, the information in the index storage area is combined into aggregate information according to preset aggregation conditions, and the aggregate information is stored in the storage data area. The embodiment of the present invention divides the product log into multiple indexes according to the log type of the product, so that the index category is clear and easy to manage; by combining the information in the index storage area into aggregate information according to preset aggregation conditions under different indexes, and storing the aggregate information in the storage data area, the demand problem of service quality control of space occupancy is solved, the full index aggregation operation of the index storage area itself is reduced, and the cluster performance and stability of the space occupied in the index storage area are improved.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a log management method provided in Embodiment 1 of the present invention;

[0022] Figure 2 A flow chart of a log management method provided in Embodiment 2 of the present invention;

[0023] Figure 3 A flow chart of a log management method provided in Embodiment 3 of the present invention;

[0024] Figure 4 A flow chart of a collection method aggregated by time unit provided in Embodiment 3 of the present invention;

[0025] Figure 5 A flowchart of a method for cleaning historical data provided in Embodiment 3 of the present invention;

[0026] Figure 6 is a structural block diagram of a log management device provided by Embodiment 4 of the present invention;

[0027] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] In one embodiment, Figure 1 This is a flow chart of a log management method provided in the first embodiment of the present invention. This embodiment can be applied to the storage method, storage space and time management of logs. The method can be executed by a log management device. The log management device can be implemented in the form of hardware and / or software. The log management device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110: Divide the product log into multiple indexes according to the log type.

[0033] The log type can be understood as the relevant log type on the product side. Of course, the log type on the product side can be multiple, and the log type can be a general query log, a slow query log, or a relay log, which is not limited in this embodiment. The product log can be understood as the log of various products, such as security products, potential products, and new products.

[0034] In this example, Elasticsearch refers to distributed document storage. An index can be viewed as an optimized collection of documents, each document can be a collection of fields, and a field can be a key-value pair containing data. Elasticsearch indexes all the data in each field, and each indexed field has a dedicated, optimized data structure.

[0035] In this embodiment, the logs of various products, such as security products, can be divided into multiple indexes according to the log type of the product side. Exemplarily, when the product on the product side is a security product, the security product can be divided into indexes such as ips, policy, traffic, av, ddos, etc. according to the log type of the security product. It should be noted that the division of the product side index is related to the relevant product business and the table structure of the relevant database. Different product businesses have different corresponding database table structures.

[0036] It should be noted that each index has a corresponding document attribute, and the document attribute of each index contains multiple fields, and different fields can be mapped to different dimensions. The fields contained in the document attributes of each index can be fields of the time dimension, fields of the space dimension, and fields of the user dimension. Of course, in addition to the above fields, you can also customize the required fields according to the corresponding needs, such as log version, log availability zone, etc.

[0037] S120. Under each index, the information in the index storage area is grouped into aggregated information according to a preset aggregation condition.

[0038] Among them, the preset aggregation conditions can be understood as the pre-set aggregation conditions of related log document information. The index storage area can be understood as the index cluster storage and management area, for example, it can be the ES index cluster management and storage area. It should be noted that ES can distribute the index data content to the hard disk in the cluster through the sharding design. Aggregation information can be understood as the information obtained by aggregating the data information in the index cluster storage and management area according to the pre-set data information aggregation conditions. Of course, the aggregation information can include the usage space information of the relevant log documents, the total number of logs, etc.

[0039] In this embodiment, the preset aggregation condition can be the time dimension in the document attributes of each index, which sets the user's storage size aggregation time unit, such as days, hours, etc.; it can also be the space dimension in the document attributes of each index, which is used to obtain the spatial size of the aggregated information; it can also be the user identity in the document attributes of each index; it can also be a combination of the time dimension, space dimension and user dimension in the document attributes of each index as a pre-set data information aggregation condition, and this embodiment does not limit this.

[0040] In this embodiment, under each index, according to the pre-set aggregation conditions of related log document information, the related log document information in the index cluster storage and management area can be aggregated accordingly to obtain aggregated information. Specifically, the document attributes of the related indexes in the index cluster storage and management area can be searched for the earliest stored related log document, and the most recently stored related data information can be searched in the storage data area, so as to determine the related information to be aggregated according to the earliest stored related log document and the most recently stored related data information, and on this basis, the related information to be aggregated can be aggregated accordingly according to the pre-set aggregation conditions of related log document information to obtain aggregated information.

[0041] S130: Store the aggregated information into a data storage area.

[0042] The storage data area can be understood as a storage area for aggregate information, such as a relational database or a non-relational database, which is not limited in this embodiment. The non-relational database can be a key-value storage database, a column storage database, and the like.

[0043] In this embodiment, the aggregate information obtained according to the pre-set aggregation conditions of the relevant log document information needs to be stored in the storage data area. It should be noted that the storage data area may include an index table, a service quality control table and a related field table, such as an overall space size table, a log storage period table, a statistical storage unit table, a total table and a total space usage information table, etc. Among them, the index table can be used to record the index name required by the product; the service quality control table can be used to record storage space limitation information and related service quality control information; the overall space size table is used to indicate the storage space limitation size; the log storage period table is used to indicate the effective time of log storage; the statistical storage unit table is used to express the time span of collecting "space usage information"; the total table is used to indicate the number of used items; the total space usage information table is used to indicate the size of space usage.

[0044] The above technical solution of the embodiment of the present invention divides the product log into multiple indexes according to the log type of the product, and under each index, the information in the index storage area is combined into aggregate information according to preset aggregation conditions, and the aggregate information is stored in the storage data area. The embodiment of the present invention divides the product log into multiple indexes according to the log type of the product, so that the index category is clear, easy to manage, and easy to maintain the stability of the ES cluster; by combining the information in the index storage area into aggregate information according to preset aggregation conditions under different indexes, and storing the aggregate information in the storage data area, the problem of the service quality control demand for space occupancy is solved, the full index aggregation operation of the index storage area itself is reduced, and the cluster performance and stability of the space occupied in the index storage area are improved.

[0045] Embodiment 2

[0046] In one embodiment, Figure 2 This is a flow chart of a log management method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further refines the process of assembling the information in the index storage area into aggregated information according to preset aggregation conditions under each index, such as Figure 2 As shown, the log management method in this embodiment may specifically include the following steps:

[0047] S210: Divide the product log into multiple indexes according to the log type.

[0048] S220: Read the document attributes of the index from the index storage area, search the document attributes for the earliest log document stored in the index, and record the storage time of the log document as the first time.

[0049] Each log document may be a piece of log data, each index may contain multiple log documents, and each log document corresponds to a corresponding log generation time. The first time may be understood as the earliest log document time generated when the log document is stored.

[0050] In this embodiment, the fields included in the document attributes of each index may be: a field of the time dimension, which can be used to identify the timestamp of log generation; a field of the space dimension, which can be used to identify the byte size of the log; and a field of the user dimension, which can be used to identify the identity of the user. Exemplarily, @timestamp is used to identify the time when the log is generated; @bytesize is used to identify the size of the log; and user_id is used to identify the user's identity. Of course, in addition to the above fields, the required fields can also be customized according to corresponding needs, such as @version, which is used to identify the log version, zone_id is used to identify the available zone, and so on. It should be noted that the fields @timestamp, @bytesize, and user_id can all be added accordingly at the log collection end.

[0051] In this embodiment, since the time when the log was generated is identified in the document attributes of each index, the time dimension sets a certain storage time and time limit for the log. It is necessary to record the time when each log is generated, and read the document attributes of the index from the index cluster storage and management area, so as to find the earliest log document stored in the index from the field of the time dimension in the document attributes of the index.

[0052] S230: Acquire the most recently stored data information in the data storage area, and record the storage time of the acquired data information as the second time.

[0053] The data information may be understood as the latest piece of data information stored in a data storage area, such as a relational database. The second time may be understood as the time when the latest data information is obtained from the data storage area.

[0054] In this embodiment, the query method can be constructed by the relevant time field, for example, it can be "select * from statistic_store_hour where primary_id = 'usr-1' and secondary_id = 'ips'

[0055] order by end_time desc limit 1;” to obtain the relevant data information with the latest storage time from the storage data area, and record the storage time of the obtained data information as the second time. It should be noted that the latest data information is the relevant data information within the time range of the hour, the whole day or the whole minute. For example, the latest data information may be 2022-04-20 10:00:00~2022-04-20

[0056] 11:00:00, or 2022-04-21 00:00:00 to 2022-04-22 00:00:00, and so on.

[0057] In this embodiment, if the most recent data information stored in the data storage area is empty, that is, the most recent data information cannot be obtained, it means that no relevant data has been collected.

[0058] S240: If the first time is earlier than the second time, the information in the index storage area whose storage time is later than the second time is used as the information to be aggregated.

[0059] The information to be aggregated can be understood as relevant information waiting to be aggregated.

[0060] In this embodiment, when the time of the earliest log document generated during storage is earlier than the time of the most recent data information obtained from the storage data area, it is necessary to use the information of the time period in the index storage area that is later than the time of the most recent data information obtained from the storage data area as the information to be aggregated, and according to the corresponding preset aggregation conditions, such as the aggregation time unit of the time dimension, the information to be aggregated is aggregated accordingly to obtain the aggregated information.

[0061] It should be noted that the earliest log document time generated when the log document is stored, i.e., the first time, and the time of the most recent data information obtained from the storage data area, i.e., the second time, may not exist. When the earliest log document time generated when the log document is stored does not exist, it can be considered that there is no data information in the index and the index ends directly; when the earliest log document time generated when the log document is stored exists, but the time of the most recent data information obtained from the storage data area does not exist, it can be considered that data information has not been collected, and the earliest log document time generated when the log document is stored, i.e., the first time, can be used as the information to be aggregated.

[0062] S250: If the first time is later than or equal to the second time, use the information in the index area as information to be aggregated.

[0063] In this embodiment, when the time of the earliest log document generated during storage is later than or equal to the time of the most recent data information obtained from the storage data area, it is necessary to use the information from the storage time to the current time in the index storage area as information waiting to be aggregated, and on this basis, according to the corresponding preset aggregation conditions, such as the aggregation time unit of the time dimension, the information to be aggregated is aggregated accordingly to obtain aggregated information.

[0064] S260: Aggregate the information to be aggregated into aggregated information according to a preset aggregation condition.

[0065] In this embodiment, according to the aggregation time period in the aggregation condition, such as the range of hours or days, the log usage space information of each information to be aggregated in the same aggregation time period can be added to obtain the aggregated information in each time period.

[0066] In one embodiment, aggregating the information to be aggregated into aggregated information according to a preset aggregation condition includes:

[0067] Extracting the aggregation time period in the preset aggregation condition;

[0068] The used spaces of the information to be aggregated whose storage times belong to the same aggregation time period are summed up as the aggregation information.

[0069] The aggregation time period can be understood as a time unit within the range of the aggregation time hour. For example, the aggregation time period can be in hours or days, which is not limited in this embodiment.

[0070] In this embodiment, each piece of information to be aggregated may be divided into different aggregation time periods according to the corresponding storage time, and the usage space information of each piece of information to be aggregated corresponding to the same aggregation time period may be added together to obtain the aggregate information. Exemplarily, time1 is 2022-03-29T00:00:00, time2 is 2022-03-29T00:12:34, time3 is 2022-03-29T01:00:00, time4 is db_end_time, and the current time is recorded as now. At this time, time1 indicates the start time of time2 at the hour of the hour; time2 indicates the earliest document time of the corresponding index of ES; time3 indicates the end time of time2 at the hour of the hour of the hour, and the expected time is the end_time of a data record in the storage data area; time4 indicates the end_time of the latest record in the storage data area; then at this time, spatial information is used for summation to obtain aggregation information, and the corresponding aggregation information can be expressed as: sum=es_data(time2~time3)+db_data(time3~time4)+es_data(time4~now); wherein es_data indicates obtaining aggregation information from ES, and db_data indicates obtaining aggregation information from the storage data area. When there are data records in the storage data area (in this embodiment, it can be a relational database, referred to as DB), the current space usage size is composed of three parts. The first part: obtained from ES, time2~time3 are aggregated and summed to obtain the information to be aggregated; the second part: obtained from DB, time3~time4 are aggregated and summed to obtain the information to be aggregated; the third part: obtained from ES, time4~now are aggregated and summed to obtain the information to be aggregated, and then the three parts of the used space information are added together to obtain the aggregate information in the corresponding time period.

[0071] S270: Store the aggregated information into a data storage area.

[0072] The above-mentioned technical scheme of the embodiment of the present invention divides the product log into multiple indexes according to the log type, so as to realize clear index categories, easy management and easy maintenance of the stability of the ES cluster; obtains the most recent data information stored in the storage data area, and records the storage time of the acquired data information as the second time. When the first time is earlier than the second time, the information stored in the index storage area later than the second time is used as the information to be aggregated; when the first time is later than or equal to the second time, the information in the index area is used as the information to be aggregated, and the information to be aggregated is aggregated into aggregated information according to preset aggregation conditions, and the aggregated information is stored in the storage data area, which further solves the problem of service quality control demand for space occupancy, reduces the full index aggregation operation of the index storage area itself, and improves the cluster performance and stability of the space occupied in the index storage area.

[0073] In one embodiment, after storing the aggregate information in the data storage area, the method further includes:

[0074] The used space information of the aggregate information in the index storage area is cleaned up according to the preset time.

[0075] Among them, the preset time can be understood as a pre-set periodic time, which is used to periodically clean up the usage space information of the aggregated information in the index storage area. Exemplarily, the preset time can be 5 minutes, 8 minutes, or 10 minutes, which is not limited in this embodiment. Of course, the preset time can be set by itself as needed, or it can be set accordingly based on experience, which is not limited in this embodiment. The usage space information can be understood as the spatial information obtained by adding the usage space of each piece of information to be aggregated.

[0076] In this embodiment, the usage space information of the log documents in the aggregate information is cleaned up to a certain extent according to the preset periodic time. Specifically, it can be cyclically determined whether the current usage space information exceeds the relevant limit value. If it does not exceed the relevant limit value, there is no need to clean up the usage space information of the aggregate information in the index storage area accordingly; if it exceeds the relevant limit value, the data is cleaned up gradually according to the order of the relevant storage time and according to a certain time step.

[0077] In one embodiment, clearing the usage space information of the aggregate information in the index storage area according to a preset time includes:

[0078] Determine whether the used space information exceeds the limit value;

[0079] When the space information usage exceeds the limit, the information in the index is sorted according to the storage time;

[0080] Clean up the data of each information based on sorting and preset time granularity to free up space.

[0081] The limit value can be understood as the relevant limit value of the aggregation information of the spatial information used by each index, which is used to determine whether data needs to be cleaned up. The storage time can be understood as the storage time of the log documents in each index, and the log documents in each index can be sorted accordingly from the earliest to the latest storage time. The preset time granularity can be understood as a certain fine-grained time step set in advance, such as hours or minutes, to gradually clean up the data information within the corresponding time range.

[0082] In this embodiment, the usage space information of the aggregate information is periodically cycled to determine whether the usage space information exceeds the limit value. If the usage space information of the aggregate information does not exceed the relevant limit value, there is no need to perform corresponding cleaning of the usage space information of the aggregate information in the index storage area; if the usage space information of the aggregate information exceeds the relevant limit value, the data is cleaned up step by step according to the sorting of the relevant storage time and a certain time step. Specifically, if the usage space information of the aggregate information exceeds the relevant limit value, the storage time of the earliest log document stored in each index is obtained from the index storage area, and the time range is obtained in combination with the preset time granularity, and each index is cycled to obtain a list of time ranges to be deleted for each index, and a certain sorting is performed, and the earliest log document is placed at the front of the list, and the data of each information is cleaned up according to a certain preset time granularity to release space until the usage space information of the aggregate information does not exceed the relevant limit value.

[0083] In this embodiment, although the delete_by_query deletion method is also used, no operation is performed on a large number of indexes, and dimensions such as time range are used as query constraints. Therefore, the impact on the stability and performance of the ES cluster is greatly reduced, and the intensity of ES aggregation queries can be greatly reduced, thereby reducing the usage load of the ES cluster and obtaining accurate space usage information.

[0084] In one embodiment, sorting the information in the index according to a preset time granularity includes:

[0085] Obtain the storage time of the earliest log document stored in each index from the index storage area;

[0086] Sort each piece of information within the time range of the current time and the storage time.

[0087] The current time is at the hourly granularity level, and the time range can be obtained according to the hourly unit.

[0088] In this embodiment, there are multiple indexes, and when the usage space information exceeds the relevant limit value, a corresponding loop can be performed according to the index, and the storage time of the earliest log document stored in each index can be obtained from the index storage area. At this time, it can be determined whether the storage time of the earliest log document obtained in the index storage area exists. If the storage time of the earliest log document exists, the information in the time range to be deleted within the time range of the entire hour can be sorted according to the hour unit, and the storage time of the earliest log document is placed at the front.

[0089] In one embodiment, cleaning the data of each information to release space according to the sorting and preset time granularity includes:

[0090] Divide the information in the sorting into multiple groups according to the time granularity;

[0091] The information belonging to each group in the index storage area is cleaned up in order from the most recent time until the used space information after cleaning is less than the limit value.

[0092] The time granularity may be understood as a certain time step, such as hours, minutes, etc.

[0093] In this embodiment, the information in the sorting is divided into different groups according to a certain time step, such as hours, minutes, etc. Different groups correspond to different time ranges, and the data information under different time steps is cleaned up in the order from the earliest storage time to the latest storage time. If the usage space information is greater than or equal to the limit value at this time, the process returns to continue to delete the information within the time range of the storage time step by step according to a certain time step until the cleaned usage space information is less than the limit value.

[0094] Embodiment 3

[0095] In one embodiment, to facilitate a better understanding of the log management method, Figure 3 This is a flow chart of a log management method provided by Embodiment 3 of the present invention. The embodiment of the present invention is described from two aspects: log storage method and storage space and time management. In this embodiment, ES represents the index storage area in the above embodiment, and the database represents the storage data area in the above embodiment. The specific steps are as follows:

[0096] S310: Create an index based on the log type on the product side and divide it into multiple indexes.

[0097] In terms of storage methods, you can create indexes according to the log types on the product side. For example, security products can be divided into ips, policy, traffic, av, ddos ​​and other indexes according to log types. Dividing products into multiple indexes makes the categories clear, easy to manage and easy to maintain the stability of the ES cluster. Three fields are added to the document attributes in each index: @timestamp identifies the time when the log is generated; @bytesize identifies the size of the log; user_id identifies the user identity; of course, you can also customize the required fields as needed, such as @version identifies the log version, zone_id identifies the available zone (this is useful when multiple available zones share the ES cluster under the region architecture), @bytesize can be aggregated and "summed" to get the storage size information; @timestamp can be used to know the time when each document was generated.

[0098] S320 , aggregate the log size of the corresponding time range by setting the aggregation time unit to obtain the log space size of each time period.

[0099] In this embodiment, the aggregation time unit represents the preset aggregation conditions in the above embodiments, sets the space size, product, index information, etc. available to each user, and then determines the aggregate log size. The aggregation time unit is set to aggregate the log size of the corresponding time range and record it. If the total amount of indexed data is directly summed, performance problems are likely to occur, especially for high-level data volumes, which can easily make performance substandard. Therefore, it is necessary to set the user's storage size aggregation time unit, such as days, hours, etc., and then periodically, for example, 5 minutes, determine whether aggregation is needed; if necessary, the "sum" of @bytesize is aggregated according to the time unit to obtain the space size of each time period.

[0100] S330: Store the aggregated information in a database.

[0101] In this embodiment, the "sum" of @bytesize is aggregated according to the time unit to obtain the space size of each time period, and the aggregated information is stored elsewhere, such as a relational database. In this way, when querying using spatial information, the results can be obtained by using the aggregated information, thereby reducing the full index aggregation operation of ES itself.

[0102] S340: Periodically clean up the usage space information of the aggregate information.

[0103] In this embodiment, data is deleted periodically, for example, every 10 minutes, to determine whether it needs to be deleted. If necessary, the earliest document record time point of each index is collected, and then deleted step by step according to the fine-grained time dimension such as "by hour" using delete_by_query (the query condition is the time range and user identity) until the usage is less than the total space available to the user or the earliest time of the document of each index meets the log storage time. Although this embodiment also uses the delete_by_query deletion method, it does not operate on a large number of indexes, and also uses dimensions such as time range as conditions for query constraints, thereby greatly reducing the impact on the stability and performance of the ES cluster.

[0104] In one embodiment, Figure 4This is a flowchart of a collection method aggregated by time unit provided in the third embodiment of the present invention, wherein es_oldest_one indicates searching for the oldest log document stored in the index in the document attributes in the above embodiment, db_newest_one indicates obtaining the most recent data information stored in the storage data area in the above embodiment, ES indicates the index storage area in the above embodiment, and SQLDB indicates the storage data area in the above embodiment. The specific steps are as follows:

[0105] S410, ES obtains the earliest indexed document es_oldest_one.

[0106] S420: Does the document es_oldest_one exist? If so, execute S430; otherwise, terminate directly.

[0107] S430, SQLDB obtains the most recent data db_newest_one

[0108] S440: Does the data db_newest_one exist? If yes, execute S450; otherwise, execute S460.

[0109] S450. Get the time range according to the time of db_newest_one['end_time'].

[0110] S460. Obtain a time range of the entire hour according to the time of es_oldest_one.

[0111] S470, the time of es_newest_one must be greater than the time range. Is the time valid? If so, execute S480; if not, end directly.

[0112] S480, ES aggregates data according to the time range.

[0113] S490. The result is stored in SQLDB.

[0114] In this embodiment, the method for obtaining the time range is as follows: time1 is represented by 2022-03-29T00:00:00Z, time2 is represented by 2022-03-29T00:12:34Z, and time3 is represented by 2022-03-29T01:00:00Z. If the time unit is hour, to obtain the time range of time2, the time range is represented as time1~time3; if the time unit is day, to obtain the time range of time2, the time range is represented as time1~2022-03-30T00:00:00Z.

[0115] In one embodiment, Figure 5This is a flowchart of a method for cleaning historical data provided by Embodiment 3 of the present invention. This embodiment is described by taking the time range obtained by hourly units as an example. The specific steps are as follows:

[0116] S510, obtain the sum of the usage sizes of each index space.

[0117] S520: Whether it exceeds the limit value, if so, execute S530, if not, directly end.

[0118] S530, loop according to the index.

[0119] S540: ES obtains the earliest indexed document time.

[0120] S550: Does the earliest indexed document time exist? If yes, execute S560; if not, execute S540.

[0121] S560, obtaining a time range of the entire hour by hour unit.

[0122] S570: After the index loop is completed, a list of time ranges to be deleted of each index is collected.

[0123] S580. Sort the list so that the earliest time is placed first.

[0124] S590, loop by index.

[0125] S5100. Delete the user data in this range according to a certain time step (such as hours or minutes).

[0126] S5110. Obtain the sum of the usage sizes of each index space.

[0127] S5120: Whether it exceeds the limit value, if so, continue to execute S5100, if not, end directly.

[0128] In this embodiment, the storage space management is mainly composed of two modules, a time unit collection module and a historical data cleaning module. Both modules perform operations periodically, but we need to control the timing and form tasks according to the user dimension. This task is first performed in the collection module and then in the historical data cleaning module to avoid conflicts. The collection module aggregates and collects data into the relational database according to the hourly time range; the historical data cleaning module obtains the hourly time range by hourly units, and gradually cleans the data according to a certain time step. In this way, the stability and performance of the ES cluster can be minimized; the use of space information summarized according to the time unit recorded in the relational database can greatly reduce the intensity of ES aggregation queries, thereby reducing the usage load of the ES cluster and obtaining accurate space usage information.

[0129] In this embodiment, in terms of storage time management, ES retrieves the earliest indexed document, es_oldest_one; the time range is es_oldest_one as the start time, and the valid time point as the end time. The data in this time range is cleaned up gradually according to a certain fine-grained time step (hours or minutes).

[0130] Embodiment 4

[0131] In one embodiment, Figure 6 : is a structural block diagram of a log management device provided in the fourth embodiment of the present invention. The device is suitable for the situation when the log storage method and storage space management are performed. The device can be implemented by hardware / software. It can be configured in an electronic device to implement a log management method in the embodiment of the present invention. Figure 6 As shown, the device includes: an index partitioning module 610, an information composition module 620 and an information storage module 630.

[0132] The index division module 610 is used to divide the product log into multiple indexes according to the log type.

[0133] The information composition module 620 is used to compose the information in the index storage area into aggregated information according to preset aggregation conditions under each index.

[0134] The information storage module 630 is used to store the aggregate information into a data storage area.

[0135] In an embodiment of the present invention, an index division module divides product logs into multiple indexes according to the log types of the products, thereby achieving clear index categories, easy management, and easy maintenance of the stability of the cluster performance of the space occupied in the index storage area; an information composition module composes the information in the index storage area into aggregated information according to preset aggregation conditions under different indexes, and the information storage module stores the aggregated information in the storage data area, thereby solving the problem of service quality control requirements for space occupancy, reducing the full index aggregation operations of the index storage area itself, and improving the cluster performance and stability of the space occupied in the index storage area.

[0136] In one embodiment, the information composition module 620 includes:

[0137] The first time acquisition unit is used to read the document attributes of the index from the index storage area, search the document attributes for the earliest log document stored in the index, and record the storage time of the log document as the first time.

[0138] The second time acquisition unit is used to acquire the most recently stored data information in the data storage area, and record the storage time of acquiring the data information as the second time.

[0139] A first information determining unit is configured to use, if the first time is earlier than the second time, information in the index storage area whose storage time is later than the second time as information to be aggregated.

[0140] If the first time is later than or equal to the second time, the information in the index area is used as the information to be aggregated.

[0141] The second information determining unit is configured to aggregate the information to be aggregated into the aggregated information according to a preset aggregation condition.

[0142] In one embodiment, the second information determining unit includes:

[0143] The time period extraction subunit is used to extract the aggregation time period in the preset aggregation condition.

[0144] The aggregation information acquisition subunit is used to sum up the usage space of each piece of information to be aggregated whose storage time belongs to the same aggregation time period as the aggregation information.

[0145] In one embodiment, the method further includes:

[0146] The usage space clearing module is used to clear the usage space information of the aggregation information in the index storage area according to a preset time after the aggregation information is stored in the data storage area.

[0147] In one embodiment, a space cleaning module is used, including:

[0148] The determination unit is used to determine whether the usage space information exceeds a limit value.

[0149] The sorting unit is used to sort the information in the index according to the storage time when the usage space information exceeds the limit value.

[0150] A cleaning unit is used to clean up the data of each of the information according to the sorting and preset time granularity to release space.

[0151] In one embodiment, the sorting unit includes:

[0152] The time acquisition subunit is used to acquire the storage time of the earliest log document stored in each of the indexes from the index storage area.

[0153] The sorting subunit is used to sort the information within the time range of the current moment and the storage time, wherein the current moment is at the hourly time granularity level.

[0154] In one embodiment, the cleaning unit comprises:

[0155] The information division subunit is used to divide the information in the sorting into multiple groups according to the time granularity.

[0156] The cleaning subunit is used to clean up the information belonging to each group in the index storage area in order from the most recent time until the used space information after cleaning is less than the limit value.

[0157] The log management device provided in the embodiment of the present invention can execute the log management method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] Embodiment 5

[0159] In one embodiment, Figure 7 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0160] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0161] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the log management method.

[0163] In some embodiments, the log management method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log management method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log management method in any other appropriate manner (e.g., by means of firmware).

[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0169] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log management method, characterized in that: include: Divide product logs into multiple indexes according to log types; Under each index, the information in the index storage area is grouped into aggregate information according to a preset aggregation condition; Storing the aggregated information in a data storage area; Cleaning up the usage space information of the aggregate information in the index storage area according to a preset time; The step of combining the information in the index storage area into aggregated information according to the preset aggregation conditions under each index includes: Reading the document attributes of the index from the index storage area, searching the document attributes for the earliest log document stored in the index, and recording the storage time of the log document as the first time; Acquire the most recently stored data information in the data storage area, and record the storage time of acquiring the data information as a second time; If the first time is earlier than the second time, the information in the index storage area whose storage time is later than the second time is used as the information to be aggregated; If the first time is later than or equal to the second time, the information in the index storage area is used as the information to be aggregated; aggregating the information to be aggregated into the aggregated information according to a preset aggregation condition; The clearing of the usage space information of the aggregate information in the index storage area according to the preset time includes: Determining whether the used space information exceeds a limit value; In the case where the used space information exceeds a limit value, sorting the information in the index according to the storage time of the log documents in each index; The data of each information is cleaned up according to the sorting and the preset time granularity to release space, until the used space information after cleaning is less than the limit value.

2. The method according to claim 1, characterized in that The aggregating the information to be aggregated into the aggregated information according to a preset aggregation condition includes: Extracting the aggregation time period in the preset aggregation condition; The used spaces of the pieces of information to be aggregated whose storage times belong to the same aggregation time period are summed up as the aggregation information.

3. The method according to claim 1, characterized in that The sorting of the information in the indexes according to the storage time of the log documents in each index includes: Acquire the storage time of the earliest log document stored in each of the indexes from the index storage area; The information within the time range of the current time and the storage time is sorted, wherein the current time is at the hourly time granularity level.

4. The method according to claim 1, characterized in that The cleaning of the data of each information according to the sorting and the preset time granularity to release space until the cleaned used space information is less than the limit value includes: Divide the information in the sorting into multiple groups according to the time granularity; The information belonging to each group in the index storage area is cleaned up in order from the most recent time until the used space information after cleaning is less than the limit value.

5. A log management device, used to implement the log management method according to claim 1, characterized in that: The device comprises: Index partitioning module, used to divide product logs into multiple indexes according to log types; An information composition module, used to compose the information in the index storage area into aggregated information according to preset aggregation conditions under each index; An information storage module, used for storing the aggregate information into a data storage area; The usage space clearing module is used to clear the usage space information of the aggregation information in the index storage area according to a preset time after the aggregation information is stored in the data storage area.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log management method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log management method according to any one of claims 1 to 4 when executed.

Images