Bucket log-based data circulation method and device, electronic equipment and storage medium

Abstract

The invention provides a bucket log-based data circulation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a bucket log of a transaction; analyzing and processing the bucket log to obtain log analysis data written into the first message queue; performing batch processing based on the log analysis data in the first message queue to obtain aggregated data, and writing the aggregated data into a second message queue; and carrying out multi-level data circulation based on the aggregated data in the second message queue. The method comprises the following steps: filtering and analyzing a bucket log to determine a corresponding log analysis result; performing batch processing on batch log analysis results, and determining corresponding aggregated data; determining the hierarchy of data of each piece of data in the aggregated data at the current time; and determining whether data circulation is needed or not according to the hierarchy change of the current data and the hierarchy change of the previous time, and if so, carrying out data circulation. Therefore, the user can quickly access the corresponding data, and the access efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data flow method, device, electronic device and storage medium based on bucket logs. Background Art

[0002] With the popularization and application of technologies such as the Internet, big data, artificial intelligence, and cloud computing, digital knowledge and information have become key production factors.

[0003] Among these explosively growing data, most are unstructured data, such as images, videos, documents, and various types of files, accounting for more than 80%. The amount of unstructured data that needs to be managed is getting bigger and bigger, and the cost of resource maintenance is getting higher and higher. How to reduce data maintenance costs and improve data service levels are the main issues to be solved in data lifecycle management.

[0004] At present, unstructured data is directly stored in various storage areas. Due to the different data usage efficiency, storage in the above manner is prone to data redundancy, increased storage area access pressure, and requires a large amount of storage memory. When storing and accessing large amounts of data, it is easy to trigger performance bottlenecks, resulting in low access efficiency and high storage costs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data flow method, device, electronic device and storage medium based on bucket logs to solve the problems of low access efficiency and high storage cost.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of the embodiment of the present invention shows a data flow method based on a bucket log, the method comprising:

[0008] Get the bucket log of the transaction;

[0009] Analyze and process the bucket log to obtain log analysis data written into the first message queue;

[0010] Batch processing is performed based on the log analysis data in the first message queue to obtain aggregated data, and the aggregated data is written into the second message queue;

[0011] Multi-level data flow is performed based on the aggregated data in the second message queue.

[0012] Optionally, analyzing and processing the bucket log to obtain log analysis data written into the first message queue includes:

[0013] For each bucket log, filter out transaction data of the same type as a preset type from the bucket log;

[0014] Summarizing the transaction data to obtain log analysis data;

[0015] The log analysis data is stored in the first message queue according to a preset format.

[0016] Optionally, batch processing is performed based on the log analysis data in the first message queue to obtain aggregated data, and the aggregated data is written into the second message queue, including:

[0017] If it is determined that there is no data of the deletion operation in the batch of log analysis data, storing the log analysis data;

[0018] If it is determined that the current time of the log analysis data is within the executable time of the data flow, the recorded log analysis data is filtered according to the transaction lifecycle management policy information to obtain aggregated data;

[0019] The aggregated data is written into the second message queue according to the second format.

[0020] Optionally, the recorded log analysis data is filtered according to the transaction lifecycle management policy information to obtain the filtered log analysis data, including:

[0021] Filtering all the recorded log analysis data according to the time threshold in the transaction lifecycle management policy information to obtain the log analysis data after the first filtering;

[0022] Based on the log analysis data after the first screening, the number of visits to each transaction object is counted;

[0023] The log analysis data after the first screening is screened according to the number of visits of each transaction object to obtain aggregated data.

[0024] Optionally, also include:

[0025] Obtaining batches of log analysis data in the first message queue at preset time intervals;

[0026] If it is determined that data of the deletion operation exists in the batch of log analysis data, the corresponding log analysis data is deleted according to the deletion operation.

[0027] Optionally, transferring multi-level data based on the aggregated data in the second message queue includes:

[0028] Reading the aggregated data in the second message queue;

[0029] For each data in the aggregated data, traverse all data storage areas based on the aggregated data to determine whether data transfer is required;

[0030] If necessary, determining the original data storage area and the target data storage area where the data transfer is required;

[0031] The original data corresponding to the data in the original data storage area is copied, and the original data and the data are transferred to the target storage area.

[0032] Optionally, after transferring the original data and the data stream to the target storage area, the method further includes:

[0033] Delete the original data in the original data storage area.

[0034] The second aspect of the embodiment of the present invention shows a data flow device based on a bucket log, the device comprising:

[0035] A data analysis unit, used to obtain a bucket log of a transaction, analyze and process the bucket log, and obtain log analysis data written into the first message queue;

[0036] A data management unit, configured to perform batch processing based on the log analysis data in the first message queue, obtain aggregated data, and write the aggregated data into the second message queue;

[0037] A data flow unit is used to flow multi-level data based on the aggregated data in the second message queue.

[0038] A third aspect of an embodiment of the present invention shows an electronic device, which is used to run a program, wherein when the program is run, the data flow method based on the bucket log as shown in the first aspect of the embodiment of the present invention is executed.

[0039] The fourth aspect of an embodiment of the present invention shows a storage medium, which includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data flow method based on bucket logs as shown in the first aspect of the embodiment of the present invention.

[0040] Based on the above-mentioned embodiment of the present invention, a data flow method, device, electronic device and storage medium based on bucket logs are provided, and the bucket logs of the transaction are obtained; the bucket logs are analyzed and processed to obtain log analysis data written into the first message queue; based on the log analysis data in the first message queue, batch processing is performed to obtain aggregated data, and the aggregated data is written into the second message queue; based on the aggregated data in the second message queue, multi-level data flow is performed. In the embodiment of the present invention, the bucket logs are first obtained, and the bucket logs are first filtered and parsed to determine the corresponding log analysis results; then the batch log analysis results are batch processed according to the lifecycle management strategy to determine the corresponding aggregated data; then the data level of the aggregated data is determined, that is, what level of data each data is at the current time, such as hot data, warm data and cold data; according to the level of the current data and the level change of the previous time, it is determined whether data flow is required, and if necessary, data flow is performed. So that users can quickly access the corresponding data, thereby improving access efficiency and reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 A data flow device based on bucket logs is shown in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of data flow of bucket logs by various units shown in an embodiment of the present invention;

[0044] Figure 3 A flowchart of a data flow method based on bucket logs according to an embodiment of the present invention is shown;

[0045] Figure 4 A data analysis flow chart of a bucket log shown in an embodiment of the present invention;

[0046] Figure 5 A data management flow chart of a bucket log according to an embodiment of the present invention;

[0047] Figure 6 This is a data flow chart of a bucket log according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0050] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0051] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0052] In order to better understand the present application, the terms shown in the present application are explained below.

[0053] Object Storage Service (OSS), also known as object-based storage, is a method of solving and processing discrete units, which can provide data storage services in the form of objects based on distributed systems. The core of object storage is a flat namespace, which is addressed in a simple key-value manner. Taking Amazon S3 as an example, object storage only has two levels: bucket (equivalent to directory, folder) and object (equivalent to file). When using object storage service, you must first establish a tenant and create a data bucket under the tenant to store and manage objects; each bucket or object has a globally unique ID, and each object is stored in a bucket. Users can access the object's data entity (Data) and metadata (Metadata) by using the bucket ID + object ID.

[0054] Bucket logs are log files that record detailed information about all request operations in a bucket, including the source IP address of the request, the time of the request, the type of request, the target object of the request, the HTTP status code of the request, etc.

[0055] Unstructured data is data that is not convenient to express using two-dimensional logical tables, relative to structured data. It includes office documents of all formats, texts, pictures, various reports, images, audio, video, etc.

[0056] Hot data Online data should be accessible instantly. It is the main stage of data generation and use. The storage device and the stored data always maintain the corresponding state and can be read and written in real time with high frequency, which has high requirements for storage performance.

[0057] Warm data is located between online and offline, and has relatively low storage performance requirements, but can provide relatively good reading performance and can be queried online in real time through the system.

[0058] Cold data has lower storage performance requirements but requires large-capacity storage and is mainly used in data backup scenarios.

[0059] See also Figure 1 , is a data flow device based on bucket logs shown in an embodiment of the present invention, the device includes:

[0060] Data analysis unit 101, data management unit 102, data flow unit 103, object storage cluster 104 and message queue 105;

[0061] The data analysis unit 101 is connected to the object storage cluster 104 and the message queue 105 respectively, and the data analysis unit 101 is connected to the data management unit 102 and the data flow unit 103 through the message queue;

[0062] The data transfer unit 103 is connected to the object storage cluster 104 , that is, all storage areas are set in the object storage cluster.

[0063] The data analysis unit 102 includes a collector 11;

[0064] The data analysis unit 101 is used to obtain the bucket log of the transaction, analyze and process the bucket log, and obtain the log analysis data written into the first message queue;

[0065] Specifically, the collector 11 obtains the bucket log file screening date T from the policy configuration information; and lists the bucket logs of transactions before the date T from the object storage cluster 104 for acquisition.

[0066] Among them, bucket log files of multiple transactions can be generated in one day, that is, the number of bucket logs obtained is multiple.

[0067] It should be noted that the policy configuration information is pre-set by the technical staff in the system, including the screening date T of each bucket log file.

[0068] The object storage cluster stores bucket logs during transactions in the banking system.

[0069] Bucket logs are log files generated according to specified rules, namely, the source IP address of the transaction request, the time of the request, the type of request, the target object of the request, the HTTP status code of the request, and other data. In other words, bucket logs include log files of multiple transactions.

[0070] The data management unit 102 is used to perform batch processing based on the log analysis data in the first message queue, obtain aggregated data, and write it into the second message queue;

[0071] The data transfer unit 103 is used to transfer multi-level data based on the aggregated data in the second message queue.

[0072] In an embodiment of the present invention, a data analysis unit is used to realize the functions of collecting, filtering, and parsing transaction bucket log data. The data management unit is used to store and manage the log analysis result data after log analysis. Then the data flow unit is used to move data from hot data to warm data, and from warm data to cold data, so as to effectively save storage costs while meeting the access performance of hot data, warm data, and cold data at all levels. To achieve the goal of "one piece of data, orderly flow, use on demand, and a beginning and an end", the data is classified and layered, and the hierarchical storage and use of data are realized, forming a closed-loop management of the data life cycle, reducing data redundancy, reducing storage and operating costs, and improving data service quality.

[0073] Optional, continue to see Figure 1The data analysis unit 102 further includes a filter 12 and a parser 13. Based on the above-mentioned device, the data analysis unit 101 analyzes and processes the bucket log to obtain log analysis data written into the first message queue, which is specifically used for:

[0074] For each bucket log, transaction data of the same type as a preset type is filtered out from the bucket log; the transaction data is aggregated to obtain log analysis data; and the log analysis data is stored in a first message queue according to a preset format.

[0075] It should be noted that the preset type refers to a type that is set according to actual conditions, and generally may be a transaction and transaction information of an operation type such as upload, download, or delete.

[0076] The transaction data includes transactions of preset types and transaction information.

[0077] Specifically, in the transaction records read from the bucket log file, filter 12 is used to filter out upload, download, and delete transactions, and other log records such as system logs are filtered out;

[0078] Next, in the filtered valid transaction records, the parser 13 is used to parse out the transaction information such as the bucket number, transaction time, operation type, object name information, etc. of each transaction. The transaction information such as the bucket number, transaction time, operation type, object name information, etc. of each transaction and the transaction data are aggregated according to the object name of each data, and the operations on the same object data in the transaction are merged, and only the record information of the last operation on the same object is retained to obtain the log analysis data. Then the aggregated log analysis result data is saved in the first message queue Q1 in the format of "object name|bucket number|operation type|transaction time".

[0079] It should be noted that the first message queue Q1 belongs to the message queue 105 .

[0080] Optionally, it also includes determining whether all bucket logs have been read. If so, it is determined that the data parsing of the bucket log is completed. If not, it returns to reading the transaction bucket log that has not been parsed.

[0081] In the embodiment of the present invention, the data analysis unit shown is used to realize the collection, filtering and parsing functions of the transaction bucket log data, and the parsed data is aggregated and stored in the first message queue so that the subsequent data management unit can obtain and process it.

[0082] Optional, continue to see Figure 1 The data management unit 102 includes a batch adding subunit 21 , a batch aggregation subunit 22 and a batch deleting subunit 23 .

[0083] The data management unit 102 performs batch processing based on the log analysis data in the first message queue to obtain aggregated data and writes the aggregated data into the second message queue, which is specifically used for:

[0084] If it is determined that there is no data of the deletion operation in the batch of log analysis data, storing the log analysis data;

[0085] If it is determined that the current time of the log analysis data is within the executable time of the data flow, the recorded log analysis data is filtered according to the transaction lifecycle management policy information to obtain aggregated data;

[0086] The aggregated data is written into the second message queue according to the second format.

[0087] The recorded log analysis data is filtered according to the transaction lifecycle management policy information to obtain the filtered log analysis data, including:

[0088] Filtering all the recorded log analysis data according to the time threshold in the transaction lifecycle management policy information to obtain the log analysis data after the first filtering;

[0089] Based on the log analysis data after the first screening, the number of visits to each transaction object is counted;

[0090] The log analysis data after the first screening is screened according to the number of visits of each transaction object to obtain aggregated data.

[0091] Specifically, all log analysis data whose operation types are not deletion operations are stored. The batch addition subunit 21 will write all log analysis data whose operation types are not deletion operations to perform batch additions for subsequent data management. If the batch aggregation subunit 22 determines that the current time of the log analysis data is within the executable time of the data flow, it obtains the pre-set transaction lifecycle management policy information from the data analysis unit, and parses the time threshold and access frequency threshold of the transaction lifecycle management policy information; then, all the recorded log analysis data are filtered according to the time threshold, and the log analysis data whose current time is less than the time threshold is filtered out, and then the log analysis data whose current time is less than the time threshold is batch aggregated to obtain the log analysis data after the first screening; data summary is performed for each aggregated log analysis data to count the number of visits to each transaction object;

[0092] Divide the log analysis data after the first screening according to the number of visits to each transaction object, and determine whether there is an access number greater than the first access frequency threshold. If so, record the log analysis data corresponding to the access number, that is, write it into the first set; write the log analysis data corresponding to the access number that is less than or equal to the first access frequency threshold and greater than or equal to the second access frequency threshold into the second set; write the log analysis data corresponding to the access number that is less than the second access frequency threshold into the third set; aggregate the three sets obtained to obtain aggregated data.

[0093] The record data satisfying the conditions, ie, the aggregated data, is stored in the second message queue Q2 in the format of "object name|bucket number".

[0094] It should be noted that the time threshold here refers to a certain time within the period from when the transaction is generated to when it is deleted. The time threshold is set after multiple experiments.

[0095] The access frequency threshold refers to the number of visits that a transaction needs to meet within the time threshold, which is set after multiple experiments.

[0096] Optionally, based on the above-mentioned device, the data management unit 102 is further configured to:

[0097] Obtaining batches of log analysis data in the first message queue at preset time intervals;

[0098] If it is determined that data of the deletion operation exists in the batch of log analysis data, the corresponding log analysis data is deleted according to the deletion operation.

[0099] Specifically, the batch deletion subunit 23 traverses the batch log analysis data in sequence, determines whether there is an operation type of deletion operation in the log analysis data, and if so, aggregates the log analysis data corresponding to the objects whose operation type is deletion operation, and performs batch deletion.

[0100] Optional, continue to see Figure 2 The data transfer unit includes a reading subunit 31, a copying subunit 32, and a cleaning subunit 33.

[0101] The data transfer unit 103 is specifically used to: read the aggregated data in the second message queue;

[0102] For each data in the aggregated data, traverse all data storage areas based on the aggregated data to determine whether data transfer is required;

[0103] If necessary, determining the original data storage area and the target data storage area where the data transfer is required;

[0104] The original data corresponding to the data in the original data storage area is copied, and the original data and the data are transferred to the target storage area.

[0105] Specifically, the reading subunit 31 reads the aggregated data written by the data management unit into the second message queue. The replication subunit 32 determines, according to the set in the aggregated data, what level of data each data is at the current time; if the data is in the first set, it is determined that the data is currently hot data; if the data is in the second set, it is determined that the data is currently warm data; if the data is in the third set, it is determined that the data is currently cold data.

[0106] Next, taking the determination that the data is currently hot data as an example, traverse all data storage areas to determine whether there is data with the same bucket ID or object ID as the data; if it is determined that there is data with the same bucket ID or object ID as the data in the hot data storage area (a storage area at the same data level), the data is directly written into the hot data storage area so that it can be aggregated with the previous data; if no corresponding data exists in all storage areas, the set of hot data is written into the hot data storage area.

[0107] Similarly, if other data has been stored in a storage area at the same level as the data, the above process can also be referred to.

[0108] It should be noted that the hot data storage area is only used to store hot data, generally memory.

[0109] Taking determining that the data is currently warm data as an example, traverse all data storage areas to determine whether there is data with the same bucket ID or object ID as the data; if it is determined that the hot data storage area, or the cold data storage area (a storage area different from the data level) has data with the same bucket ID or object ID as the data, determine that data transfer is required; then use the hot data storage area, or the cold data storage area, where there is data with the same bucket ID or object ID as the data, as the original data storage area, and use the storage area corresponding to the data level, which is the warm data storage area at this time, as the target storage area.

[0110] Similarly, if other data has been stored in a storage area different from the data level, the above process can also be referred to.

[0111] Then, the original data in the original data storage area having the same bucket ID or object ID as the data is copied; and the original data and the data are written into the target storage area to realize data transfer.

[0112] The data transfer unit 103 is further configured to: after transferring the original data and the data to the target storage area, further include:

[0113] Delete the original data in the original data storage area.

[0114] Specifically, the cleaning subunit 33 determines whether the object data, that is, the original data, is copied successfully. If the copy is successful, the original data in the original data storage area, that is, the original object data, is deleted. If the copy fails, the failed data information is recorded.

[0115] When deleting the original data in the original data storage area, determine whether the original data, that is, the object data, is deleted successfully. If the deletion is successful, traverse the next data of the aggregated data in the second message queue. If the deletion is unsuccessful, record the failed data information.

[0116] Optionally, based on the data analysis unit, data management unit and data transfer unit shown in the present invention, the process of specifically implementing the data transfer of object storage can be as follows: Figure 2 shown.

[0117] In the embodiment of the present invention, first obtain the bucket log, filter and parse the bucket log to determine the corresponding log analysis result; then batch process the batch log analysis results according to the life cycle management strategy to determine the corresponding aggregated data; then determine the data level of the aggregated data, that is, determine the data level of each data at the current time, such as hot data, warm data and cold data; determine whether data transfer is required according to the level of the current data and the level change at the previous time, and if necessary, determine the original data storage area and the target data storage area where the data transfer is required; copy the original data corresponding to the data in the original data storage area, and transfer the original data and the data to the target storage area, so as to realize data transfer. So that users can quickly access the corresponding data, can realize the storage and access of large amounts of data, thereby improving the efficiency of access and reducing storage costs.

[0118] See also Figure 3 , is a method for automatically transferring object storage data based on bucket logs according to an embodiment of the present invention, the method comprising:

[0119] Step S301: Obtain the bucket log of the transaction;

[0120] In the specific implementation of step S301 , the bucket log file screening date T is obtained from the policy configuration information; and the bucket logs of transactions before the date T are listed from the object storage cluster for acquisition.

[0121] Among them, bucket log files of multiple transactions can be generated in one day, that is, the number of bucket logs obtained is multiple.

[0122] It should be noted that the policy configuration information is pre-set by the technical staff in the system, including the screening date T of each bucket log file.

[0123] The object storage cluster stores bucket logs during transactions in the banking system. That is, the object storage cluster converts unstructured data into object storage for storage in the object storage cluster.

[0124] Bucket logs are log files generated according to specified rules, namely, the source IP address of the transaction request, the time of the request, the type of request, the target object of the request, the HTTP status code of the request, and other data. In other words, bucket logs include log files of multiple transactions.

[0125] Step S302: Analyze and process the bucket log to obtain log analysis data written into the first message queue.

[0126] The specific implementation of step S302 includes the following steps:

[0127] Step S11: for each bucket log, filter out transaction data of the same type as a preset type from the bucket log;

[0128] It should be noted that the preset type refers to a type that is set according to actual conditions, and generally may be a transaction and transaction information of an operation type such as upload, download, or delete.

[0129] The transaction data includes transactions of preset types and transaction information.

[0130] In the specific implementation of step S11, in the transaction records read from the bucket log file, upload, download, and delete transactions are filtered out, and other log records such as system log records are filtered out;

[0131] Next, from the filtered valid transaction records, the transaction information such as the bucket number, transaction time, operation type, object name information, etc. of each transaction is parsed.

[0132] Step S12: Summarize the transaction data to obtain log analysis data.

[0133] In the process of implementing step S12, the transaction information such as the bucket number, transaction time, operation type, object name information of each transaction and the transaction data are summarized according to the object name of each data, and the operations on the same object data in the transaction are merged. Only the record information of the last operation on the same object is retained to obtain log analysis data and then the summarized log analysis result data is obtained.

[0134] Step S13: storing the log analysis data in the first message queue according to a preset format.

[0135] It should be noted that the preset format is set in advance according to actual conditions.

[0136] In the specific implementation of step S13, the message is saved in the first message queue Q1 in the format of "object name|bucket number|operation type|transaction time".

[0137] Optionally, after executing step S13, it is further included to determine whether all bucket logs have been read. If so, it is determined that the data parsing of the bucket log is completed. If not, it returns to read the transaction bucket log that has not been parsed.

[0138] The workflow based on the data analysis shown above can be as follows Figure 4 shown.

[0139] Step S303: Batch processing is performed based on the log analysis data in the first message queue to obtain aggregated data, and the aggregated data is written into the second message queue.

[0140] It should be noted that the specific process of implementing step S303 includes the following steps:

[0141] Step S21: obtaining batches of log analysis data in the first message queue at preset time intervals.

[0142] In the specific implementation of step S21 , the batch data stored in the message queue Q1 by the data analysis unit is obtained, that is, the log analysis data within a preset time interval.

[0143] It should be noted that the preset time interval is set in advance by the technicians based on multiple experiments. Generally speaking, all the log analysis data stored in the first message queue can be obtained within each time interval.

[0144] Step S22: traverse the batch of log analysis data in sequence to determine whether there is data for deletion operation, if yes, execute step S23, if no, execute step S24;

[0145] In the specific implementation of step S22, the batch of log analysis data is traversed in sequence to determine whether there is an operation type of a delete operation in the log analysis data. If yes, step S23 is executed; if not, step S24 is executed.

[0146] Step S23: deleting the corresponding log analysis data according to the deletion operation.

[0147] In the specific implementation of step S23, the log analysis data corresponding to the objects whose operation type is the deletion operation are aggregated and deleted in batches.

[0148] Optionally, traverse the next log analysis data in the batch of log analysis data until all the log analysis data within the preset time interval are processed, that is, determine whether the first message queue Q1 is currently empty, if not empty, return to execute step S22, if empty, end the process;

[0149] Step S24: storing the log analysis data;

[0150] In the specific implementation of step S24, all log analysis data whose operation type is not a delete operation are stored. The batch addition subunit in the data management unit writes all log analysis data whose operation type is not a delete operation to perform batch addition for subsequent data management.

[0151] Step S25: Determine whether the current time of the log analysis data is within the data flow executable time. If so, execute step S26; if not, manage the log analysis data.

[0152] In the process of implementing step S25, the current time of each log analysis data is obtained, and it is determined whether the current time is within the executable time of the data flow. If so, step S26 is executed, and if not, the log analysis data is managed.

[0153] It should be noted that the current time of the log analysis data refers to the time from when the log was generated to the present.

[0154] Step S26: Screening the recorded log analysis data according to the transaction lifecycle management policy information to obtain aggregated data;

[0155] The filtered log analysis data includes the log analysis data after the first filtering and the aggregated data.

[0156] The specific implementation of step S26 includes the following steps:

[0157] Step S31: screening all the recorded log analysis data according to the time threshold in the transaction lifecycle management policy information to obtain the log analysis data after the first screening;

[0158] In the specific implementation of step S31, the pre-set transaction lifecycle management policy information is obtained from the data analysis unit, and the time threshold and access frequency threshold of the transaction lifecycle management policy information are parsed; then, all the recorded log analysis data are screened according to the time threshold, and the log analysis data whose current time is less than the time threshold is screened out, and then the log analysis data whose current time is less than the time threshold is batch aggregated to obtain the log analysis data after the first screening;

[0159] It should be noted that the time threshold here refers to a certain time within the period from when the transaction is generated to when it is deleted. The time threshold is set after multiple experiments.

[0160] The access frequency threshold refers to the number of visits that a transaction needs to meet within the time threshold, which is set after multiple experiments.

[0161] Step S32: Processing is performed based on the log analysis data after the first screening to count the number of visits to each transaction object.

[0162] In the specific implementation of step S32, data aggregation is performed for each aggregated log analysis data to count the number of visits to each transaction object;

[0163] Step S33: filtering the log analysis data after the first screening according to the number of visits of each transaction object to obtain aggregated data.

[0164] In the process of implementing step S33, the log analysis data after the first screening is divided according to the number of visits to each transaction object, and it is determined whether there is an access number greater than the first access frequency threshold. If so, the log analysis data corresponding to the access number is recorded, that is, written into the first set; the log analysis data corresponding to the access number that is less than or equal to the first access frequency threshold and greater than or equal to the second access frequency threshold is written into the second set; the log analysis data corresponding to the access number that is less than the second access frequency threshold is written into the third set; the three sets obtained are aggregated to obtain aggregated data.

[0165] Step S27: writing the aggregated data into the second message queue according to the second format.

[0166] In the specific implementation of step S27, the record data that meets the conditions, that is, the aggregated data, is stored in the second message queue Q2 in the format of "object name|bucket number".

[0167] The workflow based on the data management shown above can be as follows Figure 5 shown.

[0168] The data flow unit is used to move object data from hot data to warm data and from warm data to cold data, read aggregate data, copy object storage data, and clean up object storage data combinations.

[0169] Step S304: Flow multi-level data based on the aggregated data in the second message queue.

[0170] The specific implementation of step S304 includes the following steps.

[0171] Step S41: Read the aggregated data in the second message queue.

[0172] In the specific implementation of step S41, the data written into the second message queue by the data management unit is read.

[0173] Step S42: For each data in the aggregated data, traverse all data storage areas based on the aggregated data to determine whether data flow is required. If necessary, execute step S43. If not, do not perform any operation, or continue to process the next data in the aggregated data.

[0174] Step S43: determining the original data storage area and the target data storage area where the data transfer needs to be performed;

[0175] Step S44: copying the original data corresponding to the data in the original data storage area, and transferring the original data and the data to the target storage area.

[0176] Optionally, after executing step S44, the process further includes: simultaneously deleting the original data in the original data storage area.

[0177] The workflow based on the data flow of the bucket log shown above can be as follows: Figure 6 shown.

[0178] Among them, all data storage areas include hot data storage areas, warm data storage areas and cold data storage areas.

[0179] In the process of implementing steps S42 to S44, determine what level each data is at the current time according to the set in the aggregated data; if the data is in the first set, determine that the data is currently hot data; if the data is in the second set, determine that the data is currently warm data; if the data is in the third set, determine that the data is currently cold data.

[0180] Next, taking the determination that the data is currently hot data as an example, traverse all data storage areas to determine whether there is data with the same bucket ID or object ID as the data; if it is determined that there is data with the same bucket ID or object ID as the data in the hot data storage area (a storage area at the same data level), the data is directly written into the hot data storage area so that it can be aggregated with the previous data; if no corresponding data exists in all storage areas, the set of hot data is written into the hot data storage area.

[0181] Similarly, if other data has been stored in a storage area at the same level as the data, the above process can also be referred to.

[0182] It should be noted that the hot data storage area is only used to store hot data, generally memory.

[0183] Taking determining that the data is currently warm data as an example, traverse all data storage areas to determine whether there is data with the same bucket ID or object ID as the data; if it is determined that the hot data storage area, or the cold data storage area (a storage area different from the data level) has data with the same bucket ID or object ID as the data, determine that data transfer is required; then use the hot data storage area, or the cold data storage area, where there is data with the same bucket ID or object ID as the data, as the original data storage area, and use the storage area corresponding to the data level, which is the warm data storage area at this time, as the target storage area.

[0184] Similarly, if other data has been stored in a storage area different from the data level, the above process can also be referred to.

[0185] Then, the original data in the original data storage area having the same bucket ID or object ID as the data is copied; and the original data and the data are written into the target storage area to realize data transfer; then, it is determined whether the object data, i.e., the original data, is copied successfully; if the copy is successful, the original data in the original data storage area, i.e., the original object data, is deleted; if the copy fails, the failed data information is recorded;

[0186] When deleting the original data in the original data storage area, determine whether the original data, that is, the object data, is deleted successfully. If the deletion is successful, traverse the next data of the aggregated data in the second message queue. If the deletion is unsuccessful, record the failed data information.

[0187] In an embodiment of the present invention, by analyzing the transaction bucket log, aggregating and recording the object access situation, it is possible to achieve decoupling of transaction statistics from the actual transaction link, while reducing additional access to business objects, reducing access pressure on the object storage cluster, and avoiding impact on actual online business; the present invention uses a lifecycle management strategy based on the setting to aggregate result data from the access record of the data management unit, which can achieve decoupling from the actual stored business objects when the business object access frequency is compared with the threshold, while reducing additional access to business objects, reducing access pressure on the object storage cluster, and avoiding impact on actual online business.

[0188] In the embodiment of the present invention, firstly obtain the bucket log, filter and parse the bucket log to determine the corresponding log analysis result; then batch process the batch log analysis results according to the life cycle management strategy to determine the corresponding aggregated data; then determine the data level of the aggregated data, that is, determine the data level of each data at the current time, such as hot data, warm data and cold data; determine whether data transfer is required according to the level of the current data and the level change at the previous time, and if necessary, determine the original data storage area and the target data storage area where the data transfer is required; copy the original data corresponding to the data in the original data storage area, and transfer the original data and the data to the target storage area, so as to realize data transfer. So that users can quickly access the corresponding data, thereby improving the efficiency of access.

[0189] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0190] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0191] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data flow method based on bucket logs, characterized in that: The method comprises: Get the bucket log of the transaction; Analyze and process the bucket log to obtain log analysis data written into the first message queue; Batch processing is performed based on the log analysis data in the first message queue to obtain aggregated data, and the aggregated data is written into the second message queue; Multi-level data flow is performed based on the aggregated data in the second message queue.

2. The method according to claim 1, characterized in that The bucket log is analyzed and processed to obtain log analysis data written into the first message queue, including: For each bucket log, filter out transaction data of the same type as a preset type from the bucket log; Summarizing the transaction data to obtain log analysis data; The log analysis data is stored in the first message queue according to a preset format.

3. The method according to claim 1, characterized in that Batch processing is performed based on the log analysis data in the first message queue to obtain aggregated data, and the aggregated data is written into the second message queue, including: If it is determined that there is no data of the deletion operation in the batch of log analysis data, storing the log analysis data; If it is determined that the current time of the log analysis data is within the executable time of the data flow, the recorded log analysis data is filtered according to the transaction lifecycle management policy information to obtain aggregated data; The aggregated data is written into the second message queue according to the second format.

4. The method according to claim 3, characterized in that The recorded log analysis data is screened according to the transaction lifecycle management policy information to obtain screened log analysis data, including: Filter all recorded log analysis data according to the time threshold in the transaction lifecycle management policy information to obtain log analysis data after the first filtering; Based on the log analysis data after the first screening, the number of visits to each transaction object is counted; The log analysis data after the first screening is screened according to the number of visits of each transaction object to obtain aggregated data.

5. The method according to claim 1, characterized in that Also includes: Obtaining batches of log analysis data in the first message queue at preset time intervals; If it is determined that data of the deletion operation exists in the batch of log analysis data, the corresponding log analysis data is deleted according to the deletion operation.

6. The method according to claim 1, characterized in that The multi-level data flow is performed based on the aggregated data in the second message queue, including: Reading the aggregated data in the second message queue; For each data in the aggregated data, traverse all data storage areas based on the aggregated data to determine whether data transfer is required; If necessary, determining the original data storage area and the target data storage area where the data transfer is required; The original data corresponding to the data in the original data storage area is copied, and the original data and the data are transferred to the target storage area.

7. The method according to claim 6, characterized in that After transferring the original data and the data stream to the target storage area, the method further includes: Delete the original data in the original data storage area.

8. A data flow device based on bucket logs, characterized in that: The device comprises: A data analysis unit, used to obtain a bucket log of a transaction, analyze and process the bucket log, and obtain log analysis data written into the first message queue; A data management unit, configured to perform batch processing based on the log analysis data in the first message queue, obtain aggregated data, and write the aggregated data into the second message queue; A data flow unit is used to flow multi-level data based on the aggregated data in the second message queue.

9. An electronic device, characterized in that: The electronic device is used to run a program, wherein the program, when running, executes the data flow method based on the bucket log as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium includes a storage program, wherein when the program is running, the device where the storage medium is located is controlled to execute the data flow method based on the bucket log according to any one of claims 1 to 7.

Images