Operation log processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to an operation log processing method and device, electronic equipment and a storage medium. According to the invention, after the user is confirmed to have the access authority, the corresponding business processing flow is acquired and executed based on the business parameters, so that the orderliness and pertinence of business processing are ensured. In the process of executing the business processing flow, the operation logs generated by the source barrel related to the flow are collected, and the operation logs are stored in the target barrel, so that the deficiency of the original CEPH object storage engine in the barrel and object operation log unloading function is filled up, the operation logs can be effectively collected and properly stored, and the service processing efficiency is improved. Therefore, a data basis is provided for meeting the requirements of the user for quick and real-time query and multi-dimensional statistical analysis of massive operation logs, and the problem that the original system is difficult to meet the corresponding requirements due to lack of a transfer function is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for processing an operation log. Background Art

[0002] The rapid development of cloud computing, big data and artificial intelligence technologies has promoted the digital transformation of various industries, resulting in a substantial increase in the amount of enterprise data. Object storage has become increasingly important in enterprise storage architecture due to its advantages such as scalability and ease of use. Application scenarios and storage scale are constantly expanding. Enterprises now pay more attention to exploring the deep value of their stored data, and real-time query and analysis based on operation logs have become a rigid demand.

[0003] The massive amount of operation logs makes it very difficult to quickly query key information in real time and build a multi-dimensional statistical analysis framework. In addition, the native CEPH object storage engine lacks a transfer mechanism for bucket and object operation logs, which seriously hinders the subsequent effective use and further development of operation log resources. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device, electronic device and storage medium for processing operation logs to solve the problem that the native CEPH object storage engine lacks bucket and object operation log transfer functions, making it difficult to meet users' needs for quick real-time query and multi-dimensional statistical analysis of massive operation logs.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing an operation log, the method comprising:

[0006] Receive at least one service request sent by a client to a gateway, wherein the service request carries user information of a user corresponding to the client, a target bucket requested to be accessed by the user, and service parameters;

[0007] Using the user information to verify whether the user has access rights to the target bucket;

[0008] If it is confirmed that the user has access rights to the target bucket, a corresponding business process flow is obtained based on the business parameters, and the business process flow is executed;

[0009] In the process of executing the business processing flow, the operation logs generated by the source bucket involved in the business processing flow are collected, and the operation logs are stored in the target bucket according to preset rules.

[0010] Furthermore, storing the operation log into a target bucket according to a preset rule includes:

[0011] The operation log is stored in a data table in the log management subsystem in the form of a key-value pair, wherein the key name of the key-value pair is generated according to the associated parameters of the operation log, and the key value of the key-value pair is generated according to the operation content of the operation log;

[0012] Reading relevant log data of the operation log from the data table in the log management subsystem according to a preset time period;

[0013] An archive file is constructed using the relevant log data, and the archive file is stored in the target bucket.

[0014] Furthermore, storing the operation log in the form of key-value pairs in a data table in the log management subsystem includes:

[0015] Obtaining associated parameters of the operation log, and generating a key name based on the associated parameters;

[0016] Generate a key value based on the log content of the operation log;

[0017] The key name and the key value corresponding to the operation log are stored in a data table in a log management subsystem.

[0018] Further, the obtaining of the associated parameters of the operation log and generating a key name based on the associated parameters includes:

[0019] Obtain the object associated with the operation log and the timestamp corresponding to the operation log;

[0020] Calculate a hash value based on the bucket name of the target bucket and the object name of the object, and perform modulo operation on the hash value to obtain a target value;

[0021] The key name is generated based on the target value, a bucket name of the target bucket, an object name of the object, and the timestamp.

[0022] Further, the use of the relevant log data to construct an archive file includes:

[0023] Generate file content using the relevant log data and obtain a file name generated according to a file generation rule;

[0024] The archive file is constructed based on the file contents and the file name.

[0025] Further, obtaining the file name generated according to the file generation rule includes:

[0026] Get the preset file name prefix and file identification code;

[0027] The file name is generated based on the preset file name prefix, the bucket name of the source bucket, the timestamp corresponding to the operation log, and the file identification code.

[0028] Furthermore, after reading relevant log data of the operation log from the data table in the log management subsystem according to a preset time period, the method further includes:

[0029] Classify the relevant log data according to buckets to obtain log data corresponding to each bucket;

[0030] Generate the corresponding data table using the log data corresponding to each bucket;

[0031] The data table corresponding to each bucket is stored in the log analysis subsystem.

[0032] In a second aspect, an embodiment of the present invention provides a device for processing an operation log, the device comprising:

[0033] A receiving module, configured to receive at least one service request sent by a client to a gateway, wherein the service request carries user information of a user corresponding to the client, a target bucket requested to be accessed by the user, and service parameters;

[0034] A verification module, used to verify whether the user has access rights to the target bucket using the user information;

[0035] An execution module, configured to obtain a corresponding business process flow based on the business parameters and execute the business process flow if it is confirmed that the user has access rights to access the target bucket;

[0036] The processing module is used to collect the operation logs generated by the source bucket involved in the business processing process during the execution of the business processing process, and store the operation logs in the target bucket according to preset rules.

[0037] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0039] After confirming that the user has access rights, this application obtains and executes the corresponding business processing flow based on business parameters, ensuring the orderliness and pertinence of business processing. During the execution of the business processing flow, the operation logs generated by the source bucket involved in the process are collected, and these operation logs are stored in the target bucket, thereby filling the gap in the native CEPH object storage engine's bucket and object operation log transfer function, allowing operation logs to be effectively collected and properly stored, thereby providing a data basis for meeting users' needs for quick real-time query of massive operation logs and multi-dimensional statistical analysis, and effectively solving the problem that the native system is difficult to meet the corresponding needs due to the lack of transfer function. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is a flowchart of a method for processing an operation log according to some embodiments of the present invention;

[0042] Figure 2 is a schematic diagram of a processing framework of an operation log according to some embodiments of the present invention;

[0043] Figure 3 is a structural block diagram of an operation log processing device according to an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] According to an embodiment of the present invention, a method, device, electronic device and storage medium for processing an operation log are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] In this embodiment, a method for processing an operation log is provided. Figure 1 is a flow chart of a method for processing an operation log according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0048] Step 101: Receive at least one service request sent by a client to a gateway, wherein the service request carries user information of a user corresponding to the client, a target bucket requested to be accessed by the user, and service parameters.

[0049] In the embodiments of the present application, Figure 2 As shown, the present application involves an object gateway, a real-time log management subsystem, a log analysis subsystem and an object storage engine. Among them, the object gateway embeds a log transfer module in the business process, which can provide an interface to the user to enable the bucket operation log transfer function. Its business process includes receiving the user's HTTP request, performing authentication verification of the user access key, then verifying the user and bucket permissions, and then selecting the corresponding business processing flow based on the request parameters. During the processing, the operation log will be collected and stored in the real-time log management subsystem based on the distributed KV implementation, and finally the response will be returned to the user. The real-time log management subsystem is responsible for the storage and query of the operation log. The stored logs can be entered into the log analysis subsystem implemented by the OLAP column database based on the MPP architecture through the warehousing operation for data analysis, while the object storage engine is used to store object data. When the logs in the real-time log management subsystem meet certain conditions, they will be archived and stored in the object storage engine. The entire solution forms a complete system from operation log collection, storage, analysis to object data storage.

[0050] First, the object gateway needs to provide an interface for users to enable the operation log dump function for the bucket. The parameters to be set include the following key elements:

[0051] Serial number Parameter name Parameter Description 1 Target Bucket The destination bucket to be archived 2 Log File Prefix To archive log file prefix 3 Log APE nable Whether to enable the log analysis function 4 Log APTTL Analysis log storage period

[0052] Table 1-Log transfer operation parameters

[0053] Specifically, the gateway starts the network monitoring service and monitors the network connection request from the client through the configured network port (such as the common port 80 for HTTP requests, etc.). When the client initiates a service request, the gateway uses the network protocol stack (such as the TCP / IP protocol stack, etc.) to receive the data packets sent by the client, assembles these data packets according to the relevant protocol specifications such as HTTP, and restores the complete service request content.

[0054] Parse the received business request according to the established format (such as the message header and body format of the HTTP request, etc.). Extract the user information of the corresponding user of the client from the specific field position of the request, which may be obtained from the field specifically storing the user identification in the request header, or the user name, user ID and other information may be parsed from the request body according to the agreed JSON, XML and other data formats; at the same time, locate and extract the relevant identification information of the target bucket (such as the name and number of the bucket) specified in the request; further, sort out the business parameters from the request, such as the parameter content such as the requested operation type (read, write, delete, etc.), the amount of data involved, and the expected response format, which are presented in the form of a parameter list or a specific configuration segment in the request body.

[0055] Step 102: Use the user information to verify whether the user has access rights to the target bucket.

[0056] In an embodiment of the present application, the gateway establishes a connection with a back-end data source (which may be a relational database, a distributed database, or a specialized permission management middleware, etc.) that stores user permission information through a configured database connection driver (such as a JDBC driver for a MySQL database, etc.) or by calling an API interface provided by the permission management system to ensure data interaction.

[0057] Use the user information extracted from the business request as the query condition, construct the corresponding query statement (if it is a database query, it may be an SQL statement, if it is an API interface call, construct the parameters according to the interface requirements), and send it to the permission management data source for retrieval. For example, if the user ID is used as the query condition, find all the permission records corresponding to the user ID in the permission data table, focus on the permission records related to the target bucket, and check whether the permission to access the target bucket is clearly granted (for example, check whether the permission field contains the read, write, execute and other permission identifiers of the corresponding target bucket), and then judge based on the query results. If there is corresponding permission, it is determined that the user has access to the target bucket, otherwise it is determined that there is no permission.

[0058] Step 103: If it is confirmed that the user has access rights to the target bucket, a corresponding business process flow is obtained based on the business parameters, and the business process flow is executed.

[0059] In the embodiment of the present application, the gateway maintains a business process mapping table (which can be a data structure in memory, such as a hash table, mapping dictionary, etc., or stored in the form of a configuration file and loaded into the memory at startup), which records the correspondence between different business parameter combinations and corresponding business processing flows. The business parameters obtained from the business request are used as key values ​​to search and match in this business process mapping table to find the relevant description information of the corresponding business processing flow (such as the step number of the process, the functional modules involved, the calling order between the modules, etc.).

[0060] Initialize the required resources and environment based on the business process description found. For example, if the business process involves calling other external services, establish connections with these services; if temporary cache space is needed to temporarily store data, allocate the corresponding memory cache area; if file operations are involved, prepare the corresponding file read and write handles and other operations to ensure that the business process can be started and executed smoothly.

[0061] According to the logical order of each step specified in the business process, the relevant internal modules or external services are called in sequence to perform specific operations. For example, if the business process is to handle file uploads, the file data sent by the client is received first, then the integrity and format of the data are verified, and then the file is stored in the specified storage location, etc. Each link is strictly carried out according to the process settings until the entire business process is completed.

[0062] Step 104 , in the process of executing the business process flow, collect the operation logs generated by the source bucket involved in the business process flow, and store the operation logs in the target bucket according to the preset rules.

[0063] In the embodiment of the present application, storing the operation log to the target bucket according to the preset rules includes the following steps A1-A3:

[0064] Step A1, storing the operation log in the form of key-value pairs in a data table in the log management subsystem, wherein the key name of the key-value pair is generated according to the associated parameters of the operation log, and the key value of the key-value pair is generated according to the operation content of the operation log.

[0065] In the embodiment of the present application, the operation log is stored in the form of a key-value pair in a data table in the log management subsystem, including the following steps A101-A103:

[0066] Step A101, obtaining the associated parameters of the operation log, and generating a key name based on the associated parameters.

[0067] In an embodiment of the present application, the associated parameters of the operation log are obtained, and a key name is generated based on the associated parameters, including: obtaining the object associated with the operation log and the timestamp corresponding to the operation log; calculating a hash value based on the bucket name of the target bucket and the object name of the object, and taking the modulus of the hash value to obtain a target value; generating a key name based on the target value, the bucket name of the target bucket, the object name of the object and the timestamp.

[0068] Specifically, first, extract the object information and corresponding timestamp associated with it from the operation log. The object here can be a specific data storage entity such as a file or data block in the storage system, and the timestamp records the specific time when the operation occurred. Next, use the bucket name of the target bucket and the object name of the object as input, and calculate a hash value through a hash function (such as MD5, SHA-1 and other common hash algorithms). The characteristic of the hash function is that it can convert input data of any length into output data of a fixed length. Then, perform a modulus operation on the calculated hash value. The modulus operation is usually used to map the hash value to a specific range to obtain a target value. This target value will be used for subsequent key name generation.

[0069] Finally, a key name is generated based on the target value, bucket name of the target bucket, object name, and timestamp obtained above. The specific format of generating the key name may be to combine these parameters in a certain order and format, for example: "<target value><bucket name><object name><timestamp>". The key name generated in this way is unique and can uniquely identify the storage location corresponding to an operation log.

[0070] Step A102: Generate a key value based on the log content of the operation log.

[0071] Specifically, detailed log contents are extracted from the operation log. These contents include information such as the operation type (such as read, write, delete, etc.), the operator of the operation, and the specific data of the operation. Based on the extracted log content, a key value is generated through a certain conversion rule. This conversion rule may be to serialize the log content (such as converting it to JSON or XML format) or perform some encoding (such as Base64 encoding) to convert the log content into a format suitable for storage in a data table.

[0072] Step A103: store the key name and key value corresponding to the operation log into a data table in the log management subsystem.

[0073] Specifically, determine the target data table in the log management subsystem. This data table is used to store information related to the operation log. The generated key name and the generated key value are inserted into the data table in the log management subsystem as a record. This can be achieved through the insertion operation of the database, such as using the INSERT statement in a relational database and the corresponding write operation interface in a non-relational database. In this way, the operation log is stored in the data table of the log management subsystem in the form of key-value pairs, which is convenient for subsequent query, analysis and management.

[0074] First, obtaining the objects associated with the operation log and the corresponding timestamp can accurately grasp the key elements closely related to the operation, provide a basic and clearly directed data support for the subsequent key name generation, and enable the key name to accurately reflect the subject and time of the operation. Secondly, the hash value is calculated based on the bucket name of the target bucket and the object name of the object, and the target value is obtained by modulus. The characteristics of the hash algorithm can be used to convert complex name information into relatively fixed values ​​with certain distribution patterns, which is convenient for more efficient data location and differentiation in the storage system and improves the efficiency of data management. Finally, the key name is generated by combining the target value, the bucket name of the target bucket, the object name of the object and the timestamp, so that the key name has uniqueness and rich semantics. On the one hand, it can ensure that each operation log can be uniquely identified in the storage data table, avoid data confusion, and facilitate fast and accurate search and retrieval of operation logs; on the other hand, the rich constituent elements also allow the key name to intuitively understand the various key information involved in the operation log, greatly optimizing the convenience and accuracy of operation log storage, retrieval and subsequent use.

[0075] Step A2: Read relevant log data of the operation log from the data table in the log management subsystem according to a preset time period.

[0076] Specifically, the system pre-configures a time period parameter, which can be a time unit such as hours, days, weeks, etc., and is used to determine the frequency of reading operation log data from the log management subsystem. For example, it can be set to perform a reading operation at 2 a.m. every day.

[0077] The relevant service program establishes a communication connection with the log management subsystem through the configured connection parameters (such as database connection string, user name, password, etc. If it is a distributed system, it may be connected through communication protocols such as RPC) so as to be able to access the data tables therein.

[0078] According to the preset time period, when the specified time is reached, the service program sends a data reading request to the log management subsystem. This request will specify to obtain the operation log related data from the corresponding data table. The operation log data stored in the data table may have been stored in the form of key-value pairs. The service program extracts the operation log data that meets the conditions according to the storage structure and query rules of the data table.

[0079] Step A3: construct an archive file using relevant log data, and store the archive file in a target bucket.

[0080] Specifically, using relevant log data to construct an archive file includes steps A301-A302:

[0081] Step A301, generate file content using relevant log data and obtain a file name generated according to a file generation rule.

[0082] Specifically, obtaining a file name generated according to a file generation rule includes: obtaining a preset file name prefix and a file identification code; generating a file name based on the preset file name prefix, the bucket name of the source bucket, the timestamp corresponding to the operation log, and the file identification code.

[0083] First, organize the read operation log data in a fixed format. For example, put the same type of operation logs (such as all file write operation logs) together, and add corresponding header information (such as file type identification, time range, etc.) and tail information (such as checksum and other information used for data integrity verification). Generate the file name of the archive file according to the specified naming format, and the file name format is: <file name prefix><source bucket name><timestamp><32-bit unique flag>. For example, the file name prefix may be "archive_", the source bucket name is "bucket1", and the timestamp is "20230501120000" (representing 12:00 on May 1, 2023). The 32-bit unique flag can be generated by a hash algorithm or a random number generator.

[0084] Step A302, constructing an archive file based on the file content and the file name.

[0085] Create a new file based on the generated file name. This can be achieved through the file creation function of the operating system, opening up a new file storage space in the file system and assigning the file name to the file. Write the generated file content to the newly created file. This involves the file write operation, writing the sorted operation log data (including header information and tail information) to the file in sequence.

[0086] When all file contents are written to the file, the archive file is constructed. At this time, the archive file contains the operation log data organized in a fixed format and has a file name that conforms to the specified format, which is convenient for subsequent storage, management and retrieval.

[0087] The target bucket is a specific storage location in the object storage engine. The constructed archive files are uploaded to the target bucket through the API or related storage interface provided by the object storage engine. During the storage process, some attribute settings of the archive files are also involved, such as file access rights, storage levels (such as hot storage, cold storage, etc.), etc., to ensure that the archive files can be managed and accessed as expected.

[0088] When using relevant log data to construct an archive file, first generate the file content through the relevant log data and obtain the file name according to specific rules. This step-by-step operation makes the construction process of the archive file clear and orderly. Among them, in the process of obtaining the file name, first obtain the preset file name prefix and file identification code, which can determine some common identification and unique identification elements in advance, facilitate the classification and distinction of archive files, and enhance the systematic nature of file management. Then generate the file name based on the preset file name prefix, the bucket name of the source bucket, the timestamp corresponding to the operation log, and the file identification code. In this way, the file name includes elements that can reflect the source of the file, the generation time and other key information. Not only can the archive file intuitively present important attributes through the file name, which is convenient for subsequent rapid positioning, identification and retrieval, but the file name generated based on these clear rules also guarantees its uniqueness, avoiding management confusion caused by repeated file names, and greatly improving the efficiency and accuracy of archive file construction and subsequent management and use.

[0089] In an embodiment of the present application, after reading relevant log data of the operation log from the data table in the log management subsystem according to a preset time period, the method also includes: classifying the relevant log data according to buckets to obtain log data corresponding to each bucket; generating a corresponding data table using the log data corresponding to each bucket; and storing the data table corresponding to each bucket in the log analysis subsystem.

[0090] Specifically, first, obtain all relevant log data from the real-time log management subsystem. These log data contain various operation records, such as read and write operations on different buckets. Analyze each log data and identify the identification information related to the bucket. Usually, this identification information includes the name, number or other attributes that can uniquely identify the bucket. Classify the log data according to the identified bucket identification information. For example, if there are bucket A, bucket B and bucket C, then all log data related to bucket A are classified into one category, those related to bucket B are classified into one category, and so on.

[0091] Design the corresponding data table structure for the log data corresponding to each bucket. The design of the data table structure needs to consider the key information contained in the log data, such as the operation type (read, write, delete, etc.), operation time, user or program identifier of the operation, and the amount of data involved in the operation. Determine the field type and length in the data table. For example, the operation time field may use a timestamp format, and the operation type field may use an enumeration type to represent different operations.

[0092] Fill the classified log data of each bucket according to the designed data table structure. That is, extract the relevant information from each log data and fill it into the corresponding fields. Generate the corresponding data table based on the filled data through database operations (such as using the CREATE TABLE statement in a relational database, or using the corresponding table creation operation in a non-relational database).

[0093] Establish a connection with the log analysis subsystem. If the log analysis subsystem is based on a certain database architecture (such as an OLAP column database with an MPP architecture), you need to use the corresponding database connection method, such as connecting to a relational database through JDBC (Java Database Connectivity), or using a specific database client to connect to a non-relational database. Transfer the data table corresponding to each generated bucket to the log analysis subsystem. This may involve inserting batch data to ensure that all data in the data table can be accurately stored in the log analysis subsystem.

[0094] In the embodiment of the present application, the user can also perform log query operations. Specifically, the user enters the conditions to be queried in the provided interface through the corresponding application or query tool. The specific query conditions are bucket name, object name and time range. For example, the user enters the bucket name as "bucket_001", the object name as "file_abc", and the time range is set from "2024-11-01 00:00:00" to "2024-11-05 00:00:00", so as to clarify the real-time log range to be obtained.

[0095] After receiving the query conditions entered by the user, the query service in the system parses them and extracts the key parameters such as bucket name, object name and time range to prepare for subsequent searches in the distributed KV storage.

[0096] First, the prefix part used to construct the KEY is generated according to the prefix calculation rules. The hash value is calculated according to the bucket name (such as "bucket_001" above) and the object name ("file_abc"), and this calculation can be completed by a suitable hash algorithm (such as MD5, SHA-1 and other common algorithms) to obtain a hash value.

[0097] Next, this hash value is modulo the number of shards (100) to obtain the final prefix value, which determines the shard position of the data in the distributed KV storage. After obtaining the prefix, the complete KEY is constructed according to the established KEY design rule "prefix-bucket name-object name-timestamp". The previously calculated prefix is ​​combined with the bucket name "bucket_001" and object name "file_abc" entered by the user, and the timestamps within the time range to be queried (within the interval of "2024-11-0100:00:00" to "2024-11-0500:00:00", according to the specific time accuracy, such as the timestamp corresponding to each minute, etc.) to form multiple keys that meet the requirements, which are used to match and find the corresponding values ​​in the distributed KV storage later.

[0098] Through the standard JDBC query interface, the constructed KEY is sent to the storage system based on the distributed KV implementation. The system performs a search operation in the corresponding shard according to the KEY to find the matching VALUE, that is, the specific content of the query log. The distributed KV storage system here will locate the shard based on the prefix of the KEY, and then quickly retrieve the records that meet the entire KEY condition in the specific shard.

[0099] Integrate the values ​​corresponding to each KEY (i.e. the specific content of the query log) found in the distributed KV storage. For example, if multiple KEYs are searched in chronological order, the corresponding values ​​are sorted in chronological order.

[0100] Finally, the integrated query results are returned to the user, so that the user can see the real-time log content that meets his query conditions, thus completing the entire log query process.

[0101] In the embodiment of the present application, users can also perform log analysis operations. First, they must clearly define the goal to be achieved through log analysis, such as understanding the operation status of an object in a specific time period, the distribution of request status initiated by a specific client, or the time consumption statistics of different operation types. According to the analysis goal, the specific elements to be concerned about and the corresponding query statistical conditions are determined, such as focusing on "object" (object name), "operation" (operation type) or other field-related analysis.

[0102] According to the determined analysis requirements, select the corresponding elements and corresponding operation methods from the provided query conditions to build a specific query condition combination. For example, if you want to analyze the operations related to the object named "file_abc", for the "object" field, you can select the "equal" operation and use "file_abc" as the specific value; if you want to analyze the requests initiated by the client from a certain IP segment, for the "remote_addr" field, you can select the "prefix match" operation and fill in the corresponding IP prefix, etc. You can combine multiple query conditions at the same time to more accurately filter out data that meets the analysis requirements.

[0103] Since the database creates different tables according to the bucket granularity, it is necessary to determine the specific data table to be queried based on the bucket involved in the operation log. First determine the bucket corresponding to the operation log in the analysis target, and then find the table corresponding to the bucket. Subsequent query and statistical operations will be performed in this table to ensure the accuracy of the data source.

[0104] Serial number Field Name Field Type Field Description 1 user_id String User ID 2 object String Object Name 3 time Timestamp Timestamp of receiving the request 4 remote_addr String Client Address 5 request_id String Request ID 6 operation String Operation Type 7 uri String Request Parameters 8 http_status Integer Request status code 9 error_code Integer Handling error codes 10 byte_send Long integer Returns the number of bytes 11 obj_size Long integer Object size 12 total_time Long integer Total processing time 13 referrer String Referrer field in HTTP header 14 User-Agent String Client Type 15 VersionID String Version Number

[0105] Table 2 - Creating different tables according to bucket granularity

[0106] Use the selected query conditions and the query interface provided by the database (because OLAP column-based databases based on the MPP architecture usually have corresponding query languages ​​or interfaces for users to operate) to retrieve data in the located table. The database will screen each record in the table according to the set conditions (such as field matching rules, numerical value comparison, etc.), and extract the records that meet all query conditions as the preliminary query result set.

[0107] Further analyze and process the query result set, and perform corresponding statistical calculations according to the analysis objectives. For example, if you want to analyze the time consumption of different operation types, you can group the result set according to the "operation" field, and then calculate the average value, sum, and other statistical indicators of the "total_time" field in each group; if you want to analyze the distribution of request status codes, you can count the frequency of occurrence of different "http_status" values. Through these statistical analysis operations, you can obtain data insights that reflect the actual situation and help achieve the analysis objectives.

[0108] The results of statistical analysis are presented in an appropriate manner, such as generating charts (bar charts showing the time consumption comparison of different operation types, pie charts showing the distribution of request status codes, etc.) and reports (listing the values ​​of various statistical indicators, etc.) for intuitive viewing and interpretation. Analysts understand the information contained in the operation log based on the presented results, discover potential patterns, problems or trends, and provide a basis for subsequent decision-making and system optimization.

[0109] In the present embodiment, a processing device for an operation log is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the descriptions that have been made are not repeated here. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0110] This embodiment provides a device for processing an operation log, such as Figure 3 As shown, including:

[0111] The receiving module 301 is used to receive at least one service request sent by the client to the gateway, wherein the service request carries user information of the user corresponding to the client, the target bucket requested to be accessed by the user, and service parameters;

[0112] A verification module 302 is used to verify whether the user has access rights to the target bucket using the user information;

[0113] The execution module 303 is used to obtain the corresponding business process flow based on the business parameters and execute the business process flow if it is confirmed that the user has the access right to access the target bucket;

[0114] The processing module 304 is used to collect the operation logs generated by the source bucket involved in the business processing process during the execution of the business processing process, and store the operation logs in the target bucket according to the preset rules.

[0115] In an embodiment of the present application, the processing module 304 is used to store the operation log in the form of a key-value pair in a data table in the log management subsystem, wherein the key name of the key-value pair is generated according to the associated parameters of the operation log, and the key value of the key-value pair is generated according to the operation content of the operation log; read the relevant log data of the operation log from the data table in the log management subsystem according to a preset time period; use the relevant log data to build an archive file, and store the archive file in a target bucket.

[0116] In an embodiment of the present application, the processing module 304 is used to obtain the associated parameters of the operation log and generate a key name based on the associated parameters; generate a key value based on the log content of the operation log; and store the key name and key value corresponding to the operation log in a data table in the log management subsystem.

[0117] In an embodiment of the present application, processing module 304 is used to obtain the object associated with the operation log and the timestamp corresponding to the operation log; calculate the hash value based on the bucket name of the target bucket and the object name of the object, and take the modulus of the hash value to obtain the target value; generate a key name based on the target value, the bucket name of the target bucket, the object name and the timestamp.

[0118] In the embodiment of the present application, the processing module 304 is used to generate file content using relevant log data and obtain a file name generated according to a file generation rule; and construct an archive file based on the file content and the file name.

[0119] In an embodiment of the present application, processing module 304 is used to obtain a preset file name prefix and a file identification code; and generate a file name based on the preset file name prefix, the bucket name of the source bucket, the timestamp corresponding to the operation log, and the file identification code.

[0120] In an embodiment of the present application, the device also includes: a storage module, which is used to classify relevant log data according to buckets to obtain log data corresponding to each bucket; generate a corresponding data table using the log data corresponding to each bucket; and store the data table corresponding to each bucket in the log analysis subsystem.

[0121] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0122] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0123] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0124] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0126] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0127] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0128] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing an operation log, characterized in that: The method comprises: Receive at least one service request sent by a client to a gateway, wherein the service request carries user information of a user corresponding to the client, a target bucket requested to be accessed by the user, and service parameters; Using the user information to verify whether the user has access rights to the target bucket; If it is confirmed that the user has access rights to the target bucket, a corresponding business process flow is obtained based on the business parameters, and the business process flow is executed; In the process of executing the business processing flow, the operation logs generated by the source bucket involved in the business processing flow are collected, and the operation logs are stored in the target bucket according to preset rules.

2. The method according to claim 1, characterized in that The storing the operation log to the target bucket according to the preset rules includes: The operation log is stored in a data table in the log management subsystem in the form of a key-value pair, wherein the key name of the key-value pair is generated according to the associated parameters of the operation log, and the key value of the key-value pair is generated according to the operation content of the operation log; Reading relevant log data of the operation log from the data table in the log management subsystem according to a preset time period; An archive file is constructed using the relevant log data, and the archive file is stored in the target bucket.

3. The method according to claim 2, characterized in that The step of storing the operation log in a data table in the log management subsystem in the form of a key-value pair includes: Obtaining associated parameters of the operation log, and generating a key name based on the associated parameters; Generate a key value based on the log content of the operation log; The key name and the key value corresponding to the operation log are stored in a data table in a log management subsystem.

4. The method according to claim 3, characterized in that: The obtaining the associated parameters of the operation log and generating a key name based on the associated parameters includes: Obtain the object associated with the operation log and the timestamp corresponding to the operation log; Calculate a hash value based on the bucket name of the target bucket and the object name of the object, and perform modulo operation on the hash value to obtain a target value; The key name is generated based on the target value, a bucket name of the target bucket, an object name of the object, and the timestamp.

5. The method according to claim 2, characterized in that: The step of constructing an archive file using the relevant log data includes: Generate file content using the relevant log data and obtain a file name generated according to a file generation rule; The archive file is constructed based on the file contents and the file name.

6. The method according to claim 5, characterized in that The obtaining of the file name generated according to the file generation rule includes: Get the preset file name prefix and file identification code; The file name is generated based on the preset file name prefix, the bucket name of the source bucket, the timestamp corresponding to the operation log, and the file identification code.

7. The method according to claim 2, characterized in that After reading relevant log data of the operation log from the data table in the log management subsystem according to a preset time period, the method further includes: Classify the relevant log data according to buckets to obtain log data corresponding to each bucket; Generate the corresponding data table using the log data corresponding to each bucket; The data table corresponding to each bucket is stored in the log analysis subsystem.

8. A device for processing operation logs, characterized in that: The device comprises: A receiving module, configured to receive at least one service request sent by a client to a gateway, wherein the service request carries user information of a user corresponding to the client, a target bucket requested to be accessed by the user, and service parameters; A verification module, used to verify whether the user has access rights to the target bucket using the user information; An execution module, configured to obtain a corresponding business process flow based on the business parameters and execute the business process flow if it is confirmed that the user has access rights to access the target bucket; The processing module is used to collect the operation logs generated by the source bucket involved in the business processing process during the execution of the business processing process, and store the operation logs in the target bucket according to preset rules.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images