Log data processing method, system, storage medium and electronic device

Abstract

The embodiment of the present application discloses a log data processing method, system, storage medium and electronic device. The method obtains a corresponding log file based on the log record generated by the target device. The target device is any business processing device running a target business process. The target business process is any process running a target business. Each log record includes a business request identifier; the log records of each business processing device are aggregated based on the business request identifier to obtain a first index file, which is used to record the storage location of the log record carrying the target business request identifier, and the target business request identifier is any business request identifier in the log record of each business processing device. The present application can reduce the resource consumption of log record storage and improve query efficiency. The present application can be applied to message interaction, multimedia applications, smart transportation or life, such as itinerary sharing, vehicle refueling, travel or listening to books.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a log data processing method, system, storage medium, and electronic device. Background Art

[0002] Log records are important data that are relied on for finding application defects and tracing problems. Log records can usually be stored on local disks. As the amount of log record data increases, log records need to be stored on multiple machines, and in order to facilitate quick search, log records scattered on multiple machines need to be aggregated. Currently, the amount of log record data in many applications can reach hundreds of TB (terabytes) per day, several PB (petabytes) per month, and trillions of logs. Aggregation of massive log records causes excessive resource consumption, and the query efficiency of massive logs needs to be improved. Summary of the invention

[0003] In order to solve at least one of the above technical problems, embodiments of the present application provide a log data processing method, system, storage medium and electronic device.

[0004] On the one hand, an embodiment of the present application provides a log data processing method, the method is applied to a log data processing system, the log data processing system includes multiple business processing devices, the method includes:

[0005] Based on the log records generated in the target device, a log file corresponding to the target device is obtained, the target device is any of the business processing devices running the target business process, the target business process is any process running the target business, and each of the log records includes a business request identifier;

[0006] The log records generated by each of the business processing devices are aggregated based on the business request identifier to obtain a first index file, wherein the first index file is used to record the storage location of the log records carrying the target business request identifier, and the target business request identifier is any business request identifier in the log records generated by each of the business processing devices.

[0007] On the other hand, an embodiment of the present application provides a log data processing system, the system comprising a plurality of business processing devices, the system further comprising:

[0008] A log file processing module, configured to obtain a log file corresponding to a target device based on a log record generated in the target device, wherein the target device is any of the business processing devices running a target business process, and the target business process is any process running a target business, and each of the log records includes a business request identifier;

[0009] The first index file acquisition module is used to aggregate the log records generated by each of the business processing devices based on the business request identifier to obtain a first index file. The first index file is used to record the storage location of the log records carrying the target business request identifier. The target business request identifier is any business request identifier in the log records generated by each of the business processing devices.

[0010] On the other hand, an embodiment of the present application provides a computer-readable storage medium, characterized in that at least one instruction or at least one program is stored in the computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned log data processing method.

[0011] On the other hand, an embodiment of the present application provides an electronic device, characterized in that it includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the above-mentioned log data processing method by executing the instructions stored in the memory.

[0012] The embodiments of the present application provide a log data processing method, system, storage medium and electronic device. In the embodiments of the present application, the log file corresponding to each business processing device can be stored in a distributed file system, so that the directory structure generated by the log file itself is equivalent to an index that quickly distinguishes different devices, improving the efficiency of log record query, and by generating a first index file based on the aggregation of business request identifiers, the storage location of all associated log records can be located through a single IO operation, thereby further improving the efficiency of log record query. In addition, the machine resources used by the embodiments of the present application to store log files and the first index file in a distributed file system are relatively small, thereby significantly reducing the resource consumption of log record storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is a feasible implementation framework diagram of the log data processing method provided in the embodiments of this specification;

[0015] Figure 2 It is a flow chart of a log data processing method provided in an embodiment of the present application; Figure 3It is a schematic diagram of the flow of the first index file generation method provided in an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of the implementation process of log data processing provided by the embodiment of the present application in a specific scenario;

[0017] Figure 5 It is a flow chart of a log record query method in the related technology provided by an embodiment of the present application; Figure 6 This is a schematic diagram of a log record query framework provided by an embodiment of the present application;

[0018] Figure 7 It is a schematic diagram of a log record data processing framework provided in an embodiment of the present application; Figure 8 is a block diagram of a log data processing system provided by an embodiment of the present application;

[0019] Fig. 9 It is a schematic diagram of the hardware structure of a device provided in an embodiment of the present application for implementing the method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0022] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not used to limit the embodiments of the present application.

[0023] In the following, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more. In order to facilitate the understanding of the above-mentioned technical solutions and the technical effects produced by the embodiments of the present application, the embodiments of the present application first explain the relevant professional terms:

[0024] Bitcask: It is a log-type, key-value pair storage model based on a hash table structure. In the log-type data storage model, all write operations only append and do not modify the old data. In the Bitcask model, data files are written to files in a log-type manner, which only increases but does not decrease. The files have a certain size limit. When the file size increases to the corresponding limit, a new file will be generated, and the old file will only be read but not written. At any point in time, only one file is writable, which is called the Active Data File in the Bitcask model, and other files that have reached the size limit are called Older Data Files.

[0025] RocksDB: uses a log-structured KV storage engine. Designed for fast and low-latency storage

[0026] It optimizes the equipment and maximizes the read and write performance. It can be suitable for a variety of different workload types. It provides some basic operations, such as opening and closing the database, as well as advanced database operations, such as merging, compression and filtering, and also provides read and write support.

[0027] ElasticSearch: ElasticSearch is a distributed storage search engine. ElasticSearch has a single data storage mode and a high data storage cost.

[0028] QPS: Queries Per Second, is the query rate per second, which is the number of queries a device can respond to per second. It is a measure of the amount of traffic a device handles within a specified time, that is, the number of response requests per second, which is also the maximum throughput.

[0029] In order to achieve the aggregation of massive log data in related technologies, log records can usually be written to ElasticSearch, which stores and indexes the logs (for example, by creating an inverted list through word segmentation). When querying, the inverted list can be searched according to the keyword to locate the log corresponding to the keyword. However, since word segmentation and inverted list creation for log records consume a lot of resources, it causes excessive consumption of machine resources, and this problem becomes increasingly prominent as the amount of log record data increases.

[0030] In order to reduce resource consumption caused by the aggregation of massive log records, embodiments of the present application provide a log data processing method, system, storage medium, and electronic device.

[0031] See also Figure 1 , Figure 1 is a feasible implementation framework diagram of the log data processing method provided in the embodiments of this specification, such as Figure 1 As shown, the implementation framework may include at least a business processing device 01, a log recording device 02, a first log aggregation device 03, a second log aggregation device 04 and a distributed file system 05, and each business processing device 01 corresponds to a log recording device 02. Among them, the business processing device 01 is used to provide corresponding services to users, such as message query services, account query services, multimedia services, etc., and the embodiment of the present disclosure does not limit the specific service content. In addition, for each service, it can be run in one or more business processing devices 01. In one embodiment, according to the business identifier, one or more business processing devices 01 that run the service pointed to by the business identifier can be uniquely determined. The business processing device 01 can interact with the user terminal to provide the above-mentioned services, and the user terminal includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. This application does not limit the services that the business processing device 01 can provide. These services can provide corresponding services for various scenarios, such as message interaction, multimedia applications, smart transportation or life scenarios. Smart transportation scenarios can include itinerary sharing, vehicle refueling, vehicle charging and other services, and life scenarios include travel or listening to books and other services. In one embodiment, the present application can be applied to the field of autonomous driving to manage logs generated by relevant technical solutions in the field of autonomous driving. The field of autonomous driving usually requires the application of technologies such as map processing, environmental perception, behavioral decision-making, path planning, and motion control, and this field has broad application prospects. The map in map processing may include vector map data or image map data, where vector map data is stored in vector format and image map data is stored in image format. By performing data processing on the map, it can provide support for relevant technologies in the field of autonomous driving and generate corresponding log records.

[0032] For each business processing device 01, a unique log recording device 02 is set corresponding thereto. The log recording device 02 is used to generate log records of the business processing device 01, store the log records in batches in the log files corresponding to the business processing device 01 in the distributed file system 05, generate a second index file for the log file based on the time when the log records are generated, and associate the log file with the second index file in the distributed file system 05.

[0033] The first log aggregation device 03 can aggregate the logs generated by each log recording device 02 based on the business request identifier in stages to obtain a first index temporary file, and the second log aggregation device 04 can aggregate each first index temporary file based on the business request identifier to obtain a first index file, and store the first index file in the distributed file system 05. When the distributed file system 05 stores the log file, the second index file and the first index file, it can support efficient log record query services.

[0034] The implementation framework also includes a query request interaction device 06 and at least one query execution device 07. The query request interaction device serves as an interface for interacting with the user, and can be used to receive log record query requests and feedback log query results. If there are multiple query execution devices 07, the query request interaction device can provide concurrent log query services to the user by interacting with each query execution device 07. For each query execution device 07, the log query result can be obtained by interacting with the distributed file system 05, and the log query result can be fed back to the query request interaction device 06.

[0035] Any of the above-mentioned devices or systems mentioned in the embodiments of the present application may be a mobile terminal, a desktop computer, a tablet computer, a laptop computer, a digital assistant, a smart wearable device, or any other physical device that has communication and data processing capabilities, and may also include software running in the physical device. Of course, it may also be a server, which may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services, and the present application does not limit this.

[0036] The method provided in the embodiment of the present application may also involve a blockchain, that is, the method provided in the embodiment of the present application may be implemented based on a blockchain, or the data involved in the method provided in the embodiment of the present application may be stored based on a blockchain, or the execution subject of the method provided in the embodiment of the present application may be located in a blockchain. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain may include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0037] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operation monitoring. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and the maintenance of the correspondence between the user's real identity and the blockchain address (authority management), etc., and, under authorization, supervises and audits the transactions of certain real identities and provides risk control rule configuration (risk control audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and records valid requests to storage after consensus is reached. For a new business request, the basic service first adapts the interface for parsing and authentication (interface adaptation), and then encrypts the business information through the consensus algorithm (consensus management). The smart contract module is responsible for the registration and issuance of contracts, as well as contract triggering and contract execution. Developers can define the contract logic in a programming language and publish it to the blockchain (contract registration). According to the logic of the contract terms, the key or other events are called to trigger the execution and complete the contract logic. It also provides the function of contract upgrade and cancellation. The operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation and real-time status visualization output of the product during the product release process, such as alarm, network monitoring, node equipment health monitoring, etc.

[0038] The platform product service layer provides the basic capabilities and implementation framework of typical applications. Developers can superimpose business features based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on blockchain solutions for business participants to use.

[0039] The following describes a log data processing method according to an embodiment of the present application. Figure 2A flow chart of a log data processing method provided by an embodiment of the present application is shown. The embodiment of the present application provides the method operation steps as described in the embodiment or flow chart, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiment or the accompanying drawings. The above method may include:

[0040] S101. Based on the log records generated in the target device, the log file corresponding to the above target device is obtained, the above target device is any of the above business processing devices running the target business process, the above target business process is any process running the target business, and each of the above log records includes a business request identifier. The method in the embodiment of the present application can be applied to a log data processing system, the log data processing system can include multiple business processing devices, and the multiple business processing devices can process multiple businesses, and each business can run on multiple business processing devices. For example, an account processing business 10 and a message processing business 20 can be provided, the account processing business 10 runs in the business processing device 11 and the business processing device 12, and the message processing business 20 runs in the business processing device 21 and the business processing device 22, the target business can be any of the account processing business and the message processing business, and taking the target business as the account processing business as an example, the target business process runs in the business processing device 11 and the business processing device 12.

[0041] According to the above description, the target device can be any of the business processing devices 11, 12, 21 and 22. That is to say, for a business processing device that runs a certain business, a corresponding log file will be obtained. In the disclosed embodiment, the log files are stored in the distributed business system, and the logs of different business processing devices are written into their own exclusive log files, and there is no competition in the data writing process between each other, thereby significantly improving the writing efficiency of the log files.

[0042] In one embodiment, in order to reduce the coupling of the log data processing system and improve the writing speed of the log file, a corresponding log recording device can be set for each business processing device. Each of the above-mentioned log recording devices writes the log records generated by the corresponding business processing device into the log file corresponding to the above-mentioned target device in the above-mentioned distributed file system.

[0043] In the embodiment of the present application, each log record includes a service request identifier (callid). A user request is uniquely identified, and each log record carries callid information. During the processing of each service request, the associated service processing device may generate a corresponding log record. For example, user account A requests to send a message to user account B. The service request identifier of this request is (100001). In order to execute the request, service processing device 11 and service processing device 22 both perform corresponding operations, thereby generating corresponding log records, and the generated log records all carry the service request identifier (100001).

[0044] S102. Aggregate the log records generated by each of the above-mentioned business processing devices based on the business request identifier to obtain a first index file, wherein the first index file is used to record the storage location of the log records carrying the target business request identifier, and the target business request identifier is any business request identifier in the log records generated by each of the above-mentioned business processing devices.

[0045] In the embodiment of the present application, the log file corresponding to each business processing device is stored in the distributed file system, and each log record in each log file carries a business request identifier. In order to improve the speed of querying log records based on the business request identifier, a global index can be generated for the business request identifier, and the global index is referred to as the first index file in the embodiment of the present application. The first index file can be organized in the form of a key-value pair, wherein the key corresponds to a certain business request identifier (target business request identifier), and the value corresponds to the storage location of each log record carrying the target business request identifier in the distributed file system. Still using the above example, the business request pointed to by the business request identifier 100001 is in the process of being processed, the log record 100 generated by the business processing device 11 is stored in the log file A, and the log record 200 generated by the business processing device 22 is stored in the log file B. Then, according to the business request identifier 100001, the first index file is queried, and the storage location of the log record 100 and the log record 200 in the distributed file system can be obtained, so that the log record carrying the same business request identifier can be quickly located, and the log record query speed is significantly improved.

[0046] The generation process of the first index file can be understood as an aggregation process. In order to improve the aggregation speed of the above-mentioned aggregation process, in the embodiment of the present application, this aggregation process can be divided into two steps, one for inter-module aggregation and the other for global aggregation. Figure 3 , Figure 3 : is a schematic diagram of a flow chart of a method for generating a first index file provided in an embodiment of this specification, including:

[0047] S1021. Aggregate the log records generated by each of the above-mentioned business processing devices within the second target time interval based on the business request identifier to obtain a first index temporary file, and the above-mentioned second target time interval is any time interval determined according to the second time segmentation strategy.

[0048] The embodiment of the present disclosure does not limit the second time segmentation strategy, and illustratively, the time segmentation may be performed at a time interval of 5 minutes. For example, step S1021 is performed once every five minutes.

[0049] For example, when step S1021 is triggered at time T2, the log records generated by each business processing device in the time interval [T1, T2] can be aggregated based on the business request identifier, where T1 is the first five minutes of T2. The log records generated by each business processing device in these five minutes can participate in the aggregation based on the business request identifier, and the log records carrying the same business request identifier are aggregated. According to the business request identifier, the storage location of the corresponding log record in the distributed file system can be determined. This step focuses on the horizontal aggregation of the records generated by each business processing device, which is referred to as inter-module aggregation in the embodiment of the present application. Through this inter-module aggregation, the QPS can be reduced to one thousandth of the original level.

[0050] Step S1021 can be performed by Figure 1 The log first aggregation device 03 in the embodiment is implemented, and the log first aggregation device 03 aggregates the log records generated by each of the above-mentioned business processing devices based on the business request identifier in different time periods to obtain a first temporary index file corresponding to each time period.

[0051] S1022. Aggregate the first index temporary files based on the service request identifier to obtain the first index file.

[0052] The above-mentioned log first aggregation device 03 can also send each of the above-mentioned first temporary index files to the log second aggregation device 04, that is, the execution result of step S1021 is to generate multiple first index temporary files, and the log second aggregation device 04 can aggregate each first index temporary file again based on the business request identifier. The aggregation principle has been exemplified above and will not be repeated here. Since each first temporary index file contains relevant information of the log records generated by each business processing device within a specific time period, the embodiment of the present application refers to the aggregation process of the first temporary index file as full-network aggregation. Through full-network aggregation, such an effect can be achieved that the storage location of all log records associated with a certain business request identifier can be read through a single IO operation.

[0053] Specifically, the second log aggregation device 04 aggregates the first temporary index files based on the service request identifier to obtain the first index file, and writes the first index file into the distributed file system 05 .

[0054] In summary, the embodiments of the present application can store the log files corresponding to each business processing device in a distributed file system, so that the directory structure generated by the log file itself is equivalent to an index for quickly distinguishing different devices, thereby improving the efficiency of log record query, and by generating the first index file based on the aggregation of business request identifiers, the storage location of all associated log records can be located through a single IO operation, thereby further improving the efficiency of log record query.

[0055] By considering the scenario of performing log query based on the business request identifier, the first index file is designed in the embodiment of the present application. Further, the embodiment of the present application also considers another common log query scenario, that is, performing log query based on business, time and / or keywords. In order to improve the log query efficiency in this log query scenario, the embodiment of the present application may also include the following steps:

[0056] S201. Perform time-based aggregation on the log records generated in the above-mentioned target device to obtain a second index file of the above-mentioned target device, wherein the above-mentioned second index file is used to record the storage location of the log records generated in a first target time interval, and the above-mentioned first target time interval is any time interval determined according to the first time segmentation strategy.

[0057] For each log record generated by a business processing device, the embodiment of the present application can generate a corresponding second index file for it. That is to say, the second index file corresponds one-to-one to the log record corresponding to the business processing device, and can be used to record the generation time of the log record in the log file and the correspondence between the storage location of the log record.

[0058] The embodiment of the present application does not limit the specific content of the first time segmentation strategy, for example, segmentation can be performed every three minutes. In other words, time can be divided according to the strategy of segmentation every three minutes to obtain time intervals, each of which has a length of three minutes. For any time interval (the first target time interval), the second index file records the storage location of the log records generated by the target device in the time interval in the distributed file system.

[0059] S202. Associate the second index file and the log file.

[0060] Since the second index file record corresponds to the log file one by one, the second index file record can be associated with the log file in the distributed file system. The embodiment of the present application does not limit the specific association method. Specifically, the second index file and the above log file can be stored in the same directory. Exemplarily, there are 10 business processing devices, and the directory in the distributed system can include 10 locations, each location corresponds to a business processing device, and the log file and the second index file corresponding to the business processing device are stored in the location.

[0061] In one embodiment, in order to improve the writing efficiency of the log file, step S301 can also be executed: the log records generated in the above-mentioned target device are stored in the cache corresponding to the above-mentioned target device; when the data volume of the log records in the above-mentioned cache reaches a preset threshold, the log records in the above-mentioned cache are written into the above-mentioned log file.

[0062] Exemplarily, the log recording device corresponding to the target device can use a buffer queue to cache log records, accumulate 8MB of data and write them sequentially in batches to the corresponding log file. Accordingly, this operation can reduce the write QPS to 1 / 40,000 of the original. Of course, more data can be accumulated and then written in batches. This application does not limit the size of this preset threshold. Furthermore, the log recording device corresponding to the target device can also associate the above log file with the second index file corresponding to the above log file in the above distributed file system.

[0063] In the related art, log data is written into ElasticSearch, which creates an inverted index for the log data, and then locates the log data based on the established inverted index. However, this index creation method requires word segmentation of the log data content, and the creation of an inverted index is very resource-intensive, resulting in the need to consume a large amount of machine resources for the storage of log data. In addition, the efficiency of locating log data based on the inverted index is also low, and it cannot support a variety of retrieval forms. In order to solve the above problems in the related art, the embodiments of the present application provide a method for processing log data. Please refer to Figure 4 , Figure 4 It is a schematic diagram of the implementation process of log data processing provided by the embodiments of this specification in a specific scenario. Figure 4The modules in are equivalent to the target devices in this scenario, LogAgent is the log recording device corresponding to the business processing device (target device), LogMergeSvr is the first log aggregation device in this scenario, LogIdxSvr is the second log aggregation device in this scenario, and QuerySvr is the query execution device in this scenario. The aggregation of callid index by LogAgent corresponds to step S301, the aggregation of callid index based on bitcask in LogMergeSvr corresponds to step S1021, and the aggregation of callid index based on Rocksdb in LogIdxSvr corresponds to step S1022. By executing step S301, the write QPS can be reduced to the order of one ten-thousandth, and by executing steps S1021-S1022, the write QPS can be reduced to the order of one ten-millionth. In the embodiment of the present application, steps S1021, S1022 and step S301 all perform aggregation operations, that is, in Figure 4 In the illustrated implementation process, the embodiment of the present application performs a total of triple aggregation. Through the above triple aggregation, fewer machine resources can be used to establish an index for the log that can support rich retrieval forms, reduce the log storage cost, and improve the log query speed and intelligence.

[0064] At this point, the distributed file system stores the log file corresponding to each business processing device, the second index file corresponding to the log file, and the distributed file system also stores the first index file. Based on the existence of the above data, the embodiment of the present application also provides a method for quickly querying log records based on the above data. The method can be Figure 1 The query execution device 07 and the query request interaction device 06 participate in the implementation, including:

[0065] S401. The query execution device generates a log filtering condition according to query information, wherein the query information includes at least one of a service identifier, time information, keyword information, and a service request identifier, and sends the log filtering condition to the distributed file system.

[0066] Please refer to Figure 5 , which shows a schematic diagram of a log record query framework in related technology. Figure 5 It can be seen that in the related technology, fuzzy matching of logs is performed in QuerySvr (query execution device), and the matching elements can be obtained according to the query information. In order to perform fuzzy matching, QuerySvr needs to read a large number of related log records and related index data from the distributed file system into QuerySvr. The heavy IO operation of this process consumes a lot of resources and time.

[0067] In order to reduce the resource consumption and time consumption of the above-mentioned heavy IO operations, in the embodiment of the present application, the operation of log fuzzy matching is sunk to the distributed file system. The query execution device no longer needs to perform log fuzzy matching operations, but only needs to generate log filtering conditions, which are used to perform log fuzzy matching in the distributed file system.

[0068] S402. The distributed file system filters log records according to the log filtering conditions to obtain query results.

[0069] The distributed file system can perform log fuzzy matching according to the above log filtering conditions, thereby directly obtaining the query results, and feeding back the above query results to the query execution device, thereby eliminating the need to perform the above heavy IO operations, reducing the loss of resources and time.

[0070] In one embodiment, the log data processing system further includes a query request interaction device, and the method further includes:

[0071] S501. The query request interaction device obtains a log query request, where the log query request includes at least one of a service identifier, time information, keyword information, and a service request identifier.

[0072] In the disclosed embodiment, the query request interaction device can be an interface for interacting with the user. In the disclosed embodiment, the query request interaction device can provide the user with a variety of log query services based on at least one of the service identifier, time information, keyword information, and service request identifier, and a combination thereof.

[0073] S502. Determine a target query execution device according to the concurrency strategy, where the target query execution device is any query execution device among the multiple query execution devices of the log data processing system.

[0074] The specific content of the concurrency strategy is not limited in the embodiments of the present application, and reference may be made to related technologies. For example, the target query execution device may be determined based on a load balancing strategy, a proximity allocation strategy, and the like.

[0075] S503. Routing the log query request to the target query execution device.

[0076] The target query execution device can also directly feed back or integrate the queried log records to the query request interaction device, so that the query request interaction device feeds them back to the user.

[0077] In the embodiment of the present application, each business processing device has an exclusive log file in the distributed file system, which can be distinguished by the identification of the business processing device. The present application does not limit this identification. Exemplarily, it can be a MAC identification or an IP identification. Furthermore, in the embodiment of the present application, the log file corresponding to each business processing device can be recorded in pieces, thereby further improving the query efficiency. For example, each piece can record the log records generated by the business processing device within a certain hour, that is, each log file is an hourly granularity log file. Through the business identification, all business processing devices under the business identification can be located, and related log files can be quickly located. The log file can be a single log file or a log file recorded in pieces.

[0078] In some embodiments, logs can be queried based on time information in combination with the second index file. The exemplary time information is 2021-8-2015:03, and the relevant records in the second index file are [2021-8-2015:00, 2021-8-2015:05], [DB10008-DB10010]. The log records in the storage address corresponding to DB10008-DB10010 are log records generated from 2021-8-2015:00 to 2021-8-2015:05. Searching in the log records stored in DB10008-DB10010 can accurately locate the log records generated at 2021-8-2015:03. Obviously, log records can be quickly queried based on the second index.

[0079] As can be seen from the foregoing, the first index file is obtained based on the aggregation of the business request identifier. Based on the first index file, the storage address of the log record related to the business request identifier in the distributed file system can be quickly determined, thereby quickly locating the log record to be found.

[0080] For keyword query, you can refer to the relevant technology, and the embodiments of this application will not be described in detail. In some embodiments, the log records obtained based on any one of the service identifier, time information, and service request identifier and their combination can be fed back to the query execution device, and the query execution device matches the log records based on the above keywords to obtain the query results. In addition, the above query execution device can also integrate or aggregate the query results and feed back the processing results to the user.

[0081] Obviously, the embodiment of the present application can support fast log record query based on any one of the service identifier, time information, keyword information, and service request identifier, and their combination, and the specific operations will not be repeated here.

[0082] Please refer to Figure 6, which shows a schematic diagram of the log record query framework in the embodiment of the present application. Each QuerySvr can support machine-level log query. By generating log filter conditions, the log fuzzy matching can be sent to the distributed file system for execution. The distributed file system obtains the query result through the fast matching of the log file, the first index file and the second index file stored in itself, and feeds the result back to the QuerySvr, thereby completing the concurrent log query.

[0083] Please refer to Figure 7 , which shows a schematic diagram of the log record data processing framework in the embodiment of the present application. The embodiment of the present application includes two parts: warehousing and querying. Warehousing refers to the process of storing the log records and corresponding indexes generated by each business processing device (module) in a distributed file system, and querying refers to the process of searching for log records in a distributed file system. For each business processing device, a corresponding log recording device (LogAgent) is set up. The LogAgent itself can write the log records generated by the corresponding module into the distributed file system in batches to generate a corresponding second index file. Each LogAgent interacts with LogMergeSvr to perform inter-module aggregation, and LogIdxSvr performs full-network aggregation. The aggregation process can be referred to in the previous text and will not be repeated here.

[0084] On the basis of obtaining the log records, the first index file and the second index file, a quick query of the log records can be performed. The query process can also be referred to in the previous text and will not be described in detail here.

[0085] In the embodiment of the present application, the resource consumption for writing log files, the first index file and the second index file in the distributed file system is relatively small, and it can be implemented by using only a small amount of machine resources. The machine cost is only one-tenth of the solution using ElasticSearch in the related technology. In addition, based on the log files, the first index file and the second index file in the distributed file system, fast log record query can be supported, and massive log records can be stored and searched in seconds, which has achieved great advantages in the current situation of increasingly large log record data volumes.

[0086] The present application also discloses a log data processing system. Figure 8 As shown, the above system includes multiple business processing devices, and the above system includes:

[0087] The log file processing module 10 is used to obtain the log file corresponding to the target device based on the log record generated in the target device, the target device is any of the business processing devices running the target business process, the target business process is any process running the target business, and each of the log records includes a business request identifier;

[0088] The first index file acquisition module 20 is used to aggregate the log records generated by each of the above-mentioned business processing devices based on the business request identifier to obtain a first index file. The above-mentioned first index file is used to record the storage location of the log records carrying the target business request identifier. The above-mentioned target business request identifier is any business request identifier in the log records generated by each of the above-mentioned business processing devices.

[0089] In one embodiment, the system further includes a second index file acquisition module, which is used to perform the following operations: aggregating the log records generated in the target device based on time to obtain a second index file of the target device, wherein the second index file is used to record the storage location of the log records generated in the first target time interval, wherein the first target time interval is any time interval determined according to the first time segmentation strategy; associating the second index file with the log file. In one embodiment, the first index file acquisition module 20 is used to perform the following operations: aggregating the log records generated by each of the business processing devices in the second target time interval based on a business request identifier to obtain a first index temporary file, wherein the second target time interval is any time interval determined according to the second time segmentation strategy;

[0090] Aggregate the above-mentioned first index temporary files based on the service request identifier to obtain the above-mentioned first index file.

[0091] In one embodiment, the log file processing module is used to perform the following operations:

[0092] storing the log records generated in the target device in a cache corresponding to the target device;

[0093] When the data volume of the log records in the cache reaches a preset threshold, the log records in the cache are written into the log file.

[0094] In one embodiment, the log data processing system further includes a log recording device, a first log aggregation device, a second log aggregation device and a distributed file system, each business processing device corresponds to a log recording device, wherein each of the log recording devices is used to write the log records generated by the corresponding business processing device into the log file corresponding to the target device in the distributed file system, and associate the log file with the second index file corresponding to the log file in the distributed file system;

[0095] The first log aggregation device is used to aggregate the log records generated by each of the business processing devices based on the business request identifier in different time periods, obtain a first index temporary file corresponding to each time period, and send each of the first temporary index files to the second log aggregation device;

[0096] The second log aggregation device is used to aggregate the first temporary index files based on the service request identifier to obtain the first index file, and write the first index file into the distributed file system.

[0097] In one embodiment, the log data processing system further includes a query execution device, wherein the query execution device is used to generate a log filtering condition according to query information, wherein the query information includes at least one of a service identifier, time information, keyword information, and a service request identifier, and send the log filtering condition to the distributed file system;

[0098] The distributed file system is used to filter log records according to the log filtering conditions to obtain query results. In one embodiment, the log data processing system further includes a query request interaction device, wherein:

[0099] The query request interaction device is used to obtain a log query request, which includes at least one of a business identifier, time information, keyword information, and a business request identifier; a target query execution device is determined according to a concurrency strategy, and the target query execution device is any one of the multiple query execution devices of the log data processing system; and the log query request is routed to the target query execution device.

[0100] Specifically, the log data processing system disclosed in the embodiment of the present application and the corresponding method embodiment described above are based on the same inventive concept. For details, please refer to the method embodiment, which will not be repeated here.

[0101] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned log data processing method.

[0102] The embodiment of the present application further provides a computer-readable storage medium, which can store a plurality of instructions. The instructions can be suitable for a processor to load and execute the log data processing method described in the embodiment of the present application.

[0103] Further, Fig. 9A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown. The above device may participate in constituting or include the apparatus or system provided in the embodiment of the present application. Fig. 9 As shown, the device 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only and does not limit the structure of the electronic device. Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown.

[0104] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0105] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the above-mentioned method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned log data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0106] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0107] The display may be, for example, a touch screen liquid crystal display (LCD) that may enable a user to interact with a user interface of device 10 (or mobile device).

[0108] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of the embodiments of the present application are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] Each embodiment in the embodiments of the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0110] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program. The above program may be stored in a computer-readable storage medium, and the above storage medium may be a read-only memory, a disk or an optical disk, etc.

[0111] The above is only a preferred embodiment of the embodiment of the present application and is not intended to limit the embodiment of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiment of the present application should be included in the protection scope of the embodiment of the present application.

Claims

1. A log data processing method, characterized in that: The method is applied to a log data processing system, the log data processing system includes multiple business processing devices, a distributed file system and multiple query execution devices, each of the business processing devices corresponds to a unique log recording device, and the method includes: The log recording device corresponding to the target device stores the log records generated by the target device in an exclusive log file corresponding to the target device in the distributed file system, so that the directory structure generated by the log file itself is equivalent to an index for distinguishing different devices, the target device is any of the business processing devices running the target business process, the target business process is any process running the target business, and each of the log records includes a business request identifier; Aggregating the log records generated by each of the business processing devices based on the business request identifier to obtain a first index file, where the first index file is used to record the storage location of the log records carrying the target business request identifier, where the target business request identifier is any business request identifier in the log records generated by each of the business processing devices; Aggregating the log records based on time to obtain a second index file of the target device, the second index file being used to record a storage location of the log records generated within a first target time interval, the first target time interval being any time interval determined according to a first time segmentation strategy; associating the second index file with the log file; The target query execution device generates a log filtering condition according to the log query request, and sends the log filtering condition to the distributed file system; the distributed file system filters the log records according to the log filtering condition to obtain a query result, and the target query execution device is any query execution device among the multiple query execution devices; The log data processing system also includes a first log aggregation device and a second log aggregation device, and the method also includes: the first log aggregation device aggregates the log records generated by each of the business processing devices based on the business request identifier in each time period to obtain a first index temporary file corresponding to each time period, and sends each of the first index temporary files to the second log aggregation device; the second log aggregation device aggregates each first index temporary file based on the business request identifier to obtain a first index file, and writes the first index file into the distributed file system.

2. The method according to claim 1, characterized in that The aggregating the log records generated by each of the service processing devices based on the service request identifier to obtain a first index file includes: Aggregating the log records generated by each of the business processing devices within a second target time interval based on the business request identifier to obtain a first index temporary file, wherein the second target time interval is any time interval determined according to the second time segmentation strategy; Aggregate each of the first index temporary files based on the service request identifier to obtain the first index file.

3. The method according to claim 1, characterized in that The method further comprises: Storing the log records generated in the target device in a cache corresponding to the target device; When the data volume of the log records in the cache reaches a preset threshold, the log records in the cache are written into the log file.

4. The method according to claim 3, characterized in that The log data processing system further includes a query request interaction device, and the method further includes: the query request interaction device obtains a log query request, the log query request including at least one of a service identifier, time information, keyword information, and a service request identifier; Determine the target query execution device according to the concurrency strategy; The log query request is routed to the target query execution device.

5. A log data processing system, characterized in that: The system includes a plurality of business processing devices, a distributed file system and a plurality of query execution devices, each of the business processing devices corresponds to a unique log recording device, and the system also includes: A log file processing module, configured for a log recording device corresponding to a target device to store log records generated by the target device in an exclusive log file corresponding to the target device in the distributed file system, so that a directory structure generated by the log file itself is equivalent to an index for distinguishing different devices, the target device is any of the business processing devices running a target business process, the target business process is any process running a target business, and each of the log records includes a business request identifier; A first index file acquisition module, used to aggregate the log records generated by each of the business processing devices based on the business request identifier to obtain a first index file, wherein the first index file is used to record the storage location of the log records carrying the target business request identifier, and the target business request identifier is any business request identifier in the log records generated by each of the business processing devices; The system is further configured to aggregate the log records based on time to obtain a second index file of the target device, wherein the second index file is configured to record a storage location of the log records generated within a first target time interval, wherein the first target time interval is any time interval determined according to a first time segmentation strategy; and associate the second index file with the log file; The target query execution device generates a log filtering condition according to the log query request, and sends the log filtering condition to the distributed file system; the distributed file system filters the log records according to the log filtering condition to obtain a query result, and the target query execution device is any query execution device among the multiple query execution devices; The log data processing system also includes a first log aggregation device and a second log aggregation device. The first log aggregation device aggregates the log records generated by each business processing device based on the business request identifier in each time period to obtain a first index temporary file corresponding to each time period, and sends each first index temporary file to the second log aggregation device; the second log aggregation device aggregates each first index temporary file based on the business request identifier to obtain a first index file, and writes the first index file into the distributed file system.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement a log data processing method as claimed in any one of claims 1 to 4.

7. An electronic device, characterized in that: It includes at least one processor and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the at least one processor implements a log data processing method as described in any one of claims 1 to 4 by executing the instructions stored in the memory.

Images