Log file searching method and device, equipment and storage medium

Abstract

The invention provides a log file searching method and device, equipment and a storage medium, and the method comprises the steps: obtaining a log query condition in response to a log query operation, traversing log files in a log management platform, and for each traversed log file, reading an index file corresponding to the log file; wherein the index file comprises a plurality of pieces of second flag bit information generated by encrypting the file content of the log file by using a Bloom filter, and under the condition that the log query condition comprises the index information, encrypting the index information through the Bloom filter to generate first flag bit information, and if at least one piece of second flag bit information matched with the first flag bit information exists in the index file, reading a log file corresponding to the index file. According to the embodiment of the invention, the search efficiency of the log file can be improved.

Description

Technical Field

[0001] This application relates to the field of log processing technology, and more specifically, to a method, apparatus, device, and storage medium for searching log files. Background Technology

[0002] With the rapid development and widespread application of computer networks, maintaining network security has become increasingly important. Due to the current shortage of IPv4 address resources, the transition from IPv4 to IPv6 is inevitable. Since IPv4 technology has been used in networks for a long time and its hardware and software environment is relatively mature, a complete transition to IPv6 is not expected in the short term. Therefore, IPv4 and IPv6 will coexist for a considerable period. Before the complete transition to IPv6, Network Address Translation (NAT) technology has become an important method to alleviate the shortage of IPv4 addresses, and NAT technology also plays a crucial role in maintaining internal network security.

[0003] To maintain network security, real-time monitoring of network devices is crucial for ensuring network security. Log files allow for the analysis of the operational status of each device. For users accessing the network through NAT devices, the source IP address undergoes address translation via NAT, making it difficult to accurately pinpoint which host or user initiated a particular network access operation, thus reducing network security. To address this security vulnerability, NAT logging is typically used. However, for carrier-grade NAT, the daily log volume can reach terabytes, resulting in excessively large log analysis volumes and low analysis efficiency. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device, and storage medium for searching log files, in order to at least solve the problems existing in the related art.

[0005] Specifically, this application is implemented through the following technical solution: This application provides a method for finding log files, including: In response to a log query operation, retrieve the log query criteria; When the log query conditions include index information, the index information is encrypted using a Bloom filter to generate a first flag bit information; The log files in the log management platform are traversed, and for each traversed log file, the index file corresponding to the log file is read; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; If at least one second flag bit information that matches the first flag bit information exists in the index file, the log file corresponding to the index file is read.

[0006] In some implementations, the log query conditions further include filtering query information; the method further includes: Obtain the initial log file, and based on the filtering query information, perform a filtering query on the initial log file to obtain the log file corresponding to the filtering query information.

[0007] In some embodiments, the method further includes: If the log query conditions do not include index information, the log records in each log file traversed are read.

[0008] In some implementations, the filtering query information includes the query time; when the log query conditions do not include index information, reading the log records in the log file includes: If the log query conditions do not include index information, for each log file, based on the header information of the log file and the query time, a first log record matching the query time is determined from the log file; the header information includes the generation time range of the log records in the log file. Read the first log record determined from each log file.

[0009] In some implementations, after reading the first log records determined from each of the log files, the method further includes: Obtain the page display information of the client device; the page display information includes the page number and the number of log records displayed on each page; Sort the first log records according to their generation time to obtain the sorting result; Based on the sorting results, determine and display the second log record corresponding to the page display information from the first log record.

[0010] In some implementations, the index file corresponding to each log file is generated through the following steps: Obtain the log files; each log file includes multiple log records, and each log record includes multiple keywords; Determine the target keyword from the plurality of keywords; Based on the key value of the target keyword, the second flag information is generated using the Bloom filter, and the index file is generated based on the second flag information.

[0011] In some implementations, generating the second flag information using the Bloom filter based on the target keyword includes: Determine the number of target keywords in the log file, and determine and set the capacity of the Bloom filter based on the number of target keywords; Based on the target keyword, the second flag information is generated using a Bloom filter with the capacity set.

[0012] This application also provides a log file retrieval device, including: The query condition retrieval module is used to retrieve log query conditions in response to log query operations; The index information encryption module is used to encrypt the index information using a Bloom filter when the log query conditions include index information, and generate a first flag bit information. The index file reading module is used to traverse the log files in the log management platform and read the index file corresponding to each log file traversed; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; The log file search module is used to read the log file corresponding to the index file if at least one second flag bit information that matches the first flag bit information exists in the index file.

[0013] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the log file search method described in any of the foregoing embodiments.

[0014] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the log file search method described in any of the foregoing embodiments.

[0015] This application also provides a computer program product, including a computer program that, when run by a processor, performs steps such as any of the possible log file lookup methods described above.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this embodiment of the application, during the log search process, if the log query conditions include index information, the index information is encrypted using a Bloom filter to generate a first flag information, and the first flag information is matched with multiple second flag information in the index files of each log file. If at least one second flag information matching the first flag information exists in the index file, the log file corresponding to the index file is read. In this way, the query efficiency can be improved.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating an index file generation process according to an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a log file search method according to an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating another method for searching log files, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram of the structure of a log file searching device according to an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the structure of another log file searching device according to an exemplary embodiment of this application; Figure 6 This is a hardware structure diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0020] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0021] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0022] Introduction to relevant technical terms: NAT Session Log: To meet the security auditing needs of administrators, it records NAT session information (information obtained by NAT when a packet passes through a network security device, where the source IP or destination IP is translated), including IP address and port translation information, user access information, and user network traffic information.

[0023] A Bloom filter is a long binary vector and a series of random mapping functions. Bloom filters can be used to determine whether an element is in a set. Its advantages are high space efficiency and short query time; its disadvantages are a high false positive rate and difficulty in deleting elements.

[0024] In Linux systems, the core dump file mechanism for processes is as follows: the core dump file is disabled by default. It needs to be enabled by maintenance personnel or specified by setting the core dump file size in the source code of each application through setrlimit.

[0025] When a process on a network device malfunctions, the process receives a specific signal and then generates a coredump file. In this process, the network device checks the remaining disk space. If it cannot find the remaining disk space, it assumes a disk failure and cannot write the coredump file. If the remaining disk space is less than the pre-set upper limit for coredump file generation, it is considered insufficient to generate the coredump file, resulting in its loss and hindering timely fault location.

[0026] Based on the above research, this disclosure provides a method for searching log files. The method responds to a log query operation by obtaining log query conditions; then, if the log query conditions include index information, it encrypts the index information using a Bloom filter to generate a first flag bit; simultaneously, it iterates through the log files in the log management platform, and for each traversed log file, reads the index file corresponding to that log file; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; if at least one second flag bit matches the first flag bit in the index file, the log file corresponding to the index file is read.

[0027] In this embodiment, the first flag information is matched with multiple second flag information in the index file of each log file. If at least one second flag information that matches the first flag information exists in the index file, the log file corresponding to the index file is read. In this way, reading irrelevant log files can be avoided, which helps to improve query efficiency.

[0028] To facilitate understanding of this embodiment, the log management platform involved in this application will first be described in detail.

[0029] A log management platform is a system used to collect, store, analyze, query, and visualize log data from network devices. This platform is used to manage network security devices (such as firewalls, IPS, WAF, and auditing devices). The log management platform can receive log records sent by network security devices, and these log records are used to record network access data in the target network.

[0030] For example, network access data may include source IP, destination IP, port number, network protocol, network device identifier, log generation time, etc.

[0031] This allows us to trace the source of the logs and when they were generated.

[0032] This log management platform also enables hierarchical storage of logs. Specifically, logs can be stored based on log type, the network security device to which they belong, or the log generation time. Log files can be stored with their generation time as the filename, and the header information of each log file includes the earliest and latest timestamps stored in the file, as well as the total number of logs. When using generation time as the directory structure, log files can be generated in a day / hour / minute format.

[0033] After the log file is generated, an index file corresponding to the log file will also be generated. Specifically, when generating the index file, it can be generated based on the log records contained in the log file. As mentioned above, the log record can include multiple keywords, including source IP, destination IP, port number, network protocol, etc. Therefore, when generating the index file, it can be generated based on any one of the keywords (source IP, destination IP, port number, network protocol).

[0034] Specifically, the index file corresponding to each log file is generated through the following steps: (1) Obtain the log file; each log file includes multiple log records, and each log record includes multiple keywords.

[0035] Here, multiple keywords can include source IP, destination IP, port number, network protocol, etc.

[0036] (2) Determine the target keyword from the plurality of keywords.

[0037] It is understood that in this implementation, the target keyword can be determined from keywords such as source IP, destination IP, port number, and network protocol.

[0038] (3) Based on the key value of the target keyword, generate flag information using the Bloom filter, and generate the index file based on the flag information.

[0039] Once the target keyword is determined, the key value of the target keyword can be used to generate flag information based on the key value of the target keyword. For example, if the target keyword is the source IP, its key value can be 10.0.0.5. Then, a Bloom filter is used to generate flag information, and an index file is generated based on the flag information.

[0040] Specifically, when generating flag information using the Bloom filter based on the target keyword, the number of target keywords in the log file can be determined, and the capacity of the Bloom filter can be determined and set according to the number of target keywords. Based on the target keyword, the flag information is generated using the Bloom filter with the capacity set.

[0041] Taking source IPs as an example, when generating the index file, a single thread can be used to traverse each log file. For the currently traversed log file, if the file size is less than a preset threshold, then no index file needs to be generated for that log file. If the file size is not less than the preset threshold, then the total number of source IPs in the log file is determined, and the size of the Bloom filter is set based on the total number of source IPs. Then, for each source IP, a hash value is calculated using a preset number (e.g., three) of hash algorithms, and the flag bits in the Bloom filter are set based on the hash value. In this way, after each log file is stored, the content of the corresponding Bloom filter (i.e., the flag bit information) can be stored in an index file with the same file name as the log file.

[0042] In this application, the log file and the corresponding index file have the same file name, only the file extension is different. This makes it easy to directly query the corresponding index file based on the file name in subsequent query operations.

[0043] The preset threshold value in this application can be set according to actual needs, such as 5 Mbps, 10 Mbps or 100 Mbps, and is not limited here.

[0044] For example, the filename of the log file stored by day is 2024 / 1114000000.log (representing the log on November 14, 2024), and the filename of its corresponding index file is 2024 / 1114000000.index; the filename of the log file stored by hour is 20241114 / 000000.log (representing the log at 00:00 on November 14, 2024), and the filename of its corresponding index file is 20241114 / 000000.index; the filename of the log file stored by minute is 2024111400 / 0000.log (representing the log at 00:00 on November 14, 2024), and the filename of its corresponding index file is 2024111400 / 0000.index.

[0045] As mentioned above, files are stored in days, hours, and minutes. Therefore, when generating an index file, it is necessary to generate one for all log files within a period. Thus, in some implementations, the log files of the previous period can be traversed at preset time intervals, and corresponding index files can be generated for each log file of the previous period. This can improve the accuracy of the index file.

[0046] The following is combined Figure 1 The process of generating the above index files will be described. Figure 1 This is a schematic diagram illustrating an index file generation process provided for an exemplary embodiment of this application. For example... Figure 1 As shown, the log files of the previous period are traversed according to a preset time interval. For each log file, it is determined whether the file size of the log file is greater than a preset threshold. If so, the number of keywords in the log file (such as the number of source IPs) is determined. Then, the size of the Bloom filter is set according to the number of keywords, and the flag information of the Bloom filter is determined. An index file is generated based on the flag information.

[0047] Please see Figure 2 This is a flowchart illustrating a log file search method provided in an exemplary embodiment of this application. Figure 2 As shown, the log file search method in this embodiment may include the following steps S201~S204: S201: In response to a log query operation, retrieve the log query conditions.

[0048] Here, a log query page can be pre-built, and users can perform log query operations on the log query page to input log query conditions.

[0049] The log query conditions may include filtering information and / or indexing information. The filtering information is used to initially filter the range of log files and may include device identification information and / or time information.

[0050] For example, the device identification information can be "firewall device IP=192.168.1.100", and the time information can be "log time=2024-11-14".

[0051] Similarly, the index information could be "source IP=10.0.0.5", "destination IP=203.0.113.10", and "protocol type=HTTPS".

[0052] In other words, during the actual query process, users can search for logs using filtered information, indexed information, or a combination of filtered and indexed information. For example, the log query conditions could be "Device IP=192.168.1.100, Log time=2024-11-14, Source IP=10.0.0.5".

[0053] S202: If the log query conditions include index information, the index information is encrypted using a Bloom filter to generate a first flag bit information.

[0054] In this way, after obtaining the log query conditions, it can be determined whether the log query conditions include index information. If index information is included, the index information is encrypted using a Bloom filter to generate the first flag information.

[0055] The index information is encrypted using a Bloom filter, which includes encrypting the index information using a hash algorithm based on a preset number of hash values.

[0056] It should be noted that the hash algorithm used in actual application is the same as the hash algorithm used in the index file generation process.

[0057] In other implementations, if the log query criteria do not include index information, at least some log records can be determined from the traversed log files and displayed.

[0058] Optionally, if the filtering information includes time information, when determining at least some log records from the traversed log files, for each log file, based on the header information of the log file and the query time, a first log record matching the query time can be determined from the log file, and the first log record matching the query time determined in each log file can be read.

[0059] As mentioned above, the header information includes the earliest and latest times of the log records in the log file, that is, the header information includes the time range of the log records.

[0060] Furthermore, after reading the log records that match the query time in each log file, the page display information of the client device can be obtained. The page display information includes the page number and the number of log records displayed on each page. Then, the first log records are sorted according to their generation time to obtain the sorting result. Based on the sorting result, the second log record corresponding to the page display information is determined from the first log records and displayed.

[0061] Specifically, the total number of log records to be displayed can be determined based on the page number and the number of log records displayed on each page. After obtaining the sorting results, the second log record can be determined from the first log record according to the sorting results and the page display information. That is, the most recent log record and the total number of log records to be displayed are used as the final search results and displayed.

[0062] S203: Traverse the log files in the log management platform, and for each traversed log file, read the index file corresponding to the log file; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter.

[0063] In this step, the log files in the log management platform are traversed, and for each traversed log file, the index file corresponding to that log file is read based on the file name of the log file.

[0064] In some implementations, log query conditions include filtering information. Before traversing the log files in the log management platform, an initial log file can be obtained, and then the initial log file can be filtered according to the filtering information to obtain the final log file.

[0065] As mentioned above, the index file includes multiple second flag bits generated by encrypting the file content of the log file using a Bloom filter. Here, the index file of each log file currently being read is used to compare with the first flag bit information generated based on the index information.

[0066] S204: If at least one second flag bit information that matches the first flag bit information exists in the index file, the log file corresponding to the index file is read.

[0067] Here, after reading the index files corresponding to each log file, the second flag information in each index file can be compared with the first flag information. If at least one second flag information that matches the first flag information exists in the index file, it means that the log to be queried exists in the log file corresponding to the index file. Therefore, the log file corresponding to the index file can be read.

[0068] Based on the above steps, reading irrelevant log files can be avoided, which helps improve query efficiency.

[0069] Optionally, when reading the log file corresponding to the index file, the log records in the log file corresponding to the index file can be read according to the index information.

[0070] Here, since the index information may include specific content, such as the source IP, when reading the log file, only the log records related to the index information can be read, while other records do not need to be read. This avoids reading irrelevant log records.

[0071] The following is combined Figure 3 The complete process of searching for the aforementioned log files will be described. Figure 3 Another method for finding log files provided as an exemplary embodiment of this application, such as Figure 3 As shown, the method includes S301~S305: S301: Responds to a log query operation and retrieves the log query conditions.

[0072] S302: Determine whether the log query conditions include index information. If yes, proceed to step S304; otherwise, proceed to step S307.

[0073] S303: Encrypt the index information using a Bloom filter to generate the first flag information.

[0074] S304: Traverse the log files in the log management platform, and for each traversed log file, read the index file corresponding to the log file; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter.

[0075] S305: Determine whether there is at least one second flag bit information in the index file that matches the first flag bit information. If yes, proceed to step S306; otherwise, the process ends.

[0076] S306: Read the log file corresponding to the index file.

[0077] S307: Read the log records in each log file that is traversed.

[0078] Based on the above steps, the complete search process of this application is as follows: After obtaining the log query conditions, the log files are traversed, and the index file of each log file is read. At the same time, it is determined whether the log query conditions include index information. If they do, the index information is encrypted using a Bloom filter to obtain the first flag information. The first flag information is then compared with each second flag information in the index file. If at least one second flag information in the index file matches the first flag information, the log file is considered to contain the log record corresponding to the log query conditions. Therefore, the log record in the log file corresponding to the index file can be read based on the index information. Similarly, if the log query conditions do not include index information, then for each log file, the first log record matching the query time is determined from the log file based on the header information of the log file and the query time in the log query conditions, and each first log record is read.

[0079] The detailed steps for S301 to S307 have been described above and will not be repeated here.

[0080] Corresponding to the embodiments of the aforementioned log file search method, this application also provides embodiments of a log file search apparatus.

[0081] Please refer to Figure 4 This is a schematic diagram illustrating the structure of a log file searching device according to an exemplary embodiment of this application. Figure 4 As shown, the log file retrieval device 400 includes: The query condition acquisition module 410 is used to acquire log query conditions in response to log query operations; The index information encryption module 420 is used to encrypt the index information through a Bloom filter when the log query conditions include index information, and generate first flag information. The index file reading module 430 is used to traverse the log files in the log management platform and read the index file corresponding to each log file traversed; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; The log record lookup module 440 is used to read the log file corresponding to the index file if at least one second flag bit information that matches the first flag bit information exists in the index file.

[0082] Please see Figure 5 This is a schematic diagram of the structure of another log file searching device provided in an exemplary embodiment of this application. Figure 5 As shown, the log file searching device 400 further includes a log file filtering module 450, which is used for: Obtain the initial log file, and based on the filtering query information, perform a filtering query on the initial log file to obtain the log file corresponding to the filtering query information.

[0083] In some embodiments, the log record lookup module 440 is further configured to: If the log query conditions do not include index information, the log records in the log file are read.

[0084] In some implementations, the filtered query information includes the query time; the log record lookup module 440 is further configured to: If the log query conditions do not include index information, for each log file, based on the header information of the log file and the query time, a first log record matching the query time is determined from the log file; the header information includes the generation time range of the log records in the log file. Read the first log record determined from each log file.

[0085] In some embodiments, the log file retrieval device 400 further includes a log recording display module 460, which is used for: Obtain the page display information of the client device; the page display information includes the page number and the number of log records displayed on each page; Sort the first log records according to their generation time to obtain the sorting result; Based on the sorting results, determine and display the second log record corresponding to the page display information from the first log record.

[0086] In some embodiments, the log file searching device 400 further includes an index file generation module 470, which is used for: Obtain the log files; each log file includes multiple log records, and each log record includes multiple keywords; Determine the target keyword from the plurality of keywords; Based on the key value of the target keyword, the second flag information is generated using the Bloom filter, and the index file is generated based on the second flag information.

[0087] In some embodiments, the index file generation module 470 is specifically used for: Determine the number of target keywords in the log file, and determine and set the capacity of the Bloom filter based on the number of target keywords; Based on the target keyword, the second flag information is generated using a Bloom filter with the capacity set.

[0088] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0089] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0090] Corresponding to the above-described method for searching log files, this disclosure also provides a computer device, such as... Figure 6 The diagram shown is a structural schematic of a computer device provided in an embodiment of this disclosure, including: Computer device 600 includes a processor 610, an internal bus 620, memory 630, a network interface 640, and non-volatile memory 650, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 610 reads the corresponding computer program from the non-volatile memory 650 into the memory 630 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0091] The memory 630, also known as internal memory, is used to temporarily store the computational data in the processor 610, as well as the data exchanged with non-volatile memory 650 such as hard disk. The processor 610 exchanges data with non-volatile memory 650 through the memory 630.

[0092] In this embodiment, memory 630 is specifically used to store application code that executes the solution of this application, and its execution is controlled by processor 610. That is, when the computer device is running, processor 610 communicates with network interface 640, memory 630 and non-volatile memory 650 through internal bus 620, so that processor 610 executes the application code stored in memory 630 and non-volatile memory 650, and then executes the log file search method described in the above method embodiment.

[0093] Processor 610 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0094] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 600. In other embodiments of this application, the computer device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the log file search method described in the above method embodiments. The storage medium can be either volatile or non-volatile computer-readable storage.

[0096] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the log file search method in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0097] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0098] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0099] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0100] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0102] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0103] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0104] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0105] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for searching log files, characterized in that, include: In response to a log query operation, retrieve the log query criteria; When the log query conditions include index information, the index information is encrypted using a Bloom filter to generate a first flag bit information; The log files in the log management platform are traversed, and for each traversed log file, the index file corresponding to the log file is read; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; If at least one second flag bit information that matches the first flag bit information exists in the index file, the log file corresponding to the index file is read.

2. The method according to claim 1, characterized in that, The log query conditions also include filtering query information; the method further includes: Obtain the initial log file, and based on the filtering query information, perform a filtering query on the initial log file to obtain the log file corresponding to the filtering query information.

3. The method according to claim 2, characterized in that, The method further includes: If the log query conditions do not include index information, the log records in each log file traversed are read.

4. The method according to claim 3, characterized in that, The filtered query information includes the query time; when the log query conditions do not include index information, reading the log records in the log file includes: If the log query conditions do not include index information, for each log file, based on the header information of the log file and the query time, a first log record matching the query time is determined from the log file; the header information includes the generation time range of the log records in the log file. Read the first log record determined from each log file.

5. The method according to claim 4, characterized in that, After reading the first log record determined from each log file, the method further includes: Obtain the page display information of the client device; the page display information includes the page number and the number of log records displayed on each page; Sort the first log records according to their generation time to obtain the sorting result; Based on the sorting results, determine and display the second log record corresponding to the page display information from the first log record.

6. The method according to claim 1, characterized in that, The index file corresponding to each log file is generated through the following steps: Obtain the log files; each log file includes multiple log records, and each log record includes multiple keywords; Determine the target keyword from the plurality of keywords; Based on the key value of the target keyword, the second flag information is generated using the Bloom filter, and the index file is generated based on the second flag information.

7. The method according to claim 6, characterized in that, The step of generating the second flag information based on the target keyword using the Bloom filter includes: Determine the number of target keywords in the log file, and determine and set the capacity of the Bloom filter based on the number of target keywords; Based on the target keyword, the second flag information is generated using a Bloom filter with the capacity set.

8. A log file searching device, characterized in that, include: The query condition retrieval module is used to retrieve log query conditions in response to log query operations; The index information encryption module is used to encrypt the index information using a Bloom filter when the log query conditions include index information, and generate a first flag bit information. The index file reading module is used to traverse the log files in the log management platform and read the index file corresponding to each log file traversed; wherein, the index file includes multiple second flag bits generated by encrypting the file content of the log file using the Bloom filter; The log file search module is used to read the log file corresponding to the index file if at least one second flag bit information that matches the first flag bit information exists in the index file.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the log file search method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the log file search method according to any one of claims 1-7.

Images