Log data storage and processing method and device, electronic equipment and storage medium

Abstract

The invention provides a log data storage and processing method and device, electronic equipment and a storage medium. Small log files generated by a data request are collected; converting the small log file into a large log file, and processing the large log file through a log collection tool; and writing the processed log data into the shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk, the log processing efficiency is improved by converting the small log file into the large log file, and the log processing efficiency is improved. Log collection, storage and processing are separated from gateway application through the shared disk, the load of a server where the gateway application is located is reduced, and the stability and performance of the gateway application are improved.

Description

Technical Field

[0001] The present invention relates to the field of log storage technology, and in particular to a log data storage and processing method, device, electronic device and storage medium. Background Art

[0002] Log collection and storage are of great significance for system problem troubleshooting. In related technologies, log collection tools such as ELK Stack (Elasticsearch, Logstash, Kibana), Splunk, and Fluentd are used to collect and store logs. ELK Stack is a popular open source log management solution. Elasticsearch is used to store and search log data, Logstash is used for log collection, filtering and conversion, and Kibana is used for log visualization and analysis. Splunk is a commercial log management and analysis platform with powerful search and visualization capabilities. It can handle large-scale log data and provide real-time monitoring and analysis. Fluentd is an open source log collector that can collect log data to a central location and send it to a variety of storage backends, such as Elasticsearch. However, these tools store logs on the local disk, which is on the same machine as the gateway. Since each data request generates 4 small files, an average of 2000w+ small files are generated every day. Processing these small files is time-consuming and resource-intensive. Therefore, storing logs on the local disk and collecting and processing logs will occupy system resources, and gateway applications are also easily affected. Summary of the invention

[0003] The present invention provides a log data storage and processing method, device, electronic device and storage medium, which are used to solve the defects that the log collection, storage and processing method will occupy system resources when collecting and processing logs, and gateway applications are also easily affected.

[0004] The present invention provides a log data storage and processing method, comprising: Collect small log files generated by data requests; Convert the small log file into a large log file, and process the large log file using a log collection tool; The processed log data is written into the shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk.

[0005] According to the log data storage and processing method provided by the present invention, the small log file generated by the data collection request includes: Develop log collection scripts corresponding to data request format logs; The small log files generated by the data request are cached, and after the log collection condition is met, the cached small log files are collected uniformly.

[0006] According to the log data storage and processing method provided by the present invention, converting the small log file into a large log file includes: Develop a python script, define global variables in the python script, and the global variables include an acquisition path of a small log file and a generation path of a large log file; Define a producer class, which is used to generate path information of small log files and put it into a conversion queue; Define an operation method corresponding to the producer class, in which the generation time of the small log file is obtained at preset intervals, and whether there is a small log file that meets the merging condition is determined according to the generation time of the small log file and the acquisition path of the small log file; If so, the file path and generation time of the small log files that meet the merging conditions are put into the conversion queue; Define a consumer class, where the consumer class is used to retrieve the path information of the small log file from the conversion queue; Define an operation method corresponding to the consumer class, in which the multiple small log files are merged into a large log file; Create a main function, and call the corresponding running methods of the producer class and the consumer class in the main function.

[0007] According to the log data storage and processing method provided by the present invention, the merging condition is determined according to the number of command line parameters; If the number of parameters is 3, the merging condition is determined according to the specified date and specific time; If the number of parameters is 4, the merging condition is determined according to the specified date and time range.

[0008] According to the log data storage and processing method provided by the present invention, the merging of multiple small log files into a large log file includes: Setting the output file and defining a traversal function in a python script, wherein the traversal function is used to traverse the directories in the conversion queue; During the traversal process, it is determined whether there is a log file in the directory. If so, the file name is written into the output file, and the file content is read line by line and written into the output file.

[0009] According to the log data storage and processing method provided by the present invention, the large log file is processed by a log collection tool, including: Use the ELK Stack tool to store and search large log files, and filter and convert the large log files; and / or

[0010] Use the Fluentd tool to collect large log files into a central location and send them to a variety of storage backends.

[0011] According to the log data storage and processing method provided by the present invention, after writing the processed log data into the shared disk, the method further includes: The log data of the shared disk is compressed and archived regularly to a tape library or cold storage service.

[0012] The present invention also provides a log data storage and processing device, comprising: The collection module is used to collect small log files generated by data requests; A conversion module, used to convert the small log file into a large log file, and process the large log file through a log collection tool; The writing module is used to write the processed log data into the shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the log data storage and processing method as described in any one of the above items is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the log data storage and processing methods described above is implemented.

[0015] The log data storage and processing method, device, electronic device and storage medium provided by the present invention collect small log files generated by data requests; convert the small log files into large log files, and process the large log files through a log collection tool; write the processed log data into a shared disk, so that a server where a gateway is located and / or a server not including the gateway processes the log data in the shared disk. By converting small log files into large log files, the efficiency of log processing is improved, and log collection, storage and processing are separated from gateway applications through a shared disk, which reduces the load of the server where the gateway application is located and improves its stability and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a log data storage and processing method provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a log data storage and processing method provided by an embodiment of the present invention; Figure 3 It is a functional structure diagram of a log data storage and processing device provided by an embodiment of the present invention; Figure 4 It is a functional structure diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Figure 1 A flow chart of a log data storage and processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the log data storage and processing method provided by the embodiment of the present invention includes: Step 101, collect small log files generated by data requests; Step 102: convert the small log file into a large log file, and process the large log file using a log collection tool; Step 103: write the processed log data into a shared disk, so that the server where the gateway is located and / or the server excluding the gateway processes the log data in the shared disk.

[0020] The log storage of traditional gateway applications and the gateway applications run on the same machine, resulting in competition for system resources and reducing the stability and performance of the gateway applications. The embodiments of the present invention use a shared disk of a network attached storage (NAS) to store the logs of the gateway applications. The log data is written to the NAS shared disk and processed on another server, thereby minimizing the impact on the gateway application and ensuring its normal operation.

[0021] In the embodiment of the present invention, Figure 2 As shown in the figure, the NAS shared disk is used as a network storage device and is connected to the gateway application server and the log processing server through the network. The log data generated by the gateway application is written to the NAS shared disk instead of being stored on the local disk. This can reduce the occupation of the gateway application server resources, thereby improving its performance and stability. At the same time, another server is configured as a log processing server, which is responsible for reading log data from the NAS shared disk and performing real-time or batch processing. The resources of the gateway application server can be liberated, allowing it to better perform its main functions without being affected by the log processing process. The separation of log storage and processing is achieved, which minimizes the impact on the gateway application and improves the overall stability and performance of the system.

[0022] The traditional log processing method generates 4 small files for each data request, resulting in an average of 20 million+ small files per day. Processing these small files is time-consuming and resource-intensive. Therefore, the logs are stored on the local disk. Collecting and processing logs will occupy system resources and the gateway application is also easily affected.

[0023] The log data storage and processing method provided by the embodiment of the present invention collects small log files generated by data requests; converts the small log files into large log files, and processes the large log files through a log collection tool; writes the processed log data into a shared disk, so that a server where a gateway is located and / or a server not including the gateway processes the log data in the shared disk. By converting small log files into large log files, the efficiency of log processing is improved, and log collection, storage and processing are separated from gateway applications through the shared disk, which reduces the load of the server where the gateway application is located and improves its stability and performance.

[0024] Based on any of the above embodiments, the small log file generated by the data collection request includes: Step 201: Develop a log collection script corresponding to the data request format log; Step 202: cache the small log files generated by the data request, and after the log collection condition is met, collect the cached small log files in a unified manner.

[0025] By developing scripts, the problem that logs cannot be collected directly using open source log collection tools is solved. According to the application scenario, a script that meets the use scenario of the open source tool is customized and developed to utilize the open source tool with the highest efficiency. Since the use of the open source log collection tool is completed through a program, the resource consumption is large, while the embodiment of the present invention is completed using a script, the resource consumption is small, and the application scenario of the embodiment of the present invention is a gateway application log collection scenario, and most of the log data is a relatively simple format such as gateway call records and messages, so the log format can be customized to collect these gateway application logs in a unified manner. In addition, the open source log collection tools are all collected in real time, which will occupy system resources and bring unnecessary overhead to the system. In the embodiment of the present invention, the small log files generated by the data request are first cached, and after the log collection conditions are met, the cached small log files are uniformly collected, which can save system resources and does not affect the normal operation of the application.

[0026] Based on any of the foregoing embodiments, converting the small log file into a large log file includes: Step 301: Develop a python script, define global variables in the python script, and the global variables include an acquisition path of a small log file and a generation path of a large log file; For example, the original log path source_payload, the original log generation path target_payload, and the log storage path after processing is completed log_root_path.

[0027] Step 302: define a producer class, which is used to generate path information of small log files and put them into a conversion queue; Step 303: define an operation method corresponding to the producer class, in which the generation time of the small log file is obtained at preset intervals, and whether there is a small log file that meets the merging condition is determined according to the generation time of the small log file and the acquisition path of the small log file; For example, the Producer class is defined, which inherits from threading.Thread. This class is used to generate the path information of the log file and put it into the queue. In the run method, the current time is obtained every 60 seconds, and whether there is a file that meets the conditions is determined based on the time and the original log path. If there is, the file path, date, time, target directory and other information are put into the queue and the log is recorded.

[0028] In an embodiment of the present invention, the merging condition is determined according to the number of command line parameters; If the number of parameters is 3, the merging condition is determined according to the specified date and specific time; If the number of parameters is 4, the merging condition is determined according to the specified date and time range.

[0029] Step 304: If yes, the file path and generation time corresponding to the small log file that meets the merging condition are put into the conversion queue; Step 305: define a consumer class, wherein the consumer class is used to retrieve the path information of the small log file from the conversion queue; Step 306: define an operation method corresponding to the consumer class, in which the multiple small log files are merged into a large log file; For example, the Consumer class is defined, which also inherits from threading.Thread. This class is used to retrieve the path information of the log file from the queue and perform file merging operations. In the run method, the file path and other information are obtained from the queue, the target file is opened, the traverse_dir2 function is called to merge the file content into the target file, and the log is recorded.

[0030] Step 307: Create a main function, and call the corresponding operation methods of the producer class and the consumer class in the main function.

[0031] In the embodiment of the present invention, the main function is the entry point of the program, creates the Producer thread and the Consumer thread, and starts the Producer thread and the Consumer thread. In the if __name__ == '__main__' part, the log format and output file are set, and the main function is called to execute the program.

[0032] The embodiment of the present invention implements the screening and merging of original log files through the producer-consumer mode, converts small files into large files, and improves the efficiency of log processing.

[0033] In the embodiment of the present invention, merging the multiple small log files into a large log file includes: Setting the output file and defining a traversal function in a python script, wherein the traversal function is used to traverse the directories in the conversion queue; During the traversal process, it is determined whether there is a log file in the directory. If so, the file name is written into the output file, and the file content is read line by line and written into the output file.

[0034] For example, the traverse_dir2 function is defined to traverse the files in the directory. If it is a file, the file name is written to the output file, and the file content is read line by line and written to the output file.

[0035] The embodiment of the present invention explores and summarizes the rules of the generation and storage paths of small log files, and combines the log generation characteristics of the product to develop a python script to realize the merging of logs, convert small files into large files, and then process the large files through open source tools. There is no need to make cumbersome judgments on the size of small files. It is only necessary to merge the small files in the same folder to make them into a large file. This greatly simplifies the operation process and avoids the complex calculations and potential errors that may be caused by judging the file size. For example, when processing a large number of randomly generated small files, the traditional method may need to analyze the file size one by one, which consumes a lot of time and computing resources, while the embodiment of the present invention can directly skip this link and quickly perform the merging operation. In addition, there is no need to upload and transfer files, and the processing is directly carried out in the original address directory, so the operation is more efficient. The time consumption and possible data loss risk caused by the transmission of files between different locations are reduced. Taking a large data center as an example, if the traditional upload and transfer method is adopted, it may be limited by the network bandwidth, resulting in extended processing time, while direct processing in situ can make full use of local resources and quickly complete the task.

[0036] Based on any of the foregoing embodiments, the processing of the large log file by a log collection tool includes: Use the ELK Stack tool to store and search large log files, and filter and convert the large log files; and / or

[0037] Use the Fluentd tool to collect large log files into a central location and send them to a variety of storage backends.

[0038] It should be noted that, in the embodiment of the present invention, the log collection tool is an open source log collection tool. The present invention does not limit the specific tool used, and those skilled in the art can select it according to actual needs.

[0039] Based on any of the above embodiments, after writing the processed log data into the shared disk, the method further includes: The log data of the shared disk is compressed and archived regularly to a tape library or cold storage service.

[0040] The embodiments of the present invention can free up storage space, reduce storage costs, and retain historical log data to meet compliance and audit requirements.

[0041] The log data storage and processing method provided by the embodiment of the present invention reduces the load on the server where the gateway application is located and improves its stability and performance by separating log collection, storage and processing from the gateway application. The gateway application can focus on its core functions without being disturbed by the log processing process. In addition, storing the logs on shared storage facilitates centralized management and backup of the logs. NAS shared storage provides a reliable data storage solution to ensure the security and integrity of log data. The storage capacity can be flexibly expanded according to demand to meet the growing log storage needs without the need to upgrade the hardware of the server where the gateway application is located. NAS devices usually support hot-swappable hard drives and online expansion of storage space, providing flexible scalability.

[0042] The log data storage and processing device provided by the present invention is described below. The log data storage and processing device described below and the log data storage and processing method described above can be referred to each other.

[0043] Figure 3 A schematic diagram of a log data storage and processing device provided by an embodiment of the present invention, such as Figure 3 As shown, the log data storage and processing device provided by the embodiment of the present invention includes: Collection module 301, used to collect small log files generated by data requests; A conversion module 302, used to convert the small log file into a large log file, and process the large log file through a log collection tool; The writing module 303 is used to write the processed log data into the shared disk, so that the server where the gateway is located and / or the server excluding the gateway processes the log data in the shared disk.

[0044] The log data storage and processing device provided by the embodiment of the present invention collects small log files generated by data requests; converts the small log files into large log files, and processes the large log files through a log collection tool; writes the processed log data into a shared disk, so that a server where a gateway is located and / or a server not including the gateway processes the log data in the shared disk. By converting small log files into large log files, the efficiency of log processing is improved, and log collection, storage and processing are separated from gateway applications through the shared disk, which reduces the load of the server where the gateway application is located and improves its stability and performance.

[0045] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the server may include: a processor 410, a communications interface 420, a memory 430 and a communications bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communications bus 440. The memory 430 includes a computer program, an operating system and acquired data, and the processor 410 may call the logic instructions in the memory 430 to execute a log data storage and processing method, which includes: collecting small log files generated by data requests; converting the small log files into large log files, and processing the large log files through a log collection tool; writing the processed log data into a shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk.

[0046] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0047] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the log data storage and processing method provided by the above-mentioned methods. The method includes: collecting small log files generated by data requests; converting the small log files into large log files, and processing the large log files through a log collection tool; writing the processed log data to a shared disk, so that the server where the gateway is located and / or the server that does not include the gateway processes the log data in the shared disk.

[0048] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0049] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for storing and processing log data, It is characterized in that include: Collect small log files generated by data requests; Convert the small log file into a large log file, and process the large log file using a log collection tool; The processed log data is written into the shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk.

2. The log data storage and processing method according to claim 1, It is characterized in that The small log file generated by the data collection request includes: Develop log collection scripts corresponding to data request format logs; The small log files generated by the data request are cached, and after the log collection condition is met, the cached small log files are collected uniformly.

3. The log data storage and processing method according to claim 1, It is characterized in that The converting the small log file into a large log file comprises: Develop a python script, define global variables in the python script, and the global variables include an acquisition path of a small log file and a generation path of a large log file; Define a producer class, which is used to generate path information of small log files and put it into a conversion queue; Define an operation method corresponding to the producer class, in which the generation time of the small log file is obtained at preset intervals, and whether there is a small log file that meets the merging condition is determined according to the generation time of the small log file and the acquisition path of the small log file; If so, the file path and generation time of the small log files that meet the merging conditions are put into the conversion queue; Define a consumer class, where the consumer class is used to retrieve the path information of the small log file from the conversion queue; Define an operation method corresponding to the consumer class, in which the multiple small log files are merged into a large log file; Create a main function, and call the corresponding running methods of the producer class and the consumer class in the main function.

4. The log data storage and processing method according to claim 3, It is characterized in that The merging condition is determined according to the number of command line parameters; If the number of parameters is 3, the merging condition is determined according to the specified date and specific time; If the number of parameters is 4, the merging condition is determined according to the specified date and time range.

5. The log data storage and processing method according to claim 1, It is characterized in that The merging of multiple small log files into a large log file includes: Setting the output file and defining a traversal function in a python script, wherein the traversal function is used to traverse the directories in the conversion queue; During the traversal process, it is determined whether there is a log file in the directory. If so, the file name is written into the output file, and the file content is read line by line and written into the output file.

6. The log data storage and processing method according to claim 1, It is characterized in that The processing of the large log file by using a log collection tool includes: Use the ELK Stack tool to store and search large log files, and filter and convert the large log files; and / or Use the Fluentd tool to collect large log files into a central location and send them to a variety of storage backends.

7. The log data storage and processing method according to claim 1, It is characterized in that After the processed log data is written into the shared disk, the method further includes: The log data of the shared disk is compressed and archived regularly to a tape library or cold storage service.

8. A log data storage and processing device, It is characterized in that include: The collection module is used to collect small log files generated by data requests; A conversion module, used to convert the small log file into a large log file, and process the large log file through a log collection tool; The writing module is used to write the processed log data into the shared disk, so that the server where the gateway is located and / or the server not including the gateway processes the log data in the shared disk.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the log data storage and processing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the log data storage and processing method according to any one of claims 1 to 7 is implemented.

Images