Log processing method and device and computer readable storage medium
By setting up middleware in the microservice architecture and using configuration files to process and filter logs, the problem of third-party microservice logs cannot be saved is solved, which improves log processing efficiency and reduces database memory loss.
Patent Information
- Application Number
- CN202411014400.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
In the microservice architecture, non-open source microservice logs provided by third parties cannot be saved, affecting the performance of the microservice architecture.
By setting up middleware in the microservice architecture, using configuration files to configure the input parameters and output parameters of the input log, creating an input processing thread to process and filter the log, generating a target log and storing it in the target database.
It solves the problem that third-party microservice logs cannot be saved, improves log processing efficiency, reduces the memory loss of the database, and implements the input and output of middleware through configuration files to ensure that log information is correctly written to the database.
Smart Images

Figure CN119003462A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical field of log processing, and in particular to a log processing method, device and computer-readable storage medium. Background Art
[0002] In the microservice architecture, the alarm information, debugging information, and system status information generated by each microservice will be recorded in their respective log files through the print function. These information queries can be obtained by reading the relevant log files and displaying them in the form of strings. The most common way to save the relevant log information of the microservice architecture into the database is to write the log content into the database by calling the database read and write interface in the program. However, for microservices provided by third parties that are not open source, we cannot write the relevant log information into the database. As a result, the third-party microservice logs cannot be saved, affecting the performance of the microservice architecture. Summary of the invention
[0003] In view of the above problems, an example of the present invention provides a log processing method, device and computer-readable storage medium, which are used to solve the problem in the prior art that third-party microservice logs cannot be saved, affecting the performance of the microservice architecture.
[0004] According to one aspect of an example of the present invention, a log processing method is provided, which is applied to a microservice architecture provided with a middleware, wherein the middleware is preset with a configuration file, including: Get the input log of any microservice and configure the input parameters of the input log through the configuration file; Creating an input processing thread according to the input log, and processing the input log through the input processing thread to generate a target log; The output parameters of the target log are configured through a configuration file, and the input log is stored in a target database according to the output parameters.
[0005] In some optional embodiments, the input log includes any one or more of the alarm information, debugging information, error information, and status information of the microservice.
[0006] In some optional manners, configuring the input parameters of the input log through a configuration file includes: Set any one or more of the storage path, file name, parsing method, memory cache, and update time of the input log; wherein the file name is compatible with wildcards for matching.
[0007] In some optional manners, the processing of the input log by the input processing thread to generate a target log includes: Obtain the storage path of the input log, and regularly check the file content of the input log; Parsing the input log by regular expression to obtain field parameters in the input log; The input log is filtered according to the field parameters to generate a target log.
[0008] In some optional manners, the input log is matched by a regular expression, specifically including: Through regular expression: ^(? <time>[^]+)(? <stream>stdout!stderr)F(? <level>info):(? <log>.*), respectively obtain the time, data flow, category, and record from the log content of the input log.
[0009] In some optional manners, filtering the input log according to the field parameter to generate a target log specifically includes: The parsed input log is queried through the grep function. If the parsed input log has a preset specified string, the input log is filtered out; otherwise, it is not processed.
[0010] In some optional ways, when the file content of the input log is checked periodically, if the input log is modified, a new log line of the input log is obtained, and a matching process is performed on the new log line through a regular expression.
[0011] In some optional manners, configuring the output parameters of the target log through a configuration file, and storing the input log in a target database according to the output parameters, specifically includes: Configure the microservice name and communication port of the target log through the configuration file; A link is made to the output database according to the microservice name and the communication port, and the target log is written into the output database.
[0012] According to another aspect of an embodiment of the present invention, a log processing device is provided, the device comprising: The log input module is used to obtain the input log of any microservice and configure the input parameters of the input log through the configuration file; A log processing module, used to create an input processing thread according to the input log, and process the input log through the input processing thread to generate a target log; The log output module is used to configure the output parameters of the target log through a configuration file, and store the input log in a target database according to the output parameters.
[0013] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on the log processing device as described above, the log processing device performs the operation of the log processing method as described above.
[0014] The present invention provides a log processing method, device and computer-readable storage medium, and its beneficial effects are: the present invention sets a middleware in a microservice architecture, obtains input logs through the middleware execution, and configures input parameters of the input logs through a configuration file; creates an input processing thread according to the input log, and processes the input log through the input processing thread to generate a target log; configures output parameters of the target log through a configuration file, and stores the input log in a target database according to the output parameters.
[0015] The present invention can set the middleware input and output by means of configuration files. By parsing and processing the input logs of different microservices, the problem that the third-party microservice logs cannot be saved and affects the performance of the microservice architecture is solved. At the same time, the present invention can also filter the log information that does not meet the requirements, and finally the log information is written into the database by configuring the output of the middleware, thereby improving the log processing efficiency and reducing the memory loss of the database.
[0016] The above description is only an overview of the technical solution of the example of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the example of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings: Figure 1 The present invention provides a flow chart of a log processing method according to Embodiment 1; Figure 2 The present invention provides a schematic flow chart of processing an input log by the input processing thread to generate a target log according to Embodiment 1; Figure 3 The present invention provides a flowchart of configuring the output parameters of the target log through a configuration file and storing the input log in a target database according to the output parameters according to Embodiment 1; Figure 4 The present invention provides a schematic diagram of the structure of a log processing device according to Embodiment 2; Figure 5 A structural schematic diagram of the microservice architecture provided by the present invention is shown. DETAILED DESCRIPTION
[0018] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.
[0019] Embodiment 1: Figure 1 A first embodiment of a log processing method of the present invention is shown, which is applied to a microservice architecture with a middleware, and the middleware is preset with a configuration file, wherein the microservice architecture refers to Figure 5 The microservice can be a microservice that comes with the microservice architecture, or a non-open source microservice provided by a third party. Multiple microservices are connected to the database through middleware. The middleware is preset with a configuration file, and the input or output of the middleware can be configured through the configuration file. The log processing method of the present invention includes: 110, obtain the input log of any microservice, and configure the input parameters of the input log through the configuration file; in step 110, all microservices are connected to the output database through the middleware. The input log generated in the microservice is transmitted to the middleware through the communication line for processing. Among them, the middleware configures the input parameters of the input log through the configuration file.
[0020] 120, create an input processing thread according to the input log, and process the input log through the input processing thread to generate a target log; in step 200, when the middleware receives the input log, it can create an input processing thread in the log processing task main thread of the middleware according to the source or quantity of the input log, and when there are multiple input logs, create separate log processing threads for each. Processing the input log through the input processing thread to generate a target log includes: obtaining the storage path of the input log and periodically checking the file content of the input log; parsing the input log through regular expressions to obtain field parameters in the input log; filtering the input log according to the field parameters to generate a target log.
[0021] 130, configuring the output parameters of the target log through the configuration file, and storing the input log in the target database according to the output parameters. In step 130, configuring the output parameters of the target log through the configuration file, and storing the input log in the target database according to the output parameters, specifically includes: configuring the microservice name and communication port of the target log through the configuration file; linking with the output database according to the microservice name and communication port, and writing the target log into the output database.
[0022] The present invention sets a middleware in a microservice architecture, obtains an input log through the middleware execution, and configures the input parameters of the input log through a configuration file; creates an input processing thread according to the input log, and processes the input log through the input processing thread to generate a target log; configures the output parameters of the target log through the configuration file, and stores the input log in a target database according to the output parameters.
[0023] The present invention can set the middleware input and output by means of configuration files. By parsing and processing the input logs of different microservices, the problem that the third-party microservice logs cannot be saved and affects the performance of the microservice architecture is solved. At the same time, the present invention can also filter the log information that does not meet the requirements, and finally the log information is written into the database by configuring the output of the middleware, thereby improving the log processing efficiency and reducing the memory loss of the database.
[0024] In some optional ways, the input log includes any one or more of the alarm information, debugging information, error information, and status information of the microservice. In the microservice architecture, each microservice will record the alarm information, debugging information, error information, and status information generated by the program itself into its own log file. Various information is saved in a specific format. For example, the following two records: The first one: "2024-07-02T14:37:20.992658465Z stdout F [Nest] 8- 07 / 02 / 2024,2:37:20 PMLOG [NestApplication]Nest application successfully started +2484ms"; Article 2: "2024-07-02T14:37:21.016392417Z stdout F info: GraphQL Systemserver running at http: / / localhost:8013 / graphql {"context":{"locationType":"System","logType":2,"severity":4,"transition":3}}".
[0025] In some optional ways, configuring the input parameters of the input log through a configuration file includes: setting any one or more of the storage path, file name, parsing method, memory cache, and update time of the input log; wherein the file name is compatible with wildcards. In this embodiment, the present invention configures the storage path, file name, parsing method, memory cache, and update time of the input log according to the characteristics of the input log through a configuration file preset by the middleware.
[0026] For some alternatives, see Figure 2 , processing the input log through the input processing thread to generate a target log; specifically including: 210, obtaining the storage path of the input log and regularly checking the file content of the input log: In step 210, the storage path of the input log can be obtained by querying the input parameters of the input log. At the same time, the file content of the input log can be regularly checked according to the update time of the input log.
[0027] 220, parse the input log by regular expression to obtain field parameters in the input log; in step 220, match the input log by regular expression, specifically including: by regular expression: ^(? <time>[^]+)(? <stream>stdout!stderr)F(? <level>info): (? <log>.*), respectively, from the log content of the input log, obtain the time, data stream, category, and record. Among them, the regular expression can be stored in the memory or register in the middleware, and the regular expression is called by the middleware parser to match and parse the input log, so as to obtain the field parameters in the input log. time is time, stream is data stream, level is category, and log is record. In addition, the time format is %Y-%m-%dT %H:%M:%S.%L%z, such as 2000-1-1 8:00:00.00.
[0028] In this embodiment, ^(? <time>[^]+)(? <stream>stdout!stderr)F(? <level>info):(? <log>.*) is a regular expression symbol. Through the computer language of the Linux operating system, relevant data about time, stream, level, and log can be automatically obtained. Among them, time, stream, level, and log can be obtained by splitting through specific symbols in the expression. For example, through (? <time>[^]+) query to get time=2000-1-1 8:00:00.00. Similarly, stream, level, and log can all be obtained from the above regular expression.
[0029] 230, filtering the input log according to the field parameters to generate a target log. In step 230, filtering the input log according to the field parameters to generate a target log specifically includes: querying the parsed input log through the grep function, if the parsed input log has a preset specified string, filtering the input log; otherwise, not processing. Specifically, the parsed log information can be filtered through a configuration file, and the filtering method is to use the common Linux command "grep a certain string". If it matches, it needs to be processed, and the log information that does not meet the rules is discarded.
[0030] In addition, when the file content of the input log is checked regularly, if the input log is modified, a newly added log line of the input log is obtained, and a matching process is performed on the newly added log line through a regular expression.
[0031] For some alternatives, see Figure 3 , configuring the output parameters of the target log through a configuration file, and storing the input log in a target database according to the output parameters, specifically including: 310, configure the microservice name and communication port of the target log through the configuration file; in a specific example, the microservice name is: FLUENT_ELASTICSEARCH_HosT, and the communication port value is: "elasticsearch". Configure the interface for the target log microservice through the configuration file of the middleware.
[0032] 320, link with the output database according to the microservice name and communication port, and write the target log to the output database. In this embodiment, configure the output of the middleware. Set the output of common database systems, such as MongoDB database and Elasticsearch database, and configure the database microservice name and communication port. Then the middleware can automatically write data to the database according to different database insertion commands. For example, the RESTful API to insert a record into the elasticsearch database: PUT alarm { "time":"2024-07-02T14:37:20.992658465Z", "stream": "stdout", "level": "warning", "log":"This is a manual add message." }.
[0033] In the present invention, the log processing method mainly realizes the function of the middleware, and the middleware input and output can be set by means of a configuration file. The log content parsing method can be configured by means of a formal expression, and the log information that does not meet the requirements can be filtered. Finally, the output of the middleware can be configured, and the log information is finally written into the database.
[0034] This middleware solution can be applied to all microservice systems. The middleware itself is also a microservice and is monitored by the microservice management software. The input, parsing, filtering, and output settings of this middleware can all be configured in the usual way. Therefore, it is highly compatible and easy to use.
[0035] Embodiment 2: According to another aspect of an embodiment of the present invention, a log processing device is provided. Figure 4 , the device includes: 410, a log input module, used to execute step 110 in embodiment 1; 420, a log processing module, used to execute step 120 in embodiment 1; 430, a log output module, used to execute step 120 in embodiment 1. In this embodiment, the present invention sets a middleware in the microservice architecture, obtains the input log through the middleware execution, and configures the input parameters of the input log through the configuration file; creates an input processing thread according to the input log, and processes the input log through the input processing thread to generate a target log; configures the output parameters of the target log through the configuration file, and stores the input log in the target database according to the output parameters. The present invention can set the middleware input and output by means of a configuration file. By parsing and processing the input logs of different microservices, the problem that the third-party microservice log cannot be saved and affects the performance of the microservice architecture is solved. At the same time, the present invention can also filter the log information that does not meet the requirements, and finally the output of the middleware can be configured, and the log information is finally written into the database, which improves the log processing efficiency and reduces the memory loss of the database.
[0036] Embodiment 3: According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium. When the executable instruction is executed on the log processing device as described above, the log processing device performs the operation of the log processing method as described above.
[0037] The present invention performs the following steps through executable instructions of a computer-readable storage medium: 110, obtaining an input log of any microservice, and configuring input parameters of the input log through a configuration file; 120, creating an input processing thread according to the input log, and processing the input log through the input processing thread to generate a target log; 130 , configuring output parameters of the target log through a configuration file, and storing the input log in a target database according to the output parameters.
[0038] By executing the above steps, the present invention can set the middleware input and output by means of configuration files. By parsing the input logs of different microservices, the problem that the third-party microservice logs cannot be saved and affects the performance of the microservice architecture is solved. At the same time, the present invention can also filter the log information that does not meet the requirements, and finally the log information can be written into the database by configuring the output of the middleware, thereby improving the log processing efficiency and reducing the memory loss of the database.
[0039] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.
[0040] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.
[0041] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.
[0042] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.< / time> < / log> < / level> < / stream> < / time> < / log> < / level> < / stream> < / time> < / log> < / level> < / stream> < / time>
Claims
1. A log processing method, characterized in that: The method is applied to a microservice architecture provided with a middleware, wherein the middleware is preset with a configuration file, including: Get the input log of any microservice and configure the input parameters of the input log through the configuration file; Creating an input processing thread according to the input log, and processing the input log through the input processing thread to generate a target log; The output parameters of the target log are configured through a configuration file, and the input log is stored in a target database according to the output parameters.
2. The log processing method according to claim 1, characterized in that: The input log includes any one or more of the alarm information, debugging information, error information, and status information of the microservice.
3. The log processing method according to claim 1, characterized in that: The step of configuring the input parameters of the input log through a configuration file includes: Set any one or more of the storage path, file name, parsing method, memory cache, and update time of the input log; wherein the file name is compatible with wildcards for matching.
4. The log processing method according to claim 3, characterized in that: The step of processing the input log by the input processing thread to generate a target log comprises: Obtain the storage path of the input log, and regularly check the file content of the input log; Parsing the input log by regular expression to obtain field parameters in the input log; The input log is filtered according to the field parameters to generate a target log.
5. The log processing method according to claim 4, characterized in that: The input log is matched by a regular expression, specifically including: Through regular expression: ^(? <time>[^]+)(? <stream>stdout!stderr)F(? <level>info): (? <log> .*), respectively obtain the time, data flow, category, and record from the log content of the input log.< / log> < / level> < / stream> < / time> 6. The log processing method according to claim 4, characterized in that: The filtering the input log according to the field parameter to generate the target log specifically includes: The parsed input log is queried through the grep function. If the parsed input log has a preset specified string, the input log is filtered out; otherwise, it is not processed.
7. The log processing method according to claim 4, characterized in that: When the file content of the input log is checked regularly, if the input log is modified, a new log line of the input log is obtained, and a matching process is performed on the new log line through a regular expression.
8. The log processing method according to claim 1, characterized in that: The step of configuring the output parameters of the target log through a configuration file and storing the input log in a target database according to the output parameters specifically includes: Configure the microservice name and communication port of the target log through the configuration file; A link is made to the output database according to the microservice name and the communication port, and the target log is written into the output database.
9. A log processing device, characterized in that: The device comprises: The log input module is used to obtain the input log of any microservice and configure the input parameters of the input log through the configuration file; A log processing module, used to create an input processing thread according to the input log, and process the input log through the input processing thread to generate a target log; The log output module is used to configure the output parameters of the target log through a configuration file, and store the input log in a target database according to the output parameters.
10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the log processing device according to claim 9, the log processing device executes the operation of the log processing method according to any one of claims 1 to 8.
