A microservice request response viewing method and system based on nginx

Abstract

The present invention provides a method and system for viewing microservice request responses based on nginx, including using nginx's log_format to customize the format of nginx logs, using lua-nginx-module to embed lua into nginx, and writing the configuration of obtaining the response data returned by the back-end server in the lua coroutine; nginx receives request data and response data, and uses nginx's access_log to output the request data and response data to generate the nginx log, and outputs the nginx log to the back-end server; writes a configuration supporting syslog in nginx.conf of nginx, and then outputs the nginx log to the syslog server; stores the nginx log in the syslog server in ElasticSearch, and views the nginx log in ElasticSearch. This method does not affect the normal operation of online microservice applications, is convenient for querying data, and has strong applicability, is highly decoupled from the back-end application, and is easy to maintain and expand in the later stage. It can help developers quickly analyze and diagnose performance bottlenecks under distributed application architectures, and improve development and diagnosis efficiency in the era of microservices.

Description

Technical Field

[0001] The present invention relates to the field of industrial control technology, and in particular to a method and system for viewing microservice request responses based on nginx. Background Art

[0002] With the continuous development of science and technology, B / S architecture has replaced C / S architecture to become the mainstream model of software architecture. Management systems in various industries have begun to use B / S-based software architecture models for development, and have gradually transformed from early single-application architectures to microservice architectures. API gateway is an architecture model that has emerged along with microservices. It is used to solve the problem that microservices are too decentralized and there is no unified entrance and exit for entrance traffic management.

[0003] The existing microservice data viewing method has the following disadvantages:

[0004] 1. It is necessary to apply the http input and output modules in the microservice application to print the requested and returned JSON data, which seriously affects the operation of the program;

[0005] 2. If you use tcpdump to capture packets for analysis on the API gateway machine, the workload of troubleshooting and locating specific URL requests is huge because the amount of requested data is large. Summary of the invention

[0006] The present invention proposes a microservice request response viewing method and system based on nginx to solve the defects of the prior art mentioned above.

[0007] In one aspect, the present invention proposes a microservice request response viewing method based on nginx, the method comprising the following steps:

[0008] S1: Use nginx's log_format to customize the format of nginx logs, use lua-nginx-module to embed lua into nginx, and write the configuration for obtaining the response data returned by the backend server in the lua coroutine;

[0009] S2: nginx receives the request data and the response data, and uses the access_log of nginx to output the request data and the response data to generate the nginx log, and outputs the nginx log to the backend server;

[0010] S3: Write the configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server;

[0011] S4: storing the nginx log in the syslog server in ElasticSearch, and viewing the nginx log in ElasticSearch.

[0012] The above method uses nginx as an API gateway to view the microservice data output, and combines lua's coroutine to extract the content of the response data returned by the backend server, and writes the request data and the corresponding response data to the nginx log at the same time. Finally, syslog and ElasticSearch are used to output and store the nginx log. This method does not affect the normal operation of online microservice applications, facilitates data query, has strong applicability, is highly decoupled from the backend application, and is easy to maintain and expand later. It can help developers quickly analyze and diagnose performance bottlenecks under distributed application architectures and improve development and diagnosis efficiency in the microservice era.

[0013] In a specific embodiment, the format of the nginx log is customized by using the log_format of nginx, and the specific steps include:

[0014] The log_format configuration is used to obtain the content of the post request body in the request data, the IP address of the access client and the full path of the http request, and the escape=json field is configured to be converted into the json format and then written into the nginx log.

[0015] In a specific embodiment, the configuration of obtaining the response data returned by the backend server is written in the Lua coroutine, and the specific steps include:

[0016] In the location of nginx, define a variable resp_body for obtaining the post request body in the response data;

[0017] In the body_filter_by_lua running phase in the lua coroutine, the ngx.ctx.buffered variable of nginx is obtained in a loop, and the ngx.ctx.buffered variable is accumulated, and the accumulated data is assigned to the variable resp_body;

[0018] Configure the log file for viewing the response data in location.

[0019] Since Nginx does not have a built-in httpbody variable for obtaining post requests, this method uses the body processing phase of the lua-nginx-module in the lua coroutine process to obtain the variable.

[0020] In a specific embodiment, the configuration supporting syslog written in nginx.conf of nginx in S3 specifically includes:

[0021] Add the configuration of using syslog server to receive the nginx log in nginx.conf;

[0022] Add the syslog server address that receives the nginx log in nginx.conf.

[0023] Aggregate data to a centralized log server - syslog server to avoid storing a large amount of user request data on the API gateway and occupying the gateway's hard disk, making it easier to manage the requested data in a unified manner and reducing the workload and difficulty of operation and maintenance.

[0024] In a specific embodiment, viewing the nginx log in ElasticSearch specifically includes:

[0025] Search the nginx log by the index starting with nginx-access-log, so as to browse the request data and the response data;

[0026] By configuring kibana to search the nginx log, the request data and the response data can be browsed.

[0027] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer processor, the above method is implemented.

[0028] According to a third aspect of the present invention, a microservice request response viewing system based on nginx is proposed, the system comprising:

[0029] Nginx log format configuration module: configuration is used to customize the format of nginx logs using nginx's log_format, embed lua into nginx using lua-nginx-module, and write configurations in lua's coroutines to obtain the response data returned by the backend server;

[0030] Nginx log generation module: configured for nginx to receive request data and response data, and use nginx access_log to output the request data and response data to generate the nginx log, and output the nginx log to the backend server;

[0031] Nginx log output module: configured to write the configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server;

[0032] Nginx log storage module: configured to store the nginx log in the syslog server in ElasticSearch, and view the nginx log in ElasticSearch.

[0033] The present invention uses the log_format of nginx to customize the format of nginx log, uses lua-nginx-module to embed lua into nginx, and writes the configuration of obtaining the response data returned by the back-end server in the coroutine of lua; nginx receives request data and response data, and uses access_log of nginx to output the request data and response data to generate the nginx log, and outputs the nginx log to the back-end server; writes the configuration supporting syslog in nginx.conf of nginx, and then outputs the nginx log to the syslog server; stores the nginx log in the syslog server in ElasticSearch, and checks the nginx log in ElasticSearch. This method does not affect the normal operation of online microservice applications, is convenient for querying data, and has strong applicability, is highly decoupled from the application of the back end, and is easy to maintain and expand in the later stage. It can help developers quickly analyze and diagnose performance bottlenecks under distributed application architectures, and improve the development and diagnosis efficiency in the era of microservices. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0036] Figure 2 It is a flowchart of a method for viewing microservice request responses based on nginx according to an embodiment of the present invention;

[0037] Figure 3It is a framework diagram of a microservice request response viewing system based on nginx according to an embodiment of the present invention;

[0038] Figure 4 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] Figure 1 An exemplary system architecture 100 is shown to which an nginx-based microservice request response viewing method according to an embodiment of the present application can be applied.

[0042] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications can be installed on terminal devices 101, 102, 103, such as data processing applications, data visualization applications, web browser applications, etc.

[0044] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0045] The server 105 may be a server that provides various services, such as a background information processing server that provides support for request data displayed on the terminal devices 101, 102, and 103. The background information processing server may process the acquired request data and generate a processing result (such as an nginx log).

[0046] It should be noted that the method provided in the embodiment of the present application can be executed by the server 105, and can also be executed by the terminal devices 101, 102, 103. The corresponding device is generally set in the server 105, and can also be set in the terminal devices 101, 102, 103.

[0047] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0049] According to an embodiment of the present invention, a microservice request response viewing method based on nginx is provided. Figure 2 A flowchart of a method for viewing microservice request responses based on nginx according to an embodiment of the present invention is shown. Figure 2 As shown, the method comprises the following steps:

[0050] S201: Use nginx's log_format to customize the format of nginx logs, use lua-nginx-module to embed lua into nginx, and write the configuration for obtaining the response data returned by the backend server in the lua coroutine.

[0051] In a specific embodiment, the log_format is used to customize the output format as follows:

[0052] log_format api escape=json'$remote_addr"$request"request_body:"$request_body"resp_body:"$resp_body"';

[0053] Among them, escape=json is converted to json format, and there is no need to escape the variable content; $request_body is the content of the post request body; $resp_body is the variable of the post request body defined by ourselves; $remote_addr is the IP address of the access client; $request is the full path request of http.

[0054] In a specific embodiment, the configuration of obtaining the response data returned by the backend server is written in the Lua coroutine, and the specific steps include:

[0055] In the location of nginx, define a variable resp_body for obtaining the post request body in the response data;

[0056] In the body_filter_by_lua running phase in the lua coroutine, the ngx.ctx.buffered variable of nginx is obtained in a loop, and the ngx.ctx.buffered variable is accumulated, and the accumulated data is assigned to the variable resp_body;

[0057] Configure the log file for viewing the response data in location.

[0058] In a specific embodiment, since Nginx does not have a built-in variable for obtaining the http body of a post request, and this variable can be obtained in the body processing phase of lua-nginx-module, the response data returned by the backend server needs to be obtained using the nginx ngx.ctx.buffered variable. First, add the nginx configuration lua_need_request_bodyon, and use the following specific configuration:

[0059] 1) Define variables in nginx location, set$resp_body"";

[0060] 2) In body_filter_by_lua, ngx.ctx.buffered is obtained in a loop. The specific code is as follows:

[0061] local resp_body=string.sub(ngx.arg[1],1,1000)

[0062] ngx.ctx.buffered=(ngx.ctx.buffered or "")..resp_body

[0063] if ngx.arg[2]then

[0064] ngx.var.resp_body=ngx.ctx.buffered

[0065] end

[0066] The accumulated data is placed in the previously defined variable resp_body;

[0067] 3) Configure access_log / var / nignx / log / api.log api in location;

[0068] When there is a request for access, the detailed information of each http request can be viewed in the api.log file.

[0069] S202: nginx receives the request data and the response data, and uses the access_log of nginx to output the request data and the response data to generate the nginx log, and outputs the nginx log to the backend server.

[0070] S203: Write a configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server.

[0071] In a specific embodiment, the configuration of writing support for syslog in nginx.conf of nginx in S203 specifically includes:

[0072] Add the configuration of using syslog server to receive the nginx log in nginx.conf;

[0073] Add the syslog server address that receives the nginx log in nginx.conf.

[0074] In a specific embodiment, the following syslog configuration may be used:

[0075] access_log syslog:server=127.0.0.1:3303 api;

[0076] Among them, syslog means setting the log to be received by the syslog service; server means the syslog server address for receiving syslog logs, which uses the udp protocol by default and the port is 514. In this embodiment, we use port 3303.

[0077] S204: storing the nginx log in the syslog server in ElasticSearch, and viewing the nginx log in ElasticSearch.

[0078] In a specific embodiment, viewing the nginx log in ElasticSearch specifically includes:

[0079] Search the nginx log by the index starting with nginx-access-log, so as to browse the request data and the response data;

[0080] By configuring kibana to search the nginx log, the request data and the response data can be browsed.

[0081] In a specific embodiment, step S204 uses logstash to directly output the nginx log sent by nginx to ElasticSearch. The specific method is as follows:

[0082] Logstash uses the input plug-in to listen to port 3303 to receive data, and uses the output plug-in to directly output to ElasticSearch;

[0083] The implementation code is as follows:

[0084]

[0085] After the above settings, you can directly view the index starting with nginx-access-log in ElasticSearch, or view the request and response data by configuring kibana.

[0086] Figure 3 The framework diagram of a microservice request response viewing system based on nginx according to an embodiment of the present invention is shown. The system includes an nginx log format configuration module 301, an nginx log generation module 302, an nginx log output module 303 and an nginx log storage module 304.

[0087] In a specific embodiment, the nginx log format configuration module 301 is configured to customize the format of the nginx log using the log_format of nginx, embed lua into nginx using lua-nginx-module, and write a configuration for obtaining the response data returned by the backend server in the lua coroutine;

[0088] The nginx log generation module 302 is configured to receive request data and response data through nginx, and output the request data and response data using access_log of nginx to generate the nginx log, and output the nginx log to the backend server;

[0089] The nginx log output module 303 is configured to write a configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server;

[0090] The nginx log storage module 304 is configured to store the nginx log in the syslog server in ElasticSearch, and view the nginx log in ElasticSearch.

[0091] This system uses nginx as the API gateway to view the microservice data output, and combines lua's coroutine to extract the content of the response data returned by the backend server, and writes the request data and the corresponding response data to the nginx log at the same time. Finally, syslog and ElasticSearch are used to output and store the nginx log. This method does not affect the normal operation of online microservice applications, facilitates data query, has strong applicability, is highly decoupled from the backend application, and is easy to maintain and expand later. It can help developers quickly analyze and diagnose performance bottlenecks under distributed application architectures and improve development and diagnosis efficiency in the microservice era.

[0092] Reference below Figure 4 , which shows a schematic diagram of the structure of a computer system 400 suitable for implementing an electronic device of an embodiment of the present application. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0093] like Figure 4 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 704.

[0094] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0096] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be arranged in a processor, and the names of these units do not constitute limitations on the units themselves in certain circumstances.

[0099] An embodiment of the present invention further relates to a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the computer program is executed by a computer processor. The computer program includes program code for executing the method shown in the flowchart. It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two.

[0100] The present invention uses the log_format of nginx to customize the format of nginx log, uses lua-nginx-module to embed lua into nginx, and writes the configuration of obtaining the response data returned by the back-end server in the coroutine of lua; nginx receives request data and response data, and uses access_log of nginx to output the request data and response data to generate the nginx log, and outputs the nginx log to the back-end server; writes the configuration supporting syslog in nginx.conf of nginx, and then outputs the nginx log to the syslog server; stores the nginx log in the syslog server in ElasticSearch, and checks the nginx log in ElasticSearch. This method does not affect the normal operation of online microservice applications, is convenient for querying data, and has strong applicability, is highly decoupled from the application of the back end, and is easy to maintain and expand in the later stage. It can help developers quickly analyze and diagnose performance bottlenecks under distributed application architectures, and improve the development and diagnosis efficiency in the era of microservices.

[0101] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for viewing microservice request responses based on nginx, characterized in that: The following steps are involved: S1: Use nginx's log_format to customize the format of nginx logs, use lua-nginx-module to embed lua into nginx, and write the configuration for obtaining the response data returned by the backend server in the lua coroutine; including: define the variable resp_body for obtaining the post request body in the response data in the nginx location; loop to obtain the ngx.ctx.buffered variable of nginx in the body_filter_by_lua running phase in the lua coroutine, accumulate the ngx.ctx.buffered variable, and assign the accumulated data to the variable resp_body; configure the log file for viewing the response data in the location; S2: nginx receives the request data and the response data, and uses the access_log of nginx to output the request data and the response data to generate the nginx log, and outputs the nginx log to the backend server; S3: Write the configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server; S4: storing the nginx log in the syslog server in ElasticSearch, and viewing the nginx log in ElasticSearch.

2. The method according to claim 1, characterized in that The specific steps of using nginx's log_format to customize the format of nginx logs include: The log_format configuration is used to obtain the content of the post request body in the request data, the IP address of the access client and the full path of the http request, and escape processing is performed according to the configuration of escape=json, and the fields are converted into json format and written into the nginx log.

3. The method according to claim 1, characterized in that The configuration supporting syslog in the nginx.conf of nginx in S3 specifically includes: Add the configuration of using syslog server to receive the nginx log in nginx.conf; Add the syslog server address that receives the nginx log in nginx.conf.

4. The method according to claim 1, characterized in that: The checking of the nginx log in ElasticSearch specifically includes: Search the nginx log by the index starting with nginx-access-log, so as to browse the request data and the response data; By configuring kibana to search the nginx log, the request data and the response data can be browsed.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer processor, the method according to any one of claims 1 to 4 is implemented.

6. A microservice request response viewing system based on nginx, characterized in that: include: Nginx log format configuration module: configuration for customizing the format of nginx logs using nginx's log_format, embedding lua into nginx using lua-nginx-module, and writing configuration for obtaining response data returned by the backend server in lua's coroutine; including: defining a variable resp_body in nginx's location for obtaining the post request body in the response data; looping to obtain nginx's ngx.ctx.buffered variable in the body_filter_by_lua running phase in lua's coroutine, and accumulating the ngx.ctx.buffered variable, and assigning the accumulated data to the variable resp_body; configuring a log file in location for viewing the response data; Nginx log generation module: configured for nginx to receive request data and response data, and use nginx access_log to output the request data and response data to generate the nginx log, and output the nginx log to the backend server; Nginx log output module: configured to write the configuration supporting syslog in nginx.conf of nginx, and then output the nginx log to the syslog server; Nginx log storage module: configured to store the nginx log in the syslog server in ElasticSearch, and view the nginx log in ElasticSearch.

Images