Micro-service architecture non-intrusive auditing method

Abstract

The invention provides a non-intrusive auditing method of a micro-service architecture. The complexity and the intrusiveness of an auditing function are reduced by introducing a public auditing service. A lightweight proxy technology of a micro-service governance framework is utilized, lightweight proxy interception service communication flow is deployed, request data meeting conditions are put into a message queue, and auditing service consumes and generates auditing records. And a distributed link tracking system is integrated to track a request path. And the auditing service matches the operation information according to a preset rule, and records conforming to the rule are converted into auditing events and stored. According to the method, modification of service codes can be reduced, and the purity of the main business process is kept; the dependence on a specific framework or programming language is reduced; the system stability is improved because the fault of the auditing service does not influence the operation of the main service; the auditing function is flexibly expanded or modified without adjusting the micro-service; the auditing rule is allowed to be updated independent of the business service, and real-time updating is supported without affecting business operation.

Description

Technical Field

[0001] The present invention relates to the technical field of computer system development, and in particular to a non-intrusive audit method for a microservice architecture. Background Art

[0002] Auditing is a key link in the microservice architecture. It can help users track operation requests and their results for easy query, auditing, and backtracking. Auditing is usually used in a variety of scenarios such as security analysis, resource change tracking, and fault location.

[0003] In a log audit method under a microservice architecture (CN112181927A), a public SDK package is provided as a public component for log output, and its functions include: service link tracking, generating and recording a unique identifier during a service request; log output, automatically modifying the log format according to different types of logs and outputting it to a message queue. However, the above method adds an audit function to an already running microservice architecture system, especially a function like auditing that involves multiple levels (user interface, business logic, data access, etc.), which requires extensive code changes and requires the transformation of almost all business-related microservices, making the transformation difficult.

[0004] In a three-tier audit method in a microservice distributed scenario (CN113986895A), a method is provided to obtain link information and terminal information using the springboot framework, and bind the link information and terminal information and store them in a database; the database proxy component records the operation information of the database and associates it with the link information. However, microservices may be built by different development teams using different programming languages ​​and technology stacks, which increases the complexity of integrating audit functions.

[0005] In summary, designing a non-intrusive, cross-programming language audit solution will greatly simplify the transformation work of adding audit functions to microservice systems. Summary of the invention

[0006] The present invention aims to solve the problems of strong intrusiveness and difficulty in modification when adding audit functions to an already running microservice architecture system, as well as the complexity of adding audit functions to a multi-programming language system, and to provide a non-intrusive, cross-programming language audit solution.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A non-intrusive audit method for microservice architecture, deploying a lightweight agent to intercept request / response information of microservices and send audit messages; a distributed link tracking system obtains request links between services according to a unique ID of the request and records them in a full-text search engine; audit services subscribe to and consume audit messages; log information obtained according to the unique ID of the request and audit messages generate original audit events; original audit events are matched with audit rules, and after successful matching, audit events are generated and recorded in a database; cache is updated when users update or add audit rules.

[0009] As a preference, service mesh technology is used to further reduce the intrusiveness to the microservice architecture system. The service mesh deploys a lightweight agent on each service container to implement its functions. These agents are responsible for intercepting and controlling all network traffic in and out of the service, and placing qualified traffic information into the message queue.

[0010] Preferably, the service mesh is integrated with a distributed link tracing system, allowing users to trace the complete path of a request between services; the lightweight proxy injects special tracing identifiers into each request passing through it, and the distributed link tracing system uses these identifiers to reconstruct the request flowchart, providing a detailed view of the request from entry to exit.

[0011] Preferably, a single service in the microservice writes the link number, response status, and error information into a log.

[0012] Preferably, each service request in the system is intercepted and an operation audit is automatically generated.

[0013] Preferably, the audit message includes: operation time, operator, operation information, service name, request link number, response status, and error information.

[0014] Preferably, the operation information is generated by matching audit rules; the audit rules include interface path, request method, query parameter conditions, request body conditions, operation name template, and template parameter name.

[0015] Preferably, the audit rule matching includes matching of request method, query parameter conditions, and request body conditions. When the matching is completed, the audit operation is generated by filling in template parameters according to the operation name template.

[0016] Preferably, the cross-programming language audit service support is used in systems of multiple programming languages, and the cross-programming language is implemented based on a lightweight agent.

[0017] The present invention uses a microservice governance framework to intercept and control all network traffic in and out of the service by deploying a lightweight agent (called Sidecar agent) next to each service instance. When a user initiates a request to a service A in the system, the request first passes through the lightweight agent. The lightweight agent is responsible for intercepting the request, extracting information such as the request path and method, and intercepting the response information after the service is processed, obtaining the HTTP status code, and then returning the response to the user. At the same time, a distributed link tracking system is used to inject a link number, i.e., traceId, into each request of the lightweight agent, and the relevant request information is stored in the full-text retrieval engine. The lightweight agent also sends the request and response information and traceId to the message queue.

[0018] The audit service subscribes to and consumes these messages, and uses traceId to retrieve request link log information from the full-text search engine. The original audit event generator consumes these messages and combines them with the link number, response status, and error information in the log information to form original audit events. Next, these events are matched with audit rules, including matching of request methods, interface paths, query parameters, and request body conditions to determine the corresponding audit rules.

[0019] If all audit rules are stored in the cache, the rule matcher directly obtains them from it; otherwise, they are retrieved from the database and stored in the cache. The rule matcher generates audit events based on the matched rules and extracts the corresponding parameter values ​​from the request information. These values ​​are used to fill the placeholders in the audit rule template to generate audit operation records. Finally, these records are stored in the database to facilitate auditing and monitoring operations in the microservice architecture.

[0020] Developers can add or update audit rules, which will be stored in the database and cache. The updated audit rules can take effect in real time without restarting the system.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention provides a non-intrusive audit method for a microservice architecture, which adopts an audit system based on the Sidecar mode. The system intercepts request and response information between services through a lightweight proxy, and sends this data asynchronously to a message queue. Subsequently, a dedicated audit service consumes this information from the message queue, extracts relevant operation details and traceId, and generates preliminary audit records. These records are matched according to preset audit rules, and only records that meet the rules will be converted into final audit events and stored in the database.

[0023] 1. It can effectively reduce the modification of service codes, ensure the purity of the main business process, and avoid interference with business logic due to the addition of audit functions.

[0024] 2. Using lightweight agents can reduce dependence on specific frameworks or languages.

[0025] 3. Since the audit logic is separated from the business logic, the failure of the audit service will not affect the normal operation of the main service, thereby improving the stability of the system. At the same time, asynchronous processing with the help of message queues can effectively buffer sudden traffic and reduce the burden on the system.

[0026] 4. Audit functionality can be flexibly expanded or modified without adjusting the microservice itself.

[0027] 5. Allows audit rules to be updated independently of business services, and audit rules can be updated in real time without interrupting business.

[0028] On the other hand, the present invention also provides a non-intrusive audit system for a microservice architecture, characterized in that the audit system is composed of a lightweight agent, an original audit event generator, a rule matcher, an audit event generator and audit rules, the lightweight agent is responsible for intercepting request response information and sending audit messages; the original audit event generator generates original audit events based on log information and audit messages obtained according to the unique ID of the request; the rule matcher matches the original audit event with the rule, and after success, the audit event generator generates an audit event and records it in a database; the audit rules are responsible for determining whether the request needs to generate an audit and providing an audit template. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a non-intrusive audit method for a microservice architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0032] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0033] Embodiment 1:

[0034] like Figure 1 The non-intrusive audit method of a microservice architecture shown in the figure includes adding an audit process and adding an audit rule process. The specific processes are as follows:

[0035] Generate operational audit process:

[0036] A: The user initiates a request for the App1 service, and the request first passes through Sidecar;

[0037] B: Sidecar intercepts the request and obtains information such as the request path and request method;

[0038] C: App1 service completes the request processing and returns a response message, which is intercepted by Sidecar.

[0039] D: Sidecar intercepts the response information, obtains the HTTP status code in the response, and then releases the response and returns it to the user;

[0040] E: The Sidecar proxy injects special tracing identifiers into each request passing through it. Jaeger uses these identifiers to reconstruct the request flow chart, providing a detailed view of the request from ingress to egress, and stores this log information in ES.

[0041] F: Sidecar outputs the request / response information obtained by B and C and the requested traceId to the message queue; G: Audit service consumes audit request / response messages;

[0042] H: Get the request link log information from ElasticSearch based on traceId;

[0043] I, J: The original audit event generator consumes the message to obtain the request / response information, and combines it with the link number, response status, and error information in the log information to form the original audit event;

[0044] K: Match the original audit event with the audit rules. The matching methods mainly include request method matching, interface path matching, query parameter matching, and request body condition matching. Through the above four matching methods, the audit rules corresponding to the original audit event are obtained;

[0045] L: If all audit rules exist in Redis, the rule matcher directly obtains all audit rules from Redis;

[0046] M: The rule matcher generates an audit event based on the matched audit rule, and obtains the corresponding parameter in the request information based on the template parameter name of the audit rule. There are three main ways to obtain the value corresponding to the template parameter name: obtain the value from the query parameter, obtain the value from the request path, and obtain the value from the request body. The placeholder of the audit rule template is filled with the obtained value, thereby generating the corresponding audit operation record;

[0047] N: Store the generated audit operation records into the database;

[0048] O: Contrary to L, if all audit rules do not exist in Redis, all audit rules are obtained from the database;

[0049] P: Store all audit rules obtained in O into Redis.

[0050] Add audit rule process:

[0051] Q: User initiates a request to add audit rules;

[0052] R: Perform a legality check on the audit rules to be added;

[0053] S, T: Audit rules that pass verification are added to the database;

[0054] U, W: Update the cache of audit rules in Redis. If a new audit rule is added, the corresponding request will immediately generate the corresponding operation audit record without restarting the service.

[0055] V, X: After completing the audit rule adding operation, the audit rule change request / response message is also sent to the message queue.

[0056] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A non-intrusive audit method for microservice architecture, characterized in that: Deploy lightweight agents to intercept microservice request / response information and send audit messages; the distributed link tracking system obtains the request link between services based on the unique ID of the request and records it in the full-text search engine; Subscribe to and consume audit messages in the audit service; generate original audit events based on the log information and audit messages obtained from the requested unique ID; match the original audit events with the audit rules, and after a successful match, generate audit events and record them in the database; The cache is updated when the user updates or adds audit rules.

2. According to claim 1, a non-intrusive audit method for microservice architecture is characterized in that: Use service mesh technology to further reduce the intrusion into the microservice architecture system. The service mesh deploys a lightweight agent on each service container to implement its functions. These agents are responsible for intercepting and controlling all network traffic in and out of the service and placing qualified traffic information into the message queue.

3. According to claim 2, a non-intrusive audit method for microservice architecture is characterized in that: The integration of the service mesh and the distributed link tracing system allows users to trace the complete path of a request between services; the lightweight proxy injects special tracing identifiers into each request passing through it, and the distributed link tracing system uses these identifiers to reconstruct the request flow chart, providing a detailed view of the request from entry to exit.

4. A non-intrusive audit method for microservice architecture according to claim 3, characterized in that: A single service in the microservice writes the link number, response status, and error information into the log.

5. According to claim 1, a non-intrusive audit method for microservice architecture is characterized in that: Intercept each service request in the system and automatically generate an operation audit.

6. A non-intrusive audit method for microservice architecture according to claims 1-5, characterized in that: The audit message includes: operation time, operator, operation information, service name, request link number, response status, and error information.

7. A non-intrusive audit method for a microservice architecture according to claim 6, characterized in that: The operation information is generated by matching audit rules; the audit rules include interface path, request method, query parameter conditions, request body conditions, operation name template, and template parameter name.

8. A non-intrusive audit method for microservice architecture according to claim 8, characterized in that: The audit rule matching includes the matching of request method, query parameter conditions, and request body conditions. When the matching is completed, the audit operation is generated by filling in the template parameters according to the operation name template.

9. A non-intrusive audit method for microservice architecture according to claim 1, characterized in that: Cross-programming language, audit service support is used in systems of multiple programming languages, and the cross-programming language is implemented based on lightweight agents.

10. A non-intrusive audit system for microservice architecture, characterized in that: The audit system consists of a lightweight agent, an original audit event generator, a rule matcher, an audit event generator, and audit rules. The lightweight agent is responsible for intercepting request response information and sending audit messages. The original audit event generator generates original audit events based on the log information and audit messages obtained from the unique ID of the request. The rule matcher matches the original audit event with the rule. If successful, the audit event generator generates the audit event and records it in the database; Audit rules are responsible for determining whether a request needs to generate an audit and providing an audit template.

Images