Micro-service protocol invariance verification method based on self-learning

Abstract

The invention discloses a micro-service protocol invariance verification method based on self-learning. The micro-service protocol invariance verification method comprises the following steps: S1, unifying log formats of micro-services; s2, automatically learning a protocol of a production environment; s3, performing protocol verification in development, test and production environments; and S4, protocol confirmation and re-learning. According to the method, the micro-service protocol of the existing distributed system can be automatically learned, detection is carried out in development, testing and production links in software research and development, and if it is detected that the micro-service protocol is changed, related personnel are notified to process.

Description

Technical Field

[0001] The present invention relates to the fields of microservices and distributed systems, and in particular to a microservice protocol invariance verification method based on self-learning. Background Art

[0002] At present, in the practice of distributed microservice cluster architecture, enterprises have higher requirements for the delivery rate of R&D personnel due to the rapid development of Internet business. Large teams will develop dozens or even hundreds of businesses in parallel, and there are cases where the same microservice is developed simultaneously by multiple projects. The microservice cluster architecture contains hundreds or thousands of nodes, and the microservices use application layer protocols for communication. Microservices have mutual dependence and influence. It is a big challenge to ensure the correctness of the microservice cluster. If the new function developed by a microservice is launched after passing the test due to parallel business development, but the normal application layer protocol is modified incorrectly, it will cause other old businesses or other dependent microservices to fail; at the same time, because the test is designed according to the inherent scenario use case, the correctness of the test program model is an incomplete verification method, which will inevitably fail to detect all actual scenarios. Based on these two reasons, if the incorrect change of the microservice protocol is not identified in time, it will cause business interruption, and in serious cases, the enterprise will face the risk of customer complaints, public opinion, and financial losses.

[0003] Based on this, the present invention discloses a microservice protocol invariance verification method based on self-learning to solve the problem of protocol invariance verification, which can automatically identify whether the protocol has changed without manual intervention. Summary of the invention

[0004] The present invention is aimed at the deficiencies in the prior art. The present invention provides a method for detecting changes in microservice protocols. The method can automatically learn the microservice protocols of existing distributed systems and perform detection in the development, testing, and production links of software development. If changes are detected in the microservice protocols, relevant personnel are notified to handle the changes, thereby meeting actual usage requirements.

[0005] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0006] A method for verifying the invariance of a microservice protocol based on self-learning includes the following steps:

[0007] S1. Unify the log format of microservices

[0008] The same log format is defined for microservices, which includes service name, message name, request message, response message, response code, response time, etc. The service name is globally unique, the message name is unique under the same service name, and the request message and response message should contain multiple parameter names and parameter values.

[0009] S2. Automatically learn the protocol for production environment

[0010] Specifically, we collect the logs of the microservice production environment running normally in the past three days, and learn the rules of each request parameter and response parameter of each message name by parsing the corresponding log format;

[0011] Step 1: First learn about field types, field additions, field deletions, and enumeration values;

[0012] Learn field types: such as atomic types, int, double, string, json; composite types, arrayint, stringint, struct, structarray; for example, if the original protocol field type is learned to be int, and the new protocol is now detected as string, there may be errors in the upstream and downstream microservice call parameters, and the system will cause failures;

[0013] New learning fields: If an additional request parameter field is found, the system that receives this parameter may fail due to lack of corresponding processing logic;

[0014] Missing learning fields: If a response field was previously learned, but the newly discovered response message lacks this field, the response protocol will be changed, and the system that relies on this response message will be affected;

[0015] Learning whether it is an enumeration value: If the number of different values ​​is less than α, it is determined to be an enumeration value; for example, for a computer system, if the original reward types are 1 and 2, and a new type 3 is discovered, the judgment protocol is changed, and it is possible that type 3 has no corresponding code processing, which will cause a malfunction;

[0016] Step 2: Learn the specific rules of field types

[0017] 1. For atomic types int and double, the learning rules are:

[0018] (1) Learning value range: maximum value, minimum value; for example, for a payment system, if the maximum amount learned is 500 yuan, if an amount of 10,000 yuan is found later, the protocol will be changed;

[0019] (2) Learning positivity: positive, negative, and zero. For example, for a payment system, if the payment amount originally learned is a positive number, but is subsequently detected as a negative number or zero due to a protocol change, then the protocol change is considered to be a systemic risk.

[0020] 2. The learning rules for atomic type string parameters are:

[0021] (1) Learning field length: maximum length, minimum length. For example, if a field is an order number and the learned length is 32 bits, if the length is subsequently detected to be 40 bits, the protocol is determined to have changed, possibly because another incorrect field is passed as the order number.

[0022] (2) Learning character set: character set; for example: if the bill number character set is 0-9, and A is detected on a certain day, then a protocol change occurs. It is likely that other incorrect fields are passed as the number, or there is a problem with the number generator microservice;

[0023] 3. The learning rules for atomic type json parameters are:

[0024] (1) Learn the JSON structure: JSON field names and types;

[0025] (2) Learn the rules for values ​​in the JSON structure. The rules are the same as those for int, double, and string.

[0026] 4. The learning rules for composite types, arrayint, stringint, struct, and structarray are:

[0027] (1) Learning number range: maximum number, minimum number;

[0028] (2) Learn the rules for atomic types in composite types. The rules are the same as those for int, double, string, and json.

[0029] S3. Perform protocol verification in development, testing, and production environments

[0030] Perform protocol verification in the development environment. The protocol verification system processes the logs generated by the R&D joint debugging in real time. If a protocol change is detected, the protocol change is marked and notified to R&D for processing.

[0031] Perform protocol verification in the test environment. The protocol verification system processes the result logs of the test cases run by the testers in real time. If a protocol change is detected, the testers will contact the R&D personnel to confirm whether the protocol is correct. If the protocol is indeed incorrect, the defect will be submitted to the R&D personnel.

[0032] Protocol verification is performed in the production environment. The protocol verification system processes the logs of users or operators' operations on the software system in real time. If a protocol change is detected, an alarm is sent to the technical staff, requiring an immediate solution.

[0033] S4, protocol confirmation and relearning.

[0034] Some new functions require changes to the original protocol. After the protocol verification system detects the protocol change, the R&D personnel will confirm the correctness of the change. If the R&D personnel confirm that the protocol change is correct, the protocol verification system will learn the protocol that was originally determined to have changed, so that the protocol change will not be reported again for the same problem.

[0035] Similarly, if it is found that the original learning of a service name or message name is wrong, but it has not been discovered before, it should support re-learning of logs within a specified range of the service name and message name.

[0036] Compared with the prior art, the implementation effects of the present invention are as follows:

[0037] The present invention provides a microservice protocol invariance verification method based on self-learning to solve the problem that protocol changes of increasingly complex computer microservice clusters are difficult to detect. The method uses a unified log format for microservices and can detect protocol changes in real time and quickly.

[0038] The advantage of this method is that it does not require knowledge of the business field. The protocol verification system can automatically learn protocol parameters based on historical logs without manual intervention. When changes are found, personnel can be notified in time for follow-up processing. This method can be used in all aspects of software development. It reduces the difficulty for developers to find problems during the development stage; reduces the workload of testers during the testing stage, and can discover more hidden defects; and can discover protocol changes of internal and external partner companies during the production stage, and study the reasons for the changes in time. In particular, for external partner companies, their protocol changes are often not fully notified to other companies. This method can be used to self-discover problems and prevent business risks and financial risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] The content of the present invention will be described below with reference to specific embodiments.

[0041] like Figure 1 As shown, a microservice protocol invariance verification method based on self-learning according to the present invention includes the following steps:

[0042] S1. Unify the log format of microservices

[0043] The same log format is defined for microservices, which includes service name, message name, request message, response message, response code, response time, etc. The service name is globally unique, the message name is unique under the same service name, and the request message and response message should contain multiple parameter names and parameter values.

[0044] S2. Automatically learn the protocol for production environment

[0045] Specifically, we collect the logs of the microservice production environment running normally in the past three days, and learn the rules of each request parameter and response parameter of each message name by parsing the corresponding log format;

[0046] Step 1: First learn about field types, field additions, field deletions, and enumeration values;

[0047] Learn field types: such as atomic types, int, double, string, json; composite types, arrayint, stringint, struct, structarray; for example, if the original protocol field type is learned to be int, and the new protocol is now detected as string, there may be errors in the upstream and downstream microservice call parameters, and the system will cause failures;

[0048] New learning fields: If an additional request parameter field is found, the system that receives this parameter may fail due to lack of corresponding processing logic;

[0049] Missing learning fields: If a response field was previously learned, but the newly discovered response message lacks this field, the response protocol will be changed, and the system that relies on this response message will be affected;

[0050] Learning whether it is an enumeration value: If the number of different values ​​is less than α, it is determined to be an enumeration value; for example, for a computer system, if the original reward types are 1 and 2, and a new type 3 is discovered, the judgment protocol is changed, and it is possible that type 3 has no corresponding code processing, which will cause a malfunction;

[0051] Step 2: Learn the specific rules of field types

[0052] 1. For atomic types int and double, the learning rules are:

[0053] (1) Learning value range: maximum value, minimum value; for example, for a payment system, if the maximum amount learned is 500 yuan, if an amount of 10,000 yuan is found later, the protocol will be changed;

[0054] (2) Learning positivity: positive, negative, and zero. For example, for a payment system, if the payment amount originally learned is a positive number, but is subsequently detected as a negative number or zero due to a protocol change, then the protocol change is considered to be a systemic risk.

[0055] 2. The learning rules for atomic type string parameters are:

[0056] (1) Learning field length: maximum length, minimum length. For example, if a field is an order number and the learned length is 32 bits, if the length is subsequently detected to be 40 bits, the protocol is determined to have changed, possibly because another incorrect field is passed as the order number.

[0057] (2) Learning character set: character set; for example: if the bill number character set is 0-9, and A is detected on a certain day, then a protocol change occurs. It is likely that other incorrect fields are passed as the number, or there is a problem with the number generator microservice;

[0058] 3. The learning rules for atomic type json parameters are:

[0059] (1) Learn the JSON structure: JSON field names and types;

[0060] (2) Learn the rules for values ​​in the JSON structure. The rules are the same as those for int, double, and string.

[0061] 4. The learning rules for composite types, arrayint, stringint, struct, and structarray are:

[0062] (1) Learning number range: maximum number, minimum number;

[0063] (2) Learn the rules for atomic types in composite types. The rules are the same as those for int, double, string, and json.

[0064] S3. Perform protocol verification in development, testing, and production environments

[0065] Perform protocol verification in the development environment. The protocol verification system processes the logs generated by the R&D joint debugging in real time. If a protocol change is detected, the protocol change is marked and notified to R&D for processing.

[0066] Perform protocol verification in the test environment. The protocol verification system processes the result logs of the test cases run by the testers in real time. If a protocol change is detected, the testers will contact the R&D personnel to confirm whether the protocol is correct. If the protocol is indeed incorrect, the defect will be submitted to the R&D personnel.

[0067] Protocol verification is performed in the production environment. The protocol verification system processes the logs of users or operators' operations on the software system in real time. If a protocol change is detected, an alarm is sent to the technical staff, requiring an immediate solution.

[0068] S4, protocol confirmation and relearning.

[0069] Some new functions require changes to the original protocol. After the protocol verification system detects the protocol change, the R&D personnel will confirm the correctness of the change. If the R&D personnel confirm that the protocol change is correct, the protocol verification system will learn the protocol that was originally determined to have changed, so that the protocol change will not be reported again for the same problem.

[0070] Similarly, if it is found that the original learning of a service name or message name is wrong, but it has not been discovered before, it should support re-learning of logs within a specified range of the service name and message name.

[0071] The present invention provides a microservice protocol invariance verification method based on self-learning to solve the problem that protocol changes of increasingly complex computer microservice clusters are difficult to detect. The method uses a unified log format for microservices and can detect protocol changes in real time and quickly.

[0072] The advantage of this method is that it does not require knowledge of the business field. The protocol verification system can automatically learn protocol parameters based on historical logs without manual intervention. When changes are found, personnel can be notified in time for follow-up processing. This method can be used in all aspects of software development. It reduces the difficulty for developers to find problems during the development stage; reduces the workload of testers during the testing stage, and can discover more hidden defects; and can discover protocol changes of internal and external partner companies during the production stage, and study the reasons for the changes in time. In particular, for external partner companies, their protocol changes are often not fully notified to other companies. This method can be used to self-discover problems and prevent business risks and financial risks.

[0073] In summary, compared with the traditional waterfall flow software development method, it has the advantages of protocol self-learning, protocol invariance verification, and the ability to identify problems that cannot be identified by pre-established test models. It improves the governance capabilities of microservice clusters in various R&D links and strengthens the reliability of distributed software.

[0074] The above content is a detailed description of the present invention in combination with specific embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A microservice protocol invariance verification method based on self-learning, which has the following characteristics: The following steps are involved: S1. Unify the log format of microservices The same log format is defined for microservices, which includes service name, message name, request message, response message, response code, response time, etc. Its service name is globally unique, the message name is unique under the same service name, and the request message and response message should contain multiple parameter names and parameter values; S2. Automatically learn the protocol for production environment Specifically, we collect the logs of the microservice production environment running normally in the past three days, and learn the rules of each request parameter and response parameter of each message name by parsing the corresponding log format; Step 1: First learn about field types, field additions, field deletions, and enumeration values; Step 2: Learn the specific rules of field types:

1. For atomic types int and double, the learning rules are:

2. The learning rules for atomic type string parameters are:

3. The learning rules for atomic type json parameters are:

4. Learning rules for composite types, arrayint, stringint, struct, structarray; S3. Perform protocol verification in development, testing, and production environments Perform protocol verification in the development environment. The protocol verification system processes the logs generated by the R&D joint debugging in real time. If a protocol change is detected, the protocol change is marked and notified to R&D for processing. Perform protocol verification in the test environment. The protocol verification system processes the result logs of the test cases run by the testers in real time. If a protocol change is detected, the testers will contact the R&D personnel to confirm whether the protocol is correct. If the protocol is indeed incorrect, the defect will be submitted to the R&D personnel. In the production environment, the protocol verification system processes the logs of the users or operators' operations on the software system in real time. If a protocol change is detected, an alarm is sent to the technical staff, requiring an immediate solution. S4, protocol confirmation and relearning. Some new functions require changes to the original protocol. After the protocol verification system detects the protocol change, the R&D personnel will confirm the correctness of the change. If the developer confirms that the protocol change is correct, the protocol verification system will learn the protocol that was originally determined to have changed, so that the protocol change will not be reported again for the same issue; Similarly, if it is found that the original learning of a service name or message name is wrong, but it has not been discovered before, it should support re-learning of logs within a specified range of the service name and message name.

Images