System and method for automatically controlling flow, monitoring and pre-judging system calling

Abstract

The invention provides a system and a method for automatic flow control, monitoring and pre-judging system calling, the system for automatic flow control, monitoring and pre-judging system calling comprises a client subsystem and a server subsystem, the client subsystem is used for docking with a service system; and the server-side subsystem is used for performing flow control, monitoring and pre-judgment on the service system. According to the technical scheme, more servers can be saved compared with independent deployment; the jar package is introduced, so that the access is quicker and more convenient, and the development cost and the maintenance cost are greatly reduced; and the security of the system is greatly improved by a sensitive information detection technology.

Description

Background Art

[0002] As the complexity and coupling of distributed system architectures increase, the system is always facing the challenge of potential traffic peaks, which seriously threatens system stability. It is imperative to use system automated circuit breaker and current limiting technology to ensure system stability. However, existing technologies cannot achieve safe, stable and convenient access, and their functions are not perfect. They also lack monitoring modules for server performance indicators, which is not conducive to discovering and solving potential performance problems. With the increasingly stringent supervision of user sensitive data, it is necessary to ensure that user sensitive information in messages is properly protected and encrypted during service interaction. However, there are still developers who ignore security issues, and the problem of plaintext transmission of user sensitive information leads to user data leakage. It is very necessary to detect, analyze and warn of interactive messages between services to achieve real-time discovery and reduce the risk of user data leakage. Existing sensitive information detection technologies have high detection error rates and basically do not support protection at the interaction layer.

[0003] The disadvantages of sentinel technology are as follows:

[0004] 1. Sentinel does not have different rules for configuring services on multiple nodes. Due to differences in server performance, it is very necessary to make differentiated rules.

[0005] 2. Sentinel does not have the ability to persist data. For example, the real-time information of each interface is not persistent. The generated reports only retain a few minutes of data and cannot historically analyze system bottlenecks, making it difficult to predict the direction of system traffic. For example, the flow control rule service will be lost if it is stopped.

[0006] 3. The sentinel graphical interface does not have permission division. It is difficult to manage multiple application systems using one set of sentinel and it does not have the characteristics of a middle platform.

[0007] 4. Sentinel does not have the function of monitoring server monitoring indicators.

[0008] 5. Automatic monitoring and early warning function for sensitive information in interactive messages between services, no competing products yet. Common sensitive information detection methods on the market are all located at the DB layer, and the detection solutions are either slow and not real-time, or the error rate is too large. Summary of the invention

[0009] The present application provides a system and method for automated flow control, monitoring, and predicting system calls to reduce development and maintenance costs.

[0010] In a first aspect, a system for automated flow control, monitoring, and predicting system calls is provided, including a client subsystem and a server subsystem, wherein:

[0011] The client subsystem is used to connect with the business system;

[0012] The service terminal subsystem is used to perform flow control, monitoring and prediction on the business system.

[0013] In the above technical solution, the client subsystem is used to connect with the business system; the server subsystem is used to perform flow control, monitoring and prediction on the business system; more servers can be saved than deploying them separately; the introduction of jar packages makes access faster and more convenient, greatly reducing development and maintenance costs; and sensitive information detection technology greatly improves the security of the system.

[0014] In a specific implementation scheme, it is characterized in that the client subsystem uses a jar package to complete the connection with the business system.

[0015] In a specific implementation scheme, it is characterized in that the client subsystem includes:

[0016] The current limiting module and the fuse module are used to use nacos to pull rules when the business system is started, monitor nacos rule changes, and modify the rules in real time; when the interface is accessed, the rule chain is used to determine whether the request is flow controlled through the algorithm carried by the rule chain;

[0017] The service reporting module is used to register instance information to the circuit breaker and current limiting service cluster through the Http interface when the business system is started.

[0018] In a specific implementation scheme, the client subsystem further includes:

[0019] Performance indicator detection, which is used to view performance indicators on the monitoring page, will pull indicators from each application through the service monitoring module of the circuit breaker and current limiting server cluster;

[0020] The message collection module is used to obtain the required request and return messages from Spring Actuator and push them to Kafka for sensitive information detection.

[0021] In a specific implementation scheme, the client subsystem further includes:

[0022] The whitelist module is used to obtain the whitelist from Redis for user requests to perform access control.

[0023] In a specific implementation scheme, the service terminal system includes:

[0024] The flow control module is used to use the rule chain to perform rule verification on the received request. If the verification passes, the service is requested normally, otherwise the process is blocked;

[0025] The service monitoring module is used to make real-time statistics on the number of concurrent requests and time-consuming information of user URLs, and to view performance indicators through the monitoring console.

[0026] In a specific implementation scheme, the service terminal system further includes:

[0027] The permission control module is used to manage permissions for multiple systems and limit visibility;

[0028] The whitelist unit is used to maintain the whitelist of each business system and synchronize redis in real time.

[0029] In a specific implementation scheme, the service terminal system further includes:

[0030] The sensitive information detection module is used to receive messages obtained from Kafka, decompose the messages into words with attributes through natural language segmentation, and identify the attributes.

[0031] In a specific implementation scheme, the server subsystem further includes: a heartbeat detection module for heartbeat detection.

[0032] In a second aspect, a method for automated flow control, monitoring, and predicting system calls is provided, comprising the following steps:

[0033] Use the client subsystem to connect with the business system;

[0034] The service terminal system is used to control, monitor and predict the flow of the business system.

[0035] In the above technical solution, the client subsystem is used to connect with the business system; the server subsystem is used to perform flow control, monitoring and prediction on the business system; more servers can be saved than deploying them separately; the introduction of jar packages makes access faster and more convenient, greatly reducing development and maintenance costs; and sensitive information detection technology greatly improves the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of a method for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of another system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application;

[0039] Figure 4A schematic diagram of the structure of another system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0041] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0042] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0043] To facilitate understanding of the system and method for automated flow control, monitoring, and prejudgment of system calls provided in the embodiment of the present application, its application scenario is first explained. The system and method for automated flow control, monitoring, and prejudgment of system calls provided in the embodiment of the present application are used to reduce development costs and maintenance costs. With the increase in the complexity coupling degree of the distributed system architecture, the system is always facing the challenge of potential traffic peaks, which seriously threatens the stability of the system. It is imperative to use system automated fuse limiting technology to ensure system stability. However, the existing technology cannot achieve problems such as safe, stable, and convenient access and imperfect functions, and also lacks a monitoring module for server performance indicators, which is not conducive to discovering and solving potential performance problems. With the increasingly strict supervision of user sensitive data, it is necessary to ensure that user sensitive information in messages is properly protected and encrypted for interaction between services. However, there are still developers who ignore security issues, and the problem of plaintext transmission of user sensitive information leads to user data leakage. It is very necessary to detect and analyze the interactive messages between services to reduce the risk of user data leakage in real time. The existing sensitive information detection technology has a high detection error rate and basically does not support the protection of the interaction layer. The shortcomings of sentinel technology are as follows: 1. Sentinel does not have different rules for configuring services on multiple nodes. It is very necessary to make differentiated rules due to differences in server performance. 2. Sentinel does not have data persistence capabilities. For example, the real-time information of each interface is not persistent. The generated report only retains a few minutes of data and cannot historically trace back to analyze system bottlenecks, making it difficult to predict the direction of system traffic; for example, the flow control rule service is lost if it is stopped. 3. The sentinel graphical interface does not have permission division. It is difficult to manage a set of sentinels for multiple application systems, and it does not have the characteristics of the middle platform. 4. Sentinel does not have the function of monitoring server monitoring indicators. 5. There is no competitive product for the automatic monitoring and early warning function of sensitive information in interactive messages between services. Common sensitive information found on the market is located at the DB layer, and the detection scheme is either slow and not real-time, or the error rate is too large. For this reason, the embodiment of the present application provides a system and method for automated flow control, monitoring, and predicting system calls to reduce development costs and maintenance costs. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0044] refer to Figures 1 to 4 , Figure 1 A schematic diagram of the structure of a system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application; Figure 2 A flowchart of a method for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of another system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of another system for automated flow control, monitoring, and predicting system calls provided in an embodiment of the present application.

[0045] exist Figure 1 In the embodiment of the present application, a system for automatic flow control, monitoring, and predicting system calls is provided, including a client subsystem and a server subsystem, wherein:

[0046] The client subsystem is used to connect with the business system;

[0047] The service terminal subsystem is used to perform flow control, monitoring and prediction on the business system.

[0048] In the above technical solution, the client subsystem is used to connect with the business system; the server subsystem is used to perform flow control, monitoring and prediction on the business system; more servers can be saved than deploying them separately; the introduction of jar packages makes access faster and more convenient, greatly reducing development and maintenance costs; and sensitive information detection technology greatly improves the security of the system.

[0049] Specifically, this application: 1. A registration mechanism is used to record the client IP and port, and a rule chain is stored for each service node, which solves the problem that the flow control rules cannot be configured differently for multi-node applications. 2. Asynchronous push is used for persistence of massive real-time data, which provides persistent storage capabilities while ensuring throughput, solves the system bottleneck of being unable to conduct historical retrospective analysis, and improves the ability to predict the direction of system traffic. For rule information, nacos is used as a storage container. With the help of nacos's monitoring mechanism, the persistence capability is increased on the basis of ensuring real-time rule changes to ensure that it is not lost during start and stop. 3. Strict permission management is performed for multiple systems to limit visibility, making this system suitable for managing multiple application systems, with the characteristics of a middle platform, and reducing server investment rate. 4. Server performance indicator monitoring has been added to monitor the status of applications, including key indicators such as CPU usage, memory status, and thread information. This helps to promptly discover and resolve potential performance issues and prevent system crashes or performance degradation. 5. By transforming Spring Actuator, we can obtain interactive messages from various application systems, push messages asynchronously through Kafka, use natural language processing tools to segment messages, filter out sensitive information, and use the semantic attributes of natural language segmentation to discover sensitive information for some sensitive information. For some sensitive information, we use regular expression technology to discover sensitive information. When sensitive information is discovered, send an email warning. We can promptly discover and report sensitive information, effectively reducing the risk of sensitive information leakage.

[0050] In a specific implementation scheme, it is characterized in that the client subsystem uses a jar package to complete the connection with the business system.

[0051] In a specific implementation scheme, it is characterized in that the client subsystem includes:

[0052] The current limiting module and the fuse module are used to use nacos to pull rules when the business system is started, monitor nacos rule changes, and modify the rules in real time; when the interface is accessed, the rule chain is used to determine whether the request is flow controlled through the algorithm carried by the rule chain;

[0053] The service reporting module is used to register instance information to the circuit breaker and current limiting service cluster through the Http interface when the business system is started.

[0054] In a specific implementation scheme, the client subsystem further includes:

[0055] Performance indicator detection, which is used to view performance indicators on the monitoring page, will pull indicators from each application through the service monitoring module of the circuit breaker and current limiting server cluster;

[0056] The message collection module is used to obtain the required request and return messages from Spring Actuator and push them to Kafka for sensitive information detection.

[0057] In a specific implementation scheme, the client subsystem further includes:

[0058] The whitelist module is used to obtain the whitelist from Redis for user requests to perform access control.

[0059] In a specific implementation scheme, the service terminal system includes:

[0060] The flow control module is used to use the rule chain to perform rule verification on the received request. If the verification passes, the service is requested normally, otherwise the process is blocked;

[0061] The service monitoring module is used to make real-time statistics on the number of concurrent requests and time-consuming information of user URLs, and to view performance indicators through the monitoring console.

[0062] In a specific implementation scheme, the service terminal system further includes:

[0063] The permission control module is used to manage permissions for multiple systems and limit visibility;

[0064] The whitelist unit is used to maintain the whitelist of each business system and synchronize redis in real time.

[0065] In a specific implementation scheme, the service terminal system further includes:

[0066] The sensitive information detection module is used to receive messages obtained from Kafka, decompose the messages into words with attributes through natural language segmentation, and identify the attributes.

[0067] In a specific implementation scheme, the server subsystem further includes: a heartbeat detection module for heartbeat detection.

[0068] Specifically, the system for automated flow control, monitoring, and prediction system calls includes client SDK and "backend services". The business system can directly introduce the jar package "SLFM-SDK——V1" to complete the access. Including:

[0069] 1. SLFM-SDK——V1 (the client subsystem) function implementation:

[0070] When the business system is started, the current limiting module and the fuse module go to nacos to pull rules, monitor nacos rule changes, and modify the rules in real time. When the interface is accessed, the rule chain determines whether the request is flow controlled through the algorithm carried by the rule chain.

[0071] Service reporting module: registers instance information to the circuit breaker and current limiting service cluster through the Http interface when the business system is started.

[0072] Performance indicator detection: Viewing performance indicators on the monitoring page will pull indicators from each application through the service monitoring module of the circuit breaker and current limiting server cluster.

[0073] Message collection module: obtains the required request and return messages from Spring Actuator and pushes them to Kafka for sensitive information detection.

[0074] Whitelist module: Obtain the whitelist from Redis for user requests to perform access control.

[0075] 2. Server-side functions (the server-side subsystem) realizes:

[0076] Heartbeat detection: If the heartbeat packet from the same client is not received for three consecutive times, the connection is considered disconnected, and the monitoring page will show that the client is offline. A trace will be left for analyzing the stability indicators of the system.

[0077] Flow control core: It is a series of rule algorithms such as fuse and current limiting. These algorithms are strung together into a rule chain to perform rule verification on the received request. If the verification passes, the service is requested normally, otherwise the process is blocked.

[0078] Service Monitoring:

[0079] 1. Generate real-time statistics and charts on the number of concurrent requests and time consumption of the user's request URL. You can also read historical data from the database to display monitoring information for analyzing system bottlenecks and predicting system traffic trends.

[0080] 2. View performance indicators in the monitoring console. The monitoring module will connect to each application to collect performance indicators and display charts

[0081] Permission control: Strict permission management is implemented for multiple systems to limit visibility, making this system suitable for managing multiple application systems. It has the characteristics of a middle platform and reduces server investment rate.

[0082] Whitelist: Maintain whitelists for each business system, synchronize with redis in real time, and use whitelists for access control by clients

[0083] Sensitive information detection: Receive the message obtained from Kafka, and use natural language segmentation to break the message into words with attributes. For irregular words such as names and addresses, directly use the attributes after natural language segmentation for recognition. The trained natural language component can achieve a recognition accuracy of more than 95%. For regular words such as bank card numbers and mobile phone numbers, the accuracy of regular matching is as high as 99% after segmentation. The overall speed can reach tens of millions per second.

[0084] In this embodiment, 1. Sensitive information detection scheme at the network transmission layer, and server detection: the client collects messages through jar packages and pushes them to kafka, which is non-invasive and non-perceptible to the client, and asynchronously pushes them to the server for detection with extremely high performance. The server uses natural language word segmentation to classify parts of speech, uses AI models to process part of sensitive information, and uses regularization to process part of sensitive information. While ensuring a low error rate, the speed is also guaranteed. The trained AI model has a processing accuracy of 95%, and the regularization accuracy can reach 99%. The overall speed can reach tens of millions per second. 2. A fuse flow control technology solution that supports convenient access to multiple systems and can be configured differently for each node of a single service: When accessing the service, each system has strict permissions and other divisions, and supports the use of multiple services. Considering that each node of each service may have performance differences, each node can be managed in a single way. 3. Support performance monitoring of each node of the server, and real-time monitoring of the status of the application, including key indicators such as CPU usage, memory status, and thread information. This helps to promptly discover and solve potential performance problems and prevent system crashes or performance degradation.

[0085] refer to Figure 3 , the official WeChat system was implemented after the SLFM-SDK-V1 jar package was introduced. Circuit breaking and current limiting, performance indicator monitoring, sensitive information detection, business services, etc. are all completed by the interaction between the SLFM-SDK-V1 jar package and the backend service data. This component runs completely independently, and the access party does not need to do any other additional operations. There are simple default rules for circuit breaking and current limiting, and you can also configure it yourself to override the default rules. Reference Figure 4 The access party only needs to introduce SLFM-SDK-V1 jar without any other additional operations.

[0086] In the above technical solutions, compared with sentinel, this application supports multi-system integration for easy use as a middle platform, which can save more servers than separate deployment; introducing jar packages for access is faster and more convenient, greatly reducing development and maintenance costs; sensitive information detection technology greatly improves system security.

[0087] exist Figure 2 In the embodiment of the present application, a method for automatic flow control, monitoring, and predicting system calls is provided, comprising the following steps:

[0088] Use the client subsystem to connect with the business system;

[0089] The service terminal system is used to control, monitor and predict the flow of the business system.

[0090] In the above technical solution, the client subsystem is used to connect with the business system; the server subsystem is used to perform flow control, monitoring and prediction on the business system; more servers can be saved than deploying them separately; the introduction of jar packages makes access faster and more convenient, greatly reducing development and maintenance costs; and sensitive information detection technology greatly improves the security of the system.

[0091] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0092] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0093] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0094] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A system for automatic flow control, monitoring, and predicting system calls, characterized in that: It includes client subsystem and server subsystem, among which, The client subsystem is used to connect with the business system; The service terminal subsystem is used to perform flow control, monitoring and prediction on the business system.

2. The system for automatic flow control, monitoring and predicting system calls according to claim 1, characterized in that: The client subsystem uses a jar package to connect with the business system.

3. The system for automatic flow control, monitoring and predicting system calls according to claim 2, characterized in that: The client subsystem includes: The current limiting module and the fuse module are used to use nacos to pull rules when the business system is started, monitor nacos rule changes, and modify the rules in real time; when the interface is accessed, the rule chain is used to determine whether the request is flow controlled through the algorithm carried by the rule chain; The service reporting module is used to register instance information to the circuit breaker and current limiting service cluster through the Http interface when the business system is started.

4. The system for automated flow control, monitoring, and predicting system calls according to claim 3, characterized in that: The client subsystem also includes: Performance indicator detection, which is used to view performance indicators on the monitoring page, will pull indicators from each application through the service monitoring module of the circuit breaker and current limiting server cluster; The message collection module is used to obtain the required request and return messages from Spring Actuator and push them to Kafka for sensitive information detection.

5. The system for automatic flow control, monitoring and predicting system calls according to claim 4, characterized in that: The client subsystem also includes: The whitelist module is used to obtain the whitelist from Redis for user requests to perform access control.

6. The system for automatic flow control, monitoring and predicting system calls according to claim 5, characterized in that: The service terminal system comprises: The flow control module is used to use the rule chain to perform rule verification on the received request. If the verification passes, the service is requested normally, otherwise the process is blocked; The service monitoring module is used to make real-time statistics on the number of concurrent requests and time-consuming information of user URLs, and to view performance indicators through the monitoring console.

7. The system for automatic flow control, monitoring and predicting system calls according to claim 6, characterized in that: The service terminal system also includes: The permission control module is used to manage permissions for multiple systems and limit visibility; The whitelist unit is used to maintain the whitelist of each business system and synchronize redis in real time.

8. The system for automatic flow control, monitoring and predicting system calls according to claim 7, characterized in that: The service terminal system further comprises: The sensitive information detection module is used to receive messages obtained from Kafka, decompose the messages into words with attributes through natural language segmentation, and identify the attributes.

9. The system for automatic flow control, monitoring and predicting system calls according to claim 8, characterized in that: The server subsystem further includes a heartbeat detection module, which is used for heartbeat detection.

10. A method for automatic flow control, monitoring, and predicting system calls, characterized in that the following step: Use the client subsystem to connect with the business system; The service terminal system is used to control, monitor and predict the flow of the business system.

Images