Automatic identification and desensitization system

Abstract

The invention relates to the technical field of information security privacy protection, in particular to an automatic identification and desensitization system, in the system, a log acquisition module is configured to acquire original log data of an application system in real time or according to a preset time interval, a sensitive information identification engine is used for deeply analyzing the original log data, and the sensitive information identification engine is used for identifying sensitive information; the desensitization strategy management center is responsible for storing, classifying, updating and dynamically providing global and local desensitization strategies, and the system monitoring and log auditing module is used for monitoring the running state, the performance index and the desensitization effect of the system. And the strategy configuration and update interface allows an external system, a security administrator or an automatic tool to dynamically update the desensitization strategy and the verification rule in the desensitization strategy management center through a graphical interface or an API (Application Program Interface). According to the system, the invasiveness can be reduced, high-precision sensitive information identification is realized, the system is flexible and configurable, fine-grained desensitization processing can be realized, comprehensive monitoring and auditing can be realized, and the reusability and expandability of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of information security and privacy protection technology, and in particular to an automated identification and de-identification system. Background Technology

[0002] In existing technologies, such as the method for de-identifying application system log data disclosed in patent document (publication number CN117892338A), de-identifying application system log data is an important means of protecting user privacy and sensitive information. However, current log de-identification methods generally have some problems.

[0003] First, many de-identification schemes require significant intrusive modifications to the application system code, resulting in strong coupling between the de-identified code and the application system code, which increases the complexity and maintenance cost of the system.

[0004] Secondly, although some methods attempt to reduce intrusion into application systems through global desensitization strategies, these methods often struggle to strike a balance between reducing intrusion and achieving fine-grained desensitization, especially when dealing with local sensitive information.

[0005] Furthermore, with the rapid development of electronic finance and various online services, the amount of user data and operational data carried by application systems has increased dramatically, and the types and quantities of sensitive information contained in logs have also increased day by day.

[0006] Existing de-identification methods often face problems such as low identification accuracy, poor de-identification effect, and difficulty in adapting to ever-changing compliance requirements when dealing with complex and ever-changing log data.

[0007] Therefore, developing a log data anonymization system that can effectively reduce intrusion into application systems while achieving high precision, fine granularity, and flexible configurability has become a pressing technical problem in the field of information security. Summary of the Invention

[0008] The main objective of this invention is to overcome the shortcomings of the prior art and provide an automated identification and desensitization system.

[0009] The technical solution adopted by this invention to achieve its technical objective is: an automated identification and desensitization system, the system comprising: The log acquisition module is configured to automatically read and monitor the extended interface of the application system's log framework, and acquire the raw log data of the application system in real time or at predetermined time intervals, supporting multiple log frameworks and formats; The sensitive information identification engine has a built-in set of advanced automated identification algorithms for in-depth analysis of raw log data, automatically and accurately identifying and locating various types of sensitive information within it; The de-identification strategy management center, as the core of the system's strategy, is responsible for storing, classifying, updating, and dynamically providing global and local de-identification strategies; The intelligent desensitization processing engine performs highly customized automated desensitization processing on sensitive information identified by the sensitive information identification engine, based on the strategies provided by the desensitization strategy management center, and generates desensitized log data that meets security standards. The log feedback and storage module is responsible for seamlessly feeding back the de-identified log data to the application system's log framework for storage, and also supports exporting the de-identified logs to an external storage system or data analysis platform. The system monitoring and log auditing module is used to monitor the system's operating status, performance indicators, and de-identification effect, and generate detailed audit logs for subsequent security reviews and optimization.

[0010] Preferably, the sensitive information includes, but is not limited to, personal identification information, bank card number, ID number, contact information, and geographical location information.

[0011] Preferably, the desensitization strategy includes sensitive information identification rules, multi-level desensitization rules, verification rules, and anomaly handling mechanisms.

[0012] Preferably, the sensitive information recognition engine further includes: The local sensitive information precise location unit uses the log printing statements marked with specific syntax or annotations in the application system code to accurately locate and extract local sensitive information in complex raw log data. The global sensitive information intelligent scanning unit employs various scanning methods, including but not limited to intelligent parsing based on key-value pairs (KV), deep scanning based on structured data such as JSON / XML, flexible matching based on regular expressions, and pattern recognition based on machine learning. Combined with the sensitive information identification rules provided by the de-identification strategy management center, it performs a comprehensive and efficient global sensitive information scan on the original log data.

[0013] Preferably, the global sensitive information intelligent scanning unit also has an adaptive learning function, which can dynamically adjust the scanning strategy and recognition rules according to historical scanning results and feedback, thereby improving the accuracy and efficiency of sensitive information recognition.

[0014] Preferably, the intelligent desensitization processing engine further includes: The multi-level desensitization rule application unit performs layer-by-layer desensitization processing on sensitive information based on the multi-level desensitization rules (such as field level, record level, and file level) provided by the desensitization strategy management center, ensuring the thoroughness and flexibility of desensitization. The dynamic verification and anomaly handling unit performs dual verification on sensitive information before and after desensitization when the desensitization strategy specifies verification rules, ensuring the accuracy and compliance of the desensitization process. At the same time, it promptly captures, records, and reports any anomalies that occur during the process, ensuring the stable operation of the system.

[0015] Preferably, when processing sensitive information, the multi-level desensitization rule application unit supports custom retention of a specified length of prefix and suffix in the sensitive information, replacing the middle part with dynamically generated mask characters or using other encryption technologies for protection.

[0016] Preferably, the system also includes a policy configuration and update interface, which allows external systems, security administrators, or automated tools to dynamically update the desensitization policies and verification rules in the desensitization policy management center through a graphical interface or API interface, so as to quickly respond to the ever-changing sensitive information protection needs and compliance requirements.

[0017] Preferably, the system is designed with a highly scalable and modular architecture, which can be flexibly integrated into existing application systems as a plug-in or deployed as an independent service in a network environment. It can interact efficiently with the application system log framework through standard interfaces to achieve automated and intelligent identification and de-identification of various application system log data.

[0018] The working principle of this automated identification and desensitization system is as follows: The log acquisition module automatically reads and monitors the extended interfaces of the application system's log framework to obtain raw log data. The sensitive information identification engine utilizes a built-in set of automated identification algorithms, including a local sensitive information precise location unit and a global sensitive information intelligent scanning unit, to deeply analyze and accurately locate sensitive information in the logs. The desensitization strategy management center stores, classifies, and updates desensitization strategies. The intelligent desensitization processing engine performs customized desensitization processing on sensitive information according to these strategies, generating desensitized log data. The log feedback and storage module feeds back the desensitized log data to the application system's log framework for storage and supports exporting to external systems. The system monitoring and log auditing module monitors the system status and desensitization effect, generating audit logs. The strategy configuration and update interface allows for dynamic updates of desensitization strategies and verification rules externally. The system achieves automated and intelligent log data identification and desensitization processing through a highly scalable and modular architecture.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This automated identification and de-identification system reduces invasiveness. By reading the extended interface of the application system's logging framework, it obtains log data without requiring de-identification code to be written directly into the application system code, thus significantly reducing invasiveness and coupling with the application system code.

[0020] This automated identification and desensitization system can achieve high-precision identification of sensitive information. The sensitive information identification engine in this system incorporates an advanced set of automated identification algorithms, including a local sensitive information precise location unit and a global sensitive information intelligent scanning unit. It can deeply analyze raw log data, automatically and accurately identify and locate various types of sensitive information, improving the accuracy and efficiency of identification.

[0021] This automated identification and desensitization system enables fine-grained desensitization processing. The system supports multi-level desensitization rule applications, allowing for progressively deeper desensitization of sensitive information based on different desensitization strategies. In particular, the use of precise local sensitive information location units enables the system to achieve fine-grained desensitization at the statement level, meeting the desensitization needs of various scenarios.

[0022] This automated identification and desensitization system is flexibly configurable. It provides policy configuration and update interfaces, allowing external systems, security administrators, or automation tools to dynamically update desensitization policies and verification rules. This enables the system to quickly respond to ever-changing sensitive information protection needs and compliance requirements, improving its adaptability and flexibility.

[0023] This automated identification and de-identification system enables comprehensive monitoring and auditing. The system includes a system monitoring and log auditing module, which continuously monitors the system's operational status, performance indicators, and de-identification effectiveness, generating detailed audit logs. This provides strong support for subsequent security reviews and optimizations, ensuring the system's stable operation and compliance.

[0024] This automated identification and de-identification system boasts strong scalability. Designed with a highly scalable and modular architecture, it can be flexibly integrated into existing application systems as a plug-in or deployed as a standalone service in a network environment. This not only simplifies system deployment and maintenance processes but also enhances system reusability and scalability. Attached Figure Description

[0025] Figure 1 This is a system framework diagram for an automated identification and desensitization system.

[0026] Figure 2 This is a system framework diagram for a sensitive information recognition engine.

[0027] Figure 3 This is a system framework diagram for an image desensitization processing engine.

[0028] Figure 4 A flowchart illustrating the operational steps of an automated identification and desensitization system.

[0029] in: 1- Log acquisition module; 2- Sensitive information identification engine; 201- Local sensitive information precise location unit; 202- Global sensitive information intelligent scanning unit; 3- Desensitization strategy management center; 4- Intelligent desensitization processing engine; 401- Multi-level desensitization rule application unit; 402- Dynamic verification and exception handling unit; 5- Log feedback and storage module; 6- System monitoring and log auditing module; 7- Policy configuration and update interface. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0031] In the description of this invention, it should be noted that when an element is referred to as being "fixed to" or "set on" another element, it can be directly on or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to or indirectly connected to the other element.

[0032] In the description of this invention, it should be noted that the terms "center," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. "Several" means one or more, unless otherwise explicitly specified.

[0033] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. Example 1:

[0034] Please see Figures 1-4 An automated identification and desensitization system is provided, comprising a log acquisition module 1, a sensitive information identification engine 2, a desensitization policy management center 3, an intelligent desensitization processing engine 4, a log feedback and storage module 5, a system monitoring and log auditing module 6, and a policy configuration and update interface 7. The system is designed with a highly scalable and modular architecture, enabling flexible integration into existing application systems as a plug-in or deployment as a standalone service in a network environment. It interacts efficiently with application system log frameworks through standard interfaces, achieving automated and intelligent identification and desensitization of various application system log data.

[0035] In this implementation, the log acquisition module 1 is configured to automatically read and monitor the extended interface of the application system's log framework, acquire the application system's raw log data in real time or at predetermined time intervals, and support multiple log frameworks and formats.

[0036] In this implementation, the sensitive information identification engine 2 has a built-in set of advanced automated identification algorithms for in-depth analysis of raw log data, automatically and accurately identifying and locating various types of sensitive information. The sensitive information includes, but is not limited to, personal identification information, bank card numbers, ID card numbers, contact information, and geographical location information.

[0037] The sensitive information identification engine 2 further includes a local sensitive information precise positioning unit 201 and a global sensitive information intelligent scanning unit 202.

[0038] The local sensitive information precise location unit 201 uses the log printing statements marked with specific syntax or annotations in the application system code to set the identifier, and accurately locates and extracts local sensitive information in complex raw log data.

[0039] The global sensitive information intelligent scanning unit 202 employs various scanning methods, including but not limited to intelligent parsing based on key-value pairs (KV), deep scanning based on structured data such as JSON / XML, flexible matching based on regular expressions, and pattern recognition based on machine learning. Combined with the sensitive information identification rules provided by the desensitization strategy management center 3, it performs a comprehensive and efficient global sensitive information scan on the original log data.

[0040] The global sensitive information intelligent scanning unit also has an adaptive learning function, which can dynamically adjust the scanning strategy and recognition rules based on historical scanning results and feedback, thereby improving the accuracy and efficiency of sensitive information recognition.

[0041] In this implementation, the de-identification strategy management hub 3 serves as the core of the system's strategy, responsible for storing, classifying, updating, and dynamically providing global and local de-identification strategies.

[0042] The desensitization strategy includes sensitive information identification rules, multi-level desensitization rules, verification rules, and anomaly handling mechanisms.

[0043] In this implementation, the intelligent desensitization processing engine 4 performs highly customized automated desensitization processing on the sensitive information identified by the sensitive information identification engine 2 according to the strategy provided by the desensitization strategy management center 3, and generates desensitized log data that meets security standards.

[0044] Preferably, the intelligent desensitization processing engine 4 further includes a multi-level desensitization rule application unit 401 and a dynamic verification and anomaly handling unit 402.

[0045] The multi-level desensitization rule application unit 401 performs layer-by-layer desensitization processing on sensitive information based on the multi-level desensitization rules (such as field level, record level, and file level) provided by the desensitization strategy management center 3, ensuring the thoroughness and flexibility of desensitization. When processing sensitive information, the multi-level desensitization rule application unit 401 supports custom retention of a specified length of prefix and suffix in the sensitive information, replacing the middle part with dynamically generated mask characters or using other encryption technologies for protection.

[0046] When the desensitization strategy specifies verification rules, the dynamic verification and exception handling unit performs dual verification on sensitive information before and after desensitization to ensure the accuracy and compliance of the desensitization process; at the same time, it promptly captures, records and reports any abnormal situations that occur during the process to ensure the stable operation of the system.

[0047] In this implementation, the log feedback and storage module 5 is responsible for seamlessly feeding back the de-identified log data to the application system log framework for storage, and also supports exporting the de-identified logs to an external storage system or data analysis platform. In this implementation, the system monitoring and log auditing module 6 is used to monitor the system's operating status, performance indicators, and desensitization effect, and generate detailed audit logs for subsequent security review and optimization.

[0048] In this implementation, the policy configuration and update interface 7 allows external systems, security administrators, or automated tools to dynamically update the de-identification policies and verification rules in the de-identification policy management hub 3 through a graphical interface or API interface, in order to quickly respond to ever-changing sensitive information protection needs and compliance requirements.

[0049] Specifically, during use, after the system starts, the log acquisition module 1 automatically reads and monitors the extended interface of the application system's log framework, and acquires the application system's raw log data in real time or at predetermined time intervals, supporting multiple log frameworks and formats; Subsequently, the Sensitive Information Recognition Engine 2 utilizes its built-in advanced automated recognition algorithm set, including a local sensitive information precise positioning unit and a global sensitive information intelligent scanning unit, to perform in-depth analysis of the raw log data, automatically and accurately identifying and locating various types of sensitive information, such as personal identity information and bank card numbers. The desensitization strategy management center 3, as the core of the strategy, is responsible for storing, classifying, updating, and dynamically providing global and local desensitization strategies. The intelligent desensitization processing engine 4 performs highly customized automated desensitization processing on the identified sensitive information according to these strategies, generating desensitized log data that meets security standards. During this process, the multi-level desensitization rule application unit supports custom retention of the prefix and suffix of sensitive information, and performs masking or encryption processing on the middle part. The log feedback and storage module 5 is responsible for seamlessly feeding back the de-identified log data to the application system log framework for storage, and supports exporting the de-identified logs to an external storage system or data analysis platform. The system monitoring and log auditing module 6 continuously monitors the system's operating status, performance indicators, and desensitization effect, generating detailed audit logs for subsequent security reviews and optimization. Meanwhile, the policy configuration and update interface 7 allows external systems, security administrators, or automated tools to dynamically update the desensitization policies and verification rules in the desensitization policy management hub 3 through a graphical interface or API interface, in order to quickly respond to ever-changing sensitive information protection needs and compliance requirements.

[0050] The entire system, through its highly scalable and modular architecture, can be flexibly integrated into existing application systems as a plug-in or deployed as an independent service in a network environment, enabling automated and intelligent identification and de-identification of various types of system log data.

[0051] It should be noted that although the above embodiments have been described herein, this does not limit the scope of patent protection of this invention. Therefore, any changes and modifications made to the embodiments described herein based on the innovative concept of this invention, or equivalent structural, procedural, or functional transformations made using the description and drawings of this invention, directly or indirectly applying the above technical solutions to other related technical fields, are all included within the scope of protection of this invention.

Claims

1. An automated identification and desensitization system, characterized in that, The system includes: The log acquisition module (1) is configured to automatically read and monitor the extended interface of the application system log framework, and acquire the raw log data of the application system in real time or at predetermined time intervals, supporting multiple log frameworks and formats; Sensitive information identification engine (2) has an advanced set of automated identification algorithms built in, which is used to deeply analyze raw log data and automatically and accurately identify and locate various sensitive information in it; The desensitization strategy management center (3) is the core of the system's strategy and is responsible for storing, classifying, updating, and dynamically providing global and local desensitization strategies. The intelligent desensitization processing engine (4) performs highly customized automated desensitization processing on the sensitive information identified by the sensitive information identification engine (2) according to the strategy provided by the desensitization strategy management center (3), and generates desensitized log data that meets security standards. The log feedback and storage module (5) is responsible for seamlessly feeding back the de-identified log data to the application system log framework for storage, and also supports exporting the de-identified logs to an external storage system or data analysis platform. The system monitoring and log auditing module (6) is used to monitor the system's operating status, performance indicators and desensitization effect, and generate detailed audit logs for subsequent security review and optimization.

2. The automated identification and desensitization system according to claim 1, characterized in that, The sensitive information includes, but is not limited to, personal identification information, bank card number, ID number, contact information, and geographical location information.

3. The automated identification and desensitization system according to claim 1, characterized in that, The desensitization strategy includes sensitive information identification rules, multi-level desensitization rules, verification rules, and anomaly handling mechanisms.

4. The automated identification and desensitization system according to claim 1, characterized in that, The sensitive information identification engine (2) further includes: The local sensitive information precise location unit (201) uses the log printing statements marked by specific syntax or annotations in the application system code to accurately locate and extract local sensitive information in complex raw log data. The global sensitive information intelligent scanning unit (202) combines the sensitive information identification rules provided by the desensitization strategy management center (3) to perform a comprehensive and efficient global sensitive information scan on the original log data.

5. The automated identification and desensitization system according to claim 4, characterized in that, The global sensitive information intelligent scanning unit also has an adaptive learning function, which can dynamically adjust the scanning strategy and recognition rules based on historical scanning results and feedback, thereby improving the accuracy and efficiency of sensitive information recognition.

6. The automated identification and desensitization system according to claim 1, characterized in that, The intelligent desensitization processing engine (4) further includes: The multi-level desensitization rule application unit (401) performs layer-by-layer desensitization processing on sensitive information according to the multi-level desensitization rules provided by the desensitization strategy management center (3), ensuring the thoroughness and flexibility of desensitization. The dynamic verification and anomaly handling unit (402) performs dual verification on sensitive information before and after desensitization when the desensitization strategy specifies verification rules, ensuring the accuracy and compliance of the desensitization process; at the same time, it promptly captures, records and reports any abnormal situations that occur during the processing, ensuring the stable operation of the system.

7. The automated identification and desensitization system according to claim 6, characterized in that, When processing sensitive information, the multi-level desensitization rule application unit supports custom retention of a specified length of prefix and suffix in the sensitive information, replacing the middle part with dynamically generated mask characters or using other encryption technologies for protection.

8. The automated identification and desensitization system according to claim 1, characterized in that, The system also includes a policy configuration and update interface (7), which allows external systems, security administrators or automated tools to dynamically update the desensitization policies and verification rules in the desensitization policy management hub (3) through a graphical interface or API interface, so as to quickly respond to the ever-changing sensitive information protection needs and compliance requirements.

9. The automated identification and desensitization system according to any one of claims 1 to 8, characterized in that, The system is designed with a highly scalable and modular architecture, which can be flexibly integrated into existing application systems as a plug-in or deployed as an independent service in a network environment. It can interact efficiently with the application system log framework through standard interfaces to achieve automated and intelligent identification and de-identification of various application system log data.

Images