Communication message log desensitization system, method and device, medium and program product

Abstract

The invention discloses a communication message log desensitization system, method and device, a medium and a program product. The system comprises: a client adaptation module, which is used for receiving byte data and a message type identifier of an original communication message from a service system through an application program interface; the analysis rule base is used for storing message structure configuration information of various message formats; the processing function warehouse is used for registering analytic functions of various message formats; the analysis processing engine is used for analyzing the byte data into tree structure data based on the message structure configuration information and an analysis function; the desensitization rule base is used for storing desensitization rules of various message types; and the desensitization execution engine is used for synchronizing the traversed tree structure data and the desensitization rule base so as to position a target message domain, and desensitizing the target message domain to obtain desensitized structured log data. According to the embodiment of the invention, the method can desensitize the complex message log, and the desensitization process is low in invasion to a service system and is low in maintenance cost.

Description

Technical Field

[0001] This application belongs to the field of data security, and in particular relates to a communication message log desensitization system, method, device, medium and program product. Background Technology

[0002] In modern financial industry information systems, communication message logs are an important auxiliary tool for troubleshooting. However, these messages often contain a wealth of sensitive data elements, such as customer privacy information and business parameters. With increasingly stringent cybersecurity regulations and growing corporate awareness of information security, the protection of sensitive information has become paramount. Sensitive information in transmitted messages, such as bank card numbers, names, and mobile phone numbers, is easily leaked during transmission, storage, and processing, thereby affecting user privacy and the compliance of the system's handling of sensitive data. Therefore, anonymizing communication message logs has become a rigid requirement for the security and compliance of financial information systems.

[0003] There are many standard formats for communication messages between financial systems, such as ISO8583, Key-Value, TLV, JSON, and XML. The message log de-identification methods in related technologies mainly focus on processing message objects that already have a tree-like data structure or index structure, such as JSON, XML, and Key-Value formats. These methods can handle messages of structured data types. However, for messages in binary or mixed encoding formats such as ISO8583 and TLV, which are widely used in the financial field, due to their complex format and lack of self-description, it is difficult to directly use the above de-identification methods for log de-identification. They often need to be converted into intermediate formats first, which increases the processing complexity and performance overhead.

[0004] Furthermore, a common method for log masking is to integrate masking components into the business system, achieving masking by intercepting communication requests or replacing strings in the log output stream. This approach is deeply coupled with business logic, increasing the complexity of the business system, impacting its performance and stability, and making upgrades and maintenance difficult. Moreover, related solutions typically treat message parsing and masking as two independent processes, configuring and managing them separately. This separation makes it difficult to guarantee rule consistency; when message formats or masking requirements change, both sets of rules must be modified simultaneously, resulting in high maintenance costs and a high risk of errors. Summary of the Invention

[0005] This application provides a communication message log desensitization system, method, device, medium, and program product, which can desensitize complex message logs. The desensitization process has low intrusion into business systems and low maintenance costs.

[0006] In a first aspect, embodiments of this application provide a communication message log de-identification system, including: a client adaptation module and a server, wherein the server includes a service parsing rule base, a parsing processing engine, a processing function repository, a de-identification rule base, and a de-identification execution engine; The client adaptation module is used to receive message requests from the business system through the application programming interface. The message request includes the byte data of the original communication message and the message type identifier. The parsing rule base is used to store message structure configuration information for various message formats. The message structure configuration information defines the message field format in a tree structure. A function repository for registering parsing functions for various message formats; The parsing processing engine is used to call the target message structure configuration information corresponding to the message type identifier from the parsing rule base and the target parsing function corresponding to the message type identifier from the processing function repository. Based on the target message structure configuration information and the target parsing function, the byte data of the original communication message is parsed into a hierarchical tree structure data. The de-identification rule library is used to store de-identification rules corresponding to various message types. The de-identification rules include de-identification fields organized in a trie structure, and the de-identification fields are the message fields that need to be de-identified. The de-identification execution engine is used to synchronously traverse the tree-structured data and de-identification rule base output by the parsing and processing engine to locate the target message fields that need to be de-identified in the tree-structured data, and to perform de-identification processing on the target message fields to obtain de-identified structured log data.

[0007] Secondly, embodiments of this application provide a method for de-identifying communication message logs, applied to the communication message log de-identification system as described in the first aspect, the method comprising: Receive message requests from the business system through the application programming interface. The message request includes the byte data of the original communication message and the message type identifier. The pre-configured parsing rule base is queried based on the message type identifier to obtain the target message structure configuration information corresponding to the message type identifier; The pre-configured processing function repository is queried based on the message type identifier to obtain the target parsing function corresponding to the message type identifier; Based on the target message structure configuration information and the target parsing function, the byte data of the original communication message is parsed into a tree-structured data with hierarchical relationships; Synchronously traverse the tree-structured data and the pre-configured de-identification rule base to locate the target message fields that need to be de-identified in the tree-structured data; The target message field is anonymized to obtain anonymized structured log data.

[0008] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; When the processor executes computer program instructions, it implements a communication message log desensitization method as described in the second aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the communication message log desensitization method as described in the second aspect.

[0010] Fifthly, embodiments of this application provide a computer program product in which instructions are executed by the processor of an electronic device, causing the electronic device to perform the communication message log desensitization method as described in the second aspect.

[0011] In this embodiment, by establishing a parsing rule base and pre-configuring message structure configuration information for various message formats, the system can directly receive and parse raw message byte data. This achieves native support for various message protocols and is also applicable to complex non-text format raw messages, meeting the need for log anonymization of various raw message data. By providing a standardized client adaptation module, business systems only need to call a simple interface to send raw message data to the system for anonymization processing. Business systems do not need to integrate complex parsing and anonymization logic, making the anonymization processing a completely independent service. The terminal implementation decouples the system from the business system, ensuring its high performance and stable operation. By associating the parsing rule base with the de-identification rule base, the parsing configuration and de-identification configuration of the message fields correspond one-to-one. The use of a tree-structured data as an intermediate unified representation ensures the consistency of the data flow from parsing to de-identification, greatly simplifying system maintenance. Moreover, during de-identification, by synchronously traversing the tree-structured message data and de-identification rules, the message fields that need to be de-identified can be quickly and accurately located, avoiding the performance overhead of performing field-by-field rule matching across the entire message, thus achieving efficient de-identification. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 These are schematic diagrams of a communication message log desensitization system provided in some embodiments of this application; Figure 2 These are schematic diagrams of communication message log desensitization systems provided in other embodiments of this application; Figure 3 This is a flowchart illustrating a communication message log desensitization method provided in some embodiments of this application; Figure 4 This is a flowchart illustrating a communication message log desensitization method provided in some other embodiments of this application; Figure 5 These are schematic diagrams of electronic devices provided in some embodiments of this application. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] Communication message log: In a communication system, this refers to a log file or data sequence specifically recording the data exchange process. It captures in detail the content, timestamps, status information, and interaction details of messages sent and received between devices or applications, similar to an activity log of the communication process.

[0018] Coroutine: A programming concept referring to a function that can pause execution and then resume from the pause point. It allows execution to be interrupted at a specific point inside the function, with other code being executed instead, and then returned to the interruption point to continue running, thus enabling cooperative multitasking.

[0019] GZIP: GZIP is a file compression format that strictly follows the gzip format defined in RFC 1952.

[0020] Message field: refers to any node in a message tree structure, regardless of its level or whether it is a leaf node. It is an abstract general term.

[0021] Data fields: These are relatively independent basic data units in a message, and are nodes directly defined in the parsing configuration. Examples include "Field 2: Primary Account" in ISO 8583, or the top-level field "cardNo" in JSON. Data fields are a type of message field. When a data field has no substructures, it is itself a leaf node-level message field.

[0022] Subfield: A subfield refers to a further subdivided data unit nested within a data field. For example, an "address field" might contain subfields such as "province," "city," and "street." A subfield is a type of message field. It specifically refers to a non-top-level message field with a parent-child relationship.

[0023] Before providing a more detailed description of the embodiments of this application, related technologies are introduced. As mentioned earlier, message log desensitization methods in related technologies mainly focus on processing message objects that already have a tree-like data structure or index structure, such as JSON, XML, and Key-Value formats. These log desensitization methods require locating sensitive fields based on the tree-like data structure of JSON or XML, or directly performing regular expression matching on the text-based message content, and performing desensitization processing and logging based on structured message data. They cannot support log desensitization of messages and subfields in complex formats such as ISO8583 and TLV.

[0024] Some message log anonymization methods typically employ request interception mechanisms or log string replacement, which are highly intrusive to the client. In request interception mode, the system needs to intercept and anonymize each request at the application layer before forwarding it, increasing client complexity and maintenance costs. In log string replacement mode, the system needs to scan and match the original log content before replacing sensitive information. This approach not only increases the load on the application server but may also lead to delays in business logic.

[0025] Furthermore, message parsing systems often lack log masking capabilities, typically only performing conversions between different message types. This separates message parsing from log masking, making them relatively independent. During the masking phase, developers need to actively align the masking rules with the message parsing rules; or use regular expressions or sensitive feature hash sets to locate the sensitive fields to be processed. Regular expression-based matching methods struggle to handle complex message structure variations, easily leading to false or missed matches. Moreover, maintaining sensitive feature hash sets is labor-intensive and prone to sensitive information leakage due to incomplete feature libraries.

[0026] Furthermore, different types of messages often require different parsing methods and de-identification strategies, and message parsing is often implemented internally by each system, lacking unified and standardized parsing and configuration. This decentralized configuration approach results in low standardization of rule configurations, and adjustments to parsing and de-identification rules require synchronous modifications, easily leading to inconsistencies. Moreover, when adding new message types or de-identification rules, the entire configuration system often needs to be redesigned, increasing the cost and risk of system upgrades.

[0027] In view of this, and in order to solve the technical problems existing in the related technologies, embodiments of this application provide a communication message log de-identification system, method, device, medium, and program product. The core of embodiments of this application lies in providing a log de-identification system that can support multiple message formats, is low-intrusive, and highly efficient.

[0028] The communication message log desensitization system provided in the embodiments of this application will be described below.

[0029] The communication message log anonymization solution provided in this application can be applied in the financial and payment industries. For example, when business systems interact with gateways, the communication messages can be anonymized to protect user identity and account information.

[0030] like Figure 1 As shown in the embodiments of this application, the overall architecture of the communication message log de-identification system mainly includes: a client adaptation module and a server. The server includes a service parsing rule base, a parsing processing engine, a processing function repository, a de-identification rule base, and a de-identification execution engine.

[0031] The client adaptation module is used to receive message requests from the business system through the application programming interface. The message request includes the byte data of the original communication message and the message type identifier.

[0032] In some embodiments of this application, the client adaptation module includes a software development kit (SDK) for integration with business systems. Business systems can perform log anonymization by integrating the SDK. When a business system generates communication messages that require logging, such as received or sent transaction messages, it only needs to call the application programming interface (API) provided by the SDK to encapsulate the byte data and message type identifier of the original communication message into a message request and send it to the server for processing. Because the communication message log anonymization system in this embodiment directly receives the original message data in byte form, each business system does not need to convert complex message data to a unified format, resulting in low intrusion on the client.

[0033] In some embodiments of this application, the client SDK and the server can confirm each other's liveness through heartbeat messages.

[0034] In some embodiments of this application, multiple server addresses can be set in the SDK configuration file. The SDK can have built-in load balancing and failover mechanisms, so that for the client, load balancing and high availability can be achieved through multiple servers.

[0035] In some embodiments of this application, the client adaptation module further includes a server-side communication interface. The server-side communication interface is used to receive message requests sent by the business system. This interface can be based on a concurrent coroutine network input / output model, such as a multi-coroutine concurrent network model. In this way, message requests can be sent to the server asynchronously and non-blockingly, eliminating the need for the client to wait for the server's de-identification processing results, thus reducing latency. The concurrent coroutine network input / output model uses a main coroutine to listen for and receive new requests and distribute them to multiple worker coroutines. Each worker coroutine manages request processing on multiple connections. Based on this, when the server receives a message request, it can distribute the request to different processing coroutines through a coroutine pool mechanism. This allows the server-side communication interface to receive message requests from the business system with high concurrency, i.e., to process a large number of concurrent requests simultaneously. Each coroutine independently processes one message request, avoiding the performance bottleneck of traditional blocking I / O models.

[0036] In some embodiments of this application, the main input parameters of the API include the message type identifier of the original communication message (such as "ISO8583", "TLV", "JSON", etc.) and the byte data. The SDK internally encapsulates these two parameters into a structured network request, i.e., a message request. To improve transmission efficiency and effectively reduce network transmission overhead and processing burden, the business system can choose to compress the byte data using GZIP via the SDK before sending it to the server. Correspondingly, the processing coroutine can determine whether to directly process the subsequent message byte data, or to decompress the byte data using GZIP, based on the message type identifier in the message request header.

[0037] In some embodiments of this application, the client adaptation module further includes a request buffer queue. The request buffer queue is used to buffer received message requests and distribute them to processing coroutines according to a producer-consumer model. Based on this, when a message request is fully read, it is not immediately processed in the current coroutine, but is packaged into a task object and placed in the request buffer queue. Then, the worker coroutine retrieves the task from the queue for processing. This design has the following advantages: First, the queue buffering mechanism smooths out peak and valley loads in request processing, avoiding the impact of sudden high concurrency on the system; second, by adjusting the queue length and the number of worker coroutines, the system's processing capacity and resource consumption can be flexibly controlled.

[0038] The parsing rule base is used to store message structure configuration information for various message formats. The message structure configuration information defines the message field format in a tree structure.

[0039] In some embodiments of this application, to support the parsing of various message formats such as JSON, XML, ISO8583, Key-Value, and TLV, a parsing rule base is set up in the system. The parsing rule base may include message structure configuration information for various message formats such as JSON, XML, ISO8583, Key-Value, and TLV. The message structure configuration information for each message format can be configured through a configuration file. Message structure configuration information refers to a configuration file or configuration class used to describe the entire message format structure. The message structure configuration information defines detailed message field formats in a tree structure, such as: what data fields (fields) the message has, the encoding method of each field (e.g., BCD, ASCII), the definition of the length prefix, the nesting relationship of fields (e.g., sub-field structure), and the data type of each field. In the parsing rule base, the message format and parsing rules are mainly defined in a tree-structured configuration file or a separate configuration class, making configuration simple and easy to use.

[0040] For example, for ISO8583 format messages, the system supports multiple data field format configuration rules, allowing configuration of various data field types and composite subfield types. For data fields, their length can be flexibly defined as a variable-length field derived from a prefix, a fixed length field, or one without a length prefix. For composite subfields, Tag subfields, TLV subfields, or unnamed sequential subfields can be defined. For the encoding methods of length prefixes and field values, mainstream encoding formats such as BCD, left-aligned BCD, extended BCD, ASCII, Hex, and Binary are all supported. For XML format messages, only a simple configuration class needs to be defined, and the class members need to be configured for each data field and its corresponding XML tag name. For JSON, Key-Value, and TLV format messages, only the type of each data field and the description of the nested structure need to be specified in the configuration file according to the message field specification.

[0041] The processing function repository is used to register parsing functions corresponding to various message formats. The parsing function for each message format type can be configured via a configuration file, and the parsing function for each message format type can be registered in the processing function repository when the system starts.

[0042] In some embodiments of this application, for most message types, their parsing functions can be defined and registered in the processing function repository according to the corresponding message processing function template. For custom message types or special message parsing requirements, custom message processing functions can also be defined and registered in the processing function repository. For data fields that need to be de-identified for each specific message type, they can be defined in the corresponding message structure configuration information with the same data field names as those in the parsing configuration.

[0043] The parsing processing engine is used to call the target message structure configuration information corresponding to the message type identifier from the parsing rule base and the target parsing function corresponding to the message type identifier from the processing function repository. Based on the target message structure configuration information and the target parsing function, the byte data of the original communication message is parsed into a hierarchical tree structure data.

[0044] In some embodiments of this application, the parsing processing engine can, based on the message type identifier in the received message request, call the message structure configuration information corresponding to the message type identifier from the parsing rule base (i.e., the target message structure configuration information), and call the parsing function corresponding to the message type identifier from the processing function repository (i.e., the target parsing function). Based on the target message structure configuration information and the target parsing function, the byte data in the message request is parsed. Different message format types have nested structure characteristics, meaning each data field may contain subfields. Therefore, the message parsing process can ultimately convert messages of different format types into a hierarchical tree structure data. This tree structure preserves the logical relationship between the data fields of the original message, facilitates the unification of parsing and desensitization configurations, and enables access to sensitive fields during subsequent desensitization. It also improves the readability of the final recorded log results.

[0045] In some embodiments of this application, the message type identifier may include the system type and specific message specification code of the message. The parsing and processing engine can match a unique parsing function in the processing function registration repository based on the message type identifier. The parsing function is responsible for parsing and converting the message in byte data form.

[0046] In this embodiment, the parsing function is combined with the message structure configuration information to handle the specific parsing of the message. Based on this, different abstract parsing schemes can be defined for different message formats. For example, for message formats where the data is in byte form, such as ISO8583, TLV, and other binary messages, the abstract parsing scheme can be defined as a traversal on a byte buffer. The message structure configuration information determines how many bytes to read from the byte buffer each time, how to interpret these read bytes as values ​​of a specific type, and how to organize the storage location and logical relationships of these values. The parsing engine can use a byte buffer cursor to sequentially read specific bytes according to the message structure configuration information, decode the read bytes according to the specified encoding, and fill the values ​​into the corresponding nodes according to the tree configuration, recursively processing subfields. As another example, for message formats where the data is in string text form, such as JSON, XML, and Key-Value text messages, the corresponding abstract parsing scheme can be defined as structured string parsing. The message structure configuration information defines the prototype of the message string structure, and the corresponding parsing function converts the byte data into a unified tree structure data within the system. This tree structure fully preserves the hierarchical relationship of the messages, with each node containing information such as the domain name, path, and value.

[0047] In the parsed tree-structured message data, data fields at each level can be accessed by field name. For data fields containing subfields, access can be performed layer by layer by field. This process converts messages of various formats into a unified internal data structure, offering high flexibility and thus satisfying the requirements for unified message field de-identification configuration definitions and flexible de-identification processing.

[0048] The de-identification rule library stores de-identification rules for various message types. The de-identification rules include de-identification fields organized in a trie structure, and the de-identification fields are the message fields that need to be de-identified.

[0049] In some embodiments of this application, the system has a desensitization rule library, which contains desensitization rules corresponding to various message types. The message type here is more granular and does not refer to the message format type described above. Different message types can be the same message format type. The message type here is the message type jointly determined by the sending system and the type representation carried by the message.

[0050] The de-identification rules include de-identification fields organized in a trie structure. De-identification rules for each message type can be configured via configuration files. A de-identification field refers to a message field marked as requiring de-identification processing within the tree structure formed after message parsing. The de-identification fields in the rules are stored in a trie structure, which saves the overhead of traversing the tree structure during the de-identification process, improving processing efficiency and achieving efficient de-identification. Furthermore, unifying the message structure and de-identification configuration through configuration files enhances the maintainability and scalability of message parsing and de-identification rules.

[0051] In some embodiments of this application, the definition of the data to be anonymized is extended based on predefined data fields requiring anonymization, combined with message structure configuration information. Specifically, the definition of the data to be anonymized is extended by using predefined fields as a basis and then combining the message format definition in the message structure configuration information, such as which data field is what type, what subfields the data field has, etc. In other words, the configuration of the data to be anonymized in the embodiments of this application is not just a simple list of fields, but is associated with the message structure configuration, so that the definition of the data to be anonymized can utilize the information of the message structure, such as the subfield structure, thereby supporting more complex anonymization requirements. The field configuration of the data to be anonymized corresponds one-to-one with the configuration in the message structure configuration information, thereby ensuring the consistency of the field. Based on this, the data anonymization rules in the data anonymization rule base support the configuration of anonymization for ordinary data fields, named subfields, sequential subfields, and specified byte index ranges within data fields. For example, for a composite field (such as a field with subfields), it is possible to specify that its subfields be anonymized. For example, if a message has an address field containing subfields such as province, city, and street, then the "city" subfield can be desensitized separately. Another example is that for the value of a field, you can specify to desensitize only a portion of the bytes within that field's value. For instance, if a field's value is 20 bytes, you can specify to desensitize only bytes 3 to 10.

[0052] In some embodiments of this application, when configuring desensitization for named subdomains or sequential subdomains, the parent domain field name and the subdomain field name can be separated by ".". When configuring desensitization for the byte index of a domain value, the bytes from the xth to the yth index of the domain can be desensitized using "[xy]".

[0053] For example, the configured desensitization fields are shown in Table 1 below: Table 1

[0054] In some embodiments of this application, a trie structure is used to store the de-identified fields for tree-structured message data. Since the message structure configuration information is highly consistent with the de-identification rule configuration, '.' can be used as the hierarchical identifier of the trie, constructing a de-identification rule trie with field names as node names. During the de-identification process, by performing a synchronous depth-first traversal of the de-identification rule trie and the message tree with the same prototype structure, the fields requiring de-identification can be quickly located.

[0055] The de-identification execution engine is used to synchronously traverse the tree-structured data and de-identification rule base output by the parsing and processing engine to locate the target message fields that need to be de-identified in the tree-structured data, and to perform de-identification processing on the target message fields to obtain de-identified structured log data.

[0056] In some embodiments of this application, the de-identification execution engine obtains tree-structured data generated by the parsing processing engine, hereinafter referred to as the message tree. Based on the message type identifier, it loads the corresponding de-identification rules from the de-identification rule library. These de-identification rules are in a trie structure, hereinafter referred to as the de-identification rule trie. The de-identification execution engine synchronously performs a depth-first traversal of both the message tree and the de-identification rule trie. When the path of a node in the message tree completely matches the path of a node in the de-identification rule trie, that node is identified as the target message field to be de-identified. Thus, during the de-identification execution process, by synchronously performing a depth-first traversal of the de-identification rule trie and the message tree with the same prototype structure, the fields requiring de-identification can be quickly located, reducing the time overhead of repeatedly traversing the message tree for de-identification rule matching.

[0057] After identifying the target message field, the de-identification execution engine can trigger a de-identification action to process the target message field, thereby obtaining de-identified structured log data. During the de-identification process, the target message field can be replaced byte-by-byte. In this way, the de-identification process does not change the tree-structured data obtained from message parsing, but only modifies the byte data of the fields that need to be de-identified.

[0058] In some embodiments of this application, different services or regions may have different de-identification standards for the same data. Therefore, de-identification strategies corresponding to message fields can be pre-configured based on actual needs. This way, when de-identifying a target message field, the pre-configured de-identification strategy can be used. In this way, the most suitable de-identification method can be used to de-identify the message field. If no de-identification strategy corresponding to the target message field is pre-configured, then the default de-identification function (e.g., a function that replaces all characters with *) will be used to de-identify the target message field.

[0059] For example, the anonymized structured log data is shown below: { “1”:“***”, “2”:{ “A”:“ABC”, "B":"0000000000000000", } } In this embodiment, by establishing a parsing rule base and pre-configuring message structure configuration information for various message formats, the system can directly receive and parse raw message byte data. This achieves native support for various message protocols and is also applicable to complex non-text format raw messages, meeting the need for log anonymization of various raw message data. By providing a standardized client adaptation module, business systems only need to call a simple interface to send raw message data to the system for anonymization processing. Business systems do not need to integrate complex parsing and anonymization logic, making the anonymization processing a completely independent service. The terminal implementation decouples the system from the business system, ensuring its high performance and stable operation. By associating the parsing rule base with the de-identification rule base, the parsing configuration and de-identification configuration of the message fields correspond one-to-one. The use of a tree-structured data as an intermediate unified representation ensures the consistency of the data flow from parsing to de-identification, greatly simplifying system maintenance. Moreover, during de-identification, by synchronously traversing the tree-structured message data and de-identification rules, the message fields that need to be de-identified can be quickly and accurately located, avoiding the performance overhead of performing field-by-field rule matching across the entire message, thus achieving efficient de-identification.

[0060] In some embodiments of this application, see Figure 2 The server also includes a logging module and a post-monitoring and analysis module.

[0061] The log recording module is used to write the anonymized structured log data into the log storage system.

[0062] In some embodiments of this application, after the message parsing and de-identification processing are completed, the de-identified structured log data can be written to the log storage system asynchronously without returning it to the business system, thereby achieving decoupling from the business system and ensuring the high performance and high reliability of the communication message log de-identification system.

[0063] In some embodiments of this application, the log storage system may employ a Kafka message queue.

[0064] The post-monitoring analysis module is used to analyze the anonymized structured log data stored in the log storage system and obtain analysis results. The analysis results are used to indicate the risk level and sensitive fields of the anonymized structured log data. Sensitive fields are those fields that need to be anonymized but have not actually been anonymized.

[0065] In some embodiments of this application, the communication message log anonymization system can be configured with custom log analysis rules and strategies, which are then submitted to a task queue for execution to analyze and evaluate the anonymized structured log data. The analysis results can be sent as notifications to promptly identify and address any unanonymized sensitive information.

[0066] In some embodiments of this application, the post-supervision analysis module may include a timed task scheduler, a multi-dimensional analysis unit, and a comprehensive analysis unit.

[0067] The scheduled task scheduler is used to periodically trigger analysis tasks on the anonymized structured log data already stored in the log storage system. The scheduled task scheduler can automatically trigger the analysis process of the stored historical structured log data according to a preset time period (such as hourly, daily, or weekly).

[0068] The multi-dimensional analysis unit is used to perform multi-dimensional analysis on the anonymized structured log data to obtain multi-dimensional analysis results. The multi-dimensional analysis may include at least one of the following: predefined feature matching, word vector similarity calculation, and message analysis agent parsing.

[0069] In some embodiments of this application, analysis methods based on predefined feature matching, such as regular expressions and feature hashing, require a series of predefined feature matching functions. During analysis, each time a field's value in the structured log data is received as input, a Boolean value is returned, and the Boolean value is used to determine whether the field may contain sensitive information that has not been desensitized.

[0070] In some embodiments of this application, the word vector similarity evaluation method can receive the same input as the feature matching method. The desensitized structured log data is segmented by a communication message segmenter and then converted into high-dimensional feature vectors through a pre-trained message embedding model. A preset sensitive information feature vector library is called, and the similarity between the field feature vector and the standard feature vector of the sensitive class in the library is calculated. A confidence threshold is set. If the similarity is higher than the threshold, it is determined that the field may contain unsensitized sensitive information and is marked as a sensitive field.

[0071] In some embodiments of this application, the message analysis agent can be configured to: retrieve the corresponding communication message specification based on the message type associated with the de-identified structured log data; analyze the communication message specification to obtain the parsing result, which includes the definition, business attributes, and sensitivity level standards of each field in the message; determine the de-identified fields in the message that need to be de-identified based on the parsing result; and perform a field-level comparison between the de-identified fields and the actual de-identification status in the de-identified structured log data to locate the sensitive fields in the de-identified structured log data. Based on this, when using a message analysis agent for analysis, firstly, the agent can automatically retrieve and match the corresponding communication message specification document based on the input structured log data to be detected. Then, the agent performs structured parsing of the specification document, clarifies the definition, business attributes, and sensitivity level standards of each field in the message, and determines the necessity of desensitizing each field in the input structured log data based on the parsing results. On this basis, the agent compares the desensitized fields required by the specification with the actual desensitization status of the input structured log data at the field level, realizing a quantitative evaluation of the desensitization effect, and accurately locating sensitive fields that are "required to be desensitized by the specification but not actually desensitized".

[0072] For example, a message analysis agent can analyze anonymized structured log data based on the following prompts: "#Role You are an experienced and professional expert in analyzing anonymized message results, possessing profound professional knowledge and keen insight, capable of accurately and efficiently analyzing user-input messages. Through meticulous searching of relevant communication message specifications, you conduct in-depth analysis of each field of the uploaded message according to the specification definition, accurately identifying the fields that need anonymization, and providing an objective and fair evaluation of the anonymization results of the current message.

[0073] ##Skill ###Skill 1: Standardized Search 1. Carefully study the message entered by the user and use professional tools to accurately retrieve the specification that matches the message from the message specification database.

[0074] 2. Conduct a thorough examination of each field in the specification, deeply explore the meaning of each field, and use professional knowledge to accurately determine whether the field involves sensitive information.

[0075] ### Skill 2: Evaluation of Message Desensitization Results 1. Strictly adhere to the interface specification definition, comprehensively and from multiple perspectives evaluate whether the anonymization results of the uploaded messages meet expectations, and provide a scientific and reasonable anonymization score.

[0076] 2. Accurately locate and filter out un-masked fields containing sensitive information.

[0077] 3. Each sensitive information field is output in JSON format, including two fields: result_score (the score of the desensitization result) and not_desensitized (whether this field is a sensitive field that has not been desensitized), with detailed annotations of the reasons for the need for desensitization.

[0078] ##limit - Only process content closely related to message anonymization analysis, and resolutely refuse to answer irrelevant questions.

[0079] - The output content must be organized strictly according to the given format and must not deviate from the framework requirements.

[0080] - The evaluation results must be based on accurate and error-free message specification definitions, and arbitrary subjective judgments should be avoided.

[0081] - Obtain the specification information from the message return database using specialized tools. For example, taking the analysis of structured log data by a multi-dimensional analysis unit using word vector similarity calculation and message analysis agent parsing as an example, the multi-dimensional analysis unit first vectorizes the structured log data to be analyzed, extracts the semantic features of the structured log data through vector similarity calculation technology, and compares them with the sensitive feature sequences pre-stored in the database to achieve preliminary screening of sensitive information. Then, it calls the message analysis agent to automatically retrieve and match the corresponding message specifications, and performs field-by-field parsing on the message fields associated with the structured log data. On the one hand, it determines whether the fields contain sensitive information; on the other hand, it verifies whether the anonymization results of sensitive fields meet the specification requirements, and simultaneously outputs risk scores and judgment results.

[0082] The comprehensive analysis unit is used to integrate the multi-dimensional analysis results output by the multi-dimensional analysis unit and generate an analysis report that includes risk levels and a list of sensitive fields.

[0083] In some embodiments of this application, the comprehensive analysis unit may employ a de-identification result analysis agent. After the multi-dimensional analysis unit completes the analysis of the structured log data, the de-identification result comprehensive analysis agent can fuse the multi-dimensional analysis results output by the multi-dimensional analysis unit, employing a weighted intelligent analysis mechanism (assigning higher weight coefficients to high-confidence outputs) to ultimately determine the information leakage risk level and the list of de-identified fields that need improvement and de-identification.

[0084] The above methods enable intelligent post-monitoring analysis of anonymized logs, allowing for the timely detection and feedback of potentially unanonymized sensitive information, thus assisting in the implementation of sensitive information anonymization procedures. Furthermore, the use of a multi-dimensional analysis mechanism effectively reduces the risk of misjudgment and omissions associated with single analysis methods, significantly improving the overall accuracy and reliability of sensitive data leakage analysis.

[0085] This application provides a communication message log desensitization system that can parse and desensitize original messages in various formats. It provides message log desensitization and recording capabilities based on API calls, offers consistent desensitization strategies and message field parsing configurations, and intelligent desensitization log potential sensitive information analysis functions. It can efficiently complete message log desensitization and provide analysis feedback information to assist in the judgment of sensitive fields and sensitive information.

[0086] Based on the communication message log desensitization method provided in the above embodiments, this application also provides specific implementation methods for the communication message log desensitization method. Please refer to the following embodiments.

[0087] See Figure 3 This is a flowchart illustrating the communication message log desensitization method provided in this application embodiment. This method is applied to, for example... Figures 1-2 The communication message log desensitization system shown includes the following steps 310-360.

[0088] Step 310. Receive a message request from the business system through the application programming interface. The message request includes the byte data of the original communication message and the message type identifier. Step 320. Query the pre-configured parsing rule base according to the message type identifier to obtain the target message structure configuration information corresponding to the message type identifier; Step 330. Query the pre-configured processing function repository according to the message type identifier to obtain the target parsing function corresponding to the message type identifier; Step 340. Based on the target message structure configuration information and the target parsing function, parse the byte data of the original communication message into a tree-structured data with hierarchical relationships; Step 350. Synchronously traverse the tree structure data and the pre-configured de-identification rule base to locate the target message fields that need to be de-identified in the tree structure data; Step 360. De-identify the target message field to obtain de-identified structured log data.

[0089] In this embodiment, by establishing a parsing rule base and pre-configuring message structure configuration information for various message formats, the system can directly receive and parse raw message byte data. This achieves native support for various message protocols and is also applicable to complex non-text format raw messages, meeting the need for log anonymization of various raw message data. By providing a standardized client adaptation module, business systems only need to call a simple interface to send raw message data to the system for anonymization processing. Business systems do not need to integrate complex parsing and anonymization logic, making the anonymization processing a completely independent service. The terminal implementation decouples the system from the business system, ensuring its high performance and stable operation. By associating the parsing rule base with the de-identification rule base, the parsing configuration and de-identification configuration of the message fields correspond one-to-one. The use of a tree-structured data as an intermediate unified representation ensures the consistency of the data flow from parsing to de-identification, greatly simplifying system maintenance. Moreover, during de-identification, by synchronously traversing the tree-structured message data and de-identification rules, the message fields that need to be de-identified can be quickly and accurately located, avoiding the performance overhead of performing field-by-field rule matching across the entire message, thus achieving efficient de-identification.

[0090] In some embodiments of this application, see Figure 4 After obtaining the anonymized structured log data, you can also perform the following steps 370-390.

[0091] Step 370. Write the anonymized structured log data into the log storage system; Step 380. Retrieve the stored, anonymized, structured log data from the log storage system; Step 390. Analyze the obtained anonymized structured log data to obtain analysis results. The analysis results are used to indicate the risk level and sensitive fields of the anonymized structured log data. Sensitive fields are those fields that need to be anonymized but were not actually anonymized.

[0092] In some embodiments of this application, step 390 above includes the following steps 391-3902: Step 3901. Perform multi-dimensional analysis on the de-identified structured log data to obtain multi-dimensional analysis results. The multi-dimensional analysis includes at least one of the following: based on predefined feature matching, word vector similarity calculation, and message analysis agent parsing. Step 3902. Based on the multi-dimensional analysis results, comprehensively determine whether there are sensitive fields in the anonymized structured log data, and generate an analysis report containing risk levels and a list of sensitive fields.

[0093] The specific implementation of each step in the communication message log desensitization method provided in this application embodiment can be found in the relevant description in the device embodiment. To avoid repetition, it will not be repeated here.

[0094] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0095] Electronic device 500 may include processor 501 and memory 502 storing computer program instructions.

[0096] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0097] Memory 502 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to electronic device 500. In a particular embodiment, memory 502 is a non-volatile solid-state memory. Memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory 502 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it performs the operations described in any of the communication message log desensitization methods in the above embodiments.

[0098] The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any of the communication message log desensitization methods in the above embodiments.

[0099] In one example, the electronic device 500 may also include a communication interface 503 and a bus 510. For example, Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0100] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0101] Bus 510 includes hardware, software, or both, that couples components of electronic device 500 together. For example, and not limitingly, the bus may include Accelerated Graphics Port (AGP) or other graphics buses, Enhanced Industry Standard Architecture (EISA) buses, Front Side Bus (FSB), HyperTransport (HT) interconnects, Industry Standard Architecture (ISA) buses, Infinite Bandwidth Interconnects, Low Pin Count (LPC) buses, memory buses, Microchannel Architecture (MCA) buses, Peripheral Component Interconnect (PCI) buses, PCI-Express (PCI-X) buses, Serial Advanced Technology Attachment (SATA) buses, Video Electronics Standards Association Local (VLB) buses, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0102] Furthermore, in conjunction with the communication message log desensitization method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the communication message log desensitization methods in the above embodiments.

[0103] This application also provides a computer program product, including a computer program, which, when executed, implements any of the communication message log desensitization methods described in the above embodiments.

[0104] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0105] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0106] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0107] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0108] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A communication message log anonymization system, characterized in that, include: The client-side adaptation module and the server-side module include a service parsing rule base, a parsing processing engine, a processing function repository, a de-identification rule base, and a de-identification execution engine. The client adaptation module is used to receive message requests from the business system through the application programming interface. The message request includes byte data of the original communication message and a message type identifier. The parsing rule base is used to store message structure configuration information for various message formats. The message structure configuration information defines the message field format in a tree structure. The processing function repository is used to register parsing functions for the various message formats; The parsing processing engine is used to call the target message structure configuration information corresponding to the message type identifier from the parsing rule base and to call the target parsing function corresponding to the message type identifier from the processing function repository, and to parse the byte data of the original communication message into a tree structure data with hierarchical relationship based on the target parsing function and the target message structure configuration information; The de-identification rule base is used to store de-identification rules corresponding to various message types. The de-identification rules include de-identification fields organized in a trie structure, and the de-identification fields are message fields that need to be de-identified. The de-identification execution engine is used to synchronously traverse the tree structure data output by the parsing processing engine and the de-identification rule base to locate the target message field that needs to be de-identified in the tree structure data, and to perform de-identification processing on the target message field to obtain de-identified structured log data.

2. The system according to claim 1, characterized in that, The client adaptation module specifically includes: A software development kit (SDK) is provided for integration into the business system. It is used to encapsulate the byte data and message type identifier of the original communication message of the business system into a message request, and send the message request to the server through an asynchronous call. The server-side communication interface is built based on a concurrent coroutine network input / output model and is used to receive the message requests sent by the business system at high concurrency. A request buffer queue is used to buffer the received message requests and distribute them to processing coroutines in a producer-consumer pattern for processing.

3. The system according to claim 1, characterized in that, The message structure configuration information in the parsing rule base exists in the form of a unified configuration file or configuration class. The message structure configuration information is used to define the data fields, subfields, length prefixes, and encoding methods and data types of field values ​​for various message formats.

4. The system according to claim 1, characterized in that, The de-identification rules stored in the de-identification rule base support de-identification configuration for ordinary data fields, named subfields, sequential subfields, and specified byte index ranges within data fields.

5. The system according to claim 1, characterized in that, The server also includes a log recording module and a post-monitoring and analysis module; The log recording module is used to write the de-identified structured log data into the log storage system; The post-monitoring analysis module is used to analyze the anonymized structured log data stored in the log storage system to obtain analysis results. The analysis results are used to indicate the risk level and sensitive fields of the anonymized structured log data. The sensitive fields are those fields that need to be anonymized but have not actually been anonymized.

6. The system according to claim 5, characterized in that, The post-supervision analysis module includes: A timed task scheduler is used to periodically trigger analysis tasks on the anonymized structured log data already stored in the log storage system. A multi-dimensional analysis unit is used to perform multi-dimensional analysis on the de-identified structured log data to obtain multi-dimensional analysis results; the multi-dimensional analysis includes at least one of the following: based on predefined feature matching, word vector similarity calculation, and message analysis agent parsing; The comprehensive analysis unit is used to fuse the multi-dimensional analysis results output by the multi-dimensional analysis unit and generate an analysis report that includes risk levels and a list of sensitive fields.

7. The system according to claim 6, characterized in that, The message analysis agent is configured as follows: Based on the message type associated with the de-identified structured log data, retrieve the corresponding communication message specification; The communication message specification is parsed to obtain the parsing result, which includes the definition of each field in the message, the service attributes, and the sensitivity level standard; Based on the parsing results, determine the fields in the message that need to be de-identified; The sensitive fields in the de-identified structured log data are compared at the field level with the actual de-identification status in the de-identified structured log data to locate the sensitive fields in the de-identified structured log data.

8. A method for desensitizing communication message logs, characterized in that, The method, applied to the communication message log desensitization system as described in any one of claims 1 to 7, comprises: The application programming interface (API) receives message requests from the business system, the message requests including the byte data of the original communication message and the message type identifier; The pre-configured parsing rule base is queried based on the message type identifier to obtain the target message structure configuration information corresponding to the message type identifier; The pre-configured processing function repository is queried based on the message type identifier to obtain the target parsing function corresponding to the message type identifier; Based on the target message structure configuration information and the target parsing function, the byte data of the original communication message is parsed into a tree-structured data with hierarchical relationships; Synchronously traverse the tree-structured data and the pre-configured de-identification rule base to locate the target message fields that need to be de-identified in the tree-structured data; The target message field is de-identified to obtain de-identified structured log data.

9. The method according to claim 8, characterized in that, After obtaining the anonymized structured log data, the method further includes: Write the de-identified structured log data into the log storage system; Retrieve the stored, de-identified, structured log data from the log storage system; The obtained anonymized structured log data is analyzed to obtain analysis results. The analysis results are used to indicate the risk level and sensitive fields of the anonymized structured log data. The sensitive fields are those fields that need to be anonymized but were not actually anonymized.

10. The method according to claim 9, characterized in that, The analysis of the acquired, de-identified structured log data yields the following results: The anonymized structured log data is subjected to multi-dimensional analysis to obtain multi-dimensional analysis results. The multi-dimensional analysis includes at least one of the following: based on predefined feature matching, word vector similarity calculation, and message analysis agent parsing. Based on the multi-dimensional analysis results, a comprehensive judgment is made as to whether the anonymized structured log data contains sensitive fields, and an analysis report containing risk levels and a list of sensitive fields is generated.

11. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the communication message log desensitization method as described in any one of claims 8-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the communication message log desensitization method as described in any one of claims 8-10.

13. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the communication message log desensitization method as described in any one of claims 8-10.

Images