An information security data acquisition and preprocessing method, system, device and medium

Abstract

The application relates to the technical field of information security, and discloses an information security data acquisition and preprocessing method, system, device and medium, the method comprising the following steps: matching independent analysis parameters according to data source configurations, and obtaining information security data based on the independent analysis parameters, wherein the independent analysis parameters comprise log analysis logic corresponding to the data source configurations; performing semantic alignment on the information security data by using a unified security semantic model to obtain standard security events; performing hierarchical desensitization on the standard security events according to current hierarchical desensitization parameters, and transmitting the standard security events to a user database based on a national encryption algorithm. The application realizes decoupling of data sources and acquisition logic through independent analysis parameters, supports hot plug quick access, and can complete data source change without restarting the process; the accuracy of security event correlation analysis is improved, and the whole process meets the requirements of the network security protection and industry compliance; the overall scheme is efficient, unified in semantics, compliant in nature, and improves the security operation efficiency and emergency response speed.

Description

Technical Field

[0001] This invention relates to the field of information security technology, specifically to an information security data acquisition and preprocessing method, system, device, and medium. Background Technology

[0002] In existing technologies, enterprise security operations teams generally use the built-in collectors or open-source proxies of Security Information and Event Management (SIEM) platforms to aggregate logs from security devices such as firewalls. Traditional solutions receive raw logs from different sources, such as hosts, network devices, middleware, and business systems, one by one through pre-built input plugins. After being parsed using regular expressions / artificial intelligence, the logs are written to backends such as Kafka / Elasticsearch, and then further analyzed by rule engines.

[0003] This model can be quickly implemented in scenarios where the data format is fixed and the meaning of the fields is known. However, when facing industries with high security requirements such as field structure changes with version, vendor proprietary protocols, encrypted channels, compliance desensitization, high concurrency and low latency (e.g., finance, government, energy), the following defects are exposed: (1) Deep plugin coupling: Adding or changing data sources requires secondary development of plugins and restarting the process, resulting in a long online cycle and the inability to achieve minute-level hot-swap; (2) Fragmented semantic mapping: Each plugin maintains its own field mapping table, lacking a unified security semantic model. The same event is parsed into heterogeneous fields in different systems, resulting in additional ETL required for subsequent correlation analysis and a decrease in accuracy; (3) Insufficient compliance and encryption support: Mainstream open-source collectors do not support the national cryptographic transport layer security protocol (Transport Layer). Security (TLS), SM series algorithms, and data classification and desensitization lack native support, which cannot meet the rigid requirements of the Information Security Protection 2.0, the China Banking and Insurance Regulatory Commission, and the National Energy Administration for transmission encryption, storage desensitization, and audit traceability; (4) Lack of dynamic orchestration: When an emergency scenario requires the temporary collection of a certain type of high-frequency event, the existing solution requires manual script distribution and configuration changes on a per-machine basis, and it is impossible to complete policy orchestration and gray rollback with one click in the headquarters console. These defects lead to enterprises still facing pain points such as slow data source access, inconsistent fields, difficulty in compliance auditing, and slow emergency response in actual operation. Summary of the Invention

[0004] This invention provides a method, system, device, and medium for information security data acquisition and preprocessing to address the problems of deep coupling, fragmented semantic mapping, and insufficient compliance support in existing open-source acquisition plugins.

[0005] In a first aspect, the present invention provides an information security data acquisition and preprocessing method, the method comprising: The independent parsing parameters are matched according to the data source configuration, and information security data is obtained based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source. By using a unified security semantic model to semantically align information security data, standard security events can be obtained. According to the current graded desensitization parameters, standard security events are graded and desensitized, and then transmitted to the user database in encrypted form based on the national cryptographic algorithm.

[0006] The information security data acquisition and preprocessing method provided by this invention decouples the data source and acquisition logic by independently parsing parameters, supports hot-swappable and rapid access, and can complete data source changes without restarting the process; it adopts a unified security semantic model to complete field alignment, unit normalization, and enumeration standardization, eliminating semantic fragmentation and improving the accuracy of security event correlation analysis; it relies on hierarchical desensitization and national cryptographic algorithms to achieve automatic protection of sensitive data, fully meeting the requirements of information security level protection and industry compliance; the overall solution is highly efficient in access, semantically unified, and natively compliant, significantly improving security operation efficiency and emergency response speed, and reducing operation and maintenance and transformation costs.

[0007] In one alternative implementation, the method further includes: Verify information security data, standard security events, graded desensitization and encrypted transmission processes, and obtain verification results; If the verification result does not meet the preset policy and compliance requirements, the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithms will be dynamically arranged and redistributed until the verification result meets the preset policy and compliance requirements.

[0008] The information security data collection and preprocessing method provided by this invention can detect process deviations and compliance risks in real time by performing policy and compliance verification on the entire process of data collection, semantic alignment, and de-identification encryption. Combined with dynamic orchestration capabilities, it automatically adjusts parsing parameters, semantic models, de-identification rules and national cryptographic algorithms to achieve closed-loop policy optimization and immediate effect. Without manual intervention and process restart, it ensures full compliance and controllability, rapid anomaly repair, significantly improves system stability, compliance rigidity and emergency self-healing capabilities, simplifies operation and maintenance and strengthens the closed-loop security operation.

[0009] In one alternative implementation, matching independent parsing parameters is configured based on the data source, including: If the data source is configured as a firewall, the independently parsed parameters include: the corresponding vendor log parsing script, the security policy field mapping table, and the regular expression dependency library; If the data source is configured for terminal detection and response, the independently parsed parameters include: terminal parsing, terminal behavior field mapping table, and PowerShell dependency library; If the data source is configured as Linux application logs, the independent parsing parameters include: general log parsing script, application log field mapping table, and Grok schema library.

[0010] The information security data collection and preprocessing method provided by this invention adopts differentiated and independent parsing parameter configurations for different types of data sources such as firewalls, terminal detection and response, and Linux application logs. It accurately matches dedicated parsing scripts, field mapping tables, and adaptation dependency libraries, thereby decoupling the collection logic from the core process. Corresponding components can be loaded on demand to avoid redundant dependencies and improve parsing efficiency and compatibility. It supports independent hot-swappable expansion of data sources, allowing for quick access without restarting, significantly reducing deployment costs and improving system adaptability and stability.

[0011] In one optional implementation, information security data is obtained based on independently parsed parameters, including: Check whether the data source device contains independent parsing parameters. If not, push the independent parsing parameters to the data source device. A hot update mechanism is used to update the data source device. Based on independent parsing parameters, protocol parsing, field extraction, and format standardization are performed to obtain the original information security data.

[0012] The information security data acquisition and preprocessing method provided by this invention automatically detects and remotely pushes independent parsing parameters, and completes data source loading by combining a hot update mechanism. It can achieve rapid distribution of parsing capabilities within minutes without restarting the core process or interrupting the existing acquisition task. It relies on independent parsing parameters to complete protocol parsing, field extraction and format standardization, ensuring stable and reliable structured data output. The overall process is lightweight and non-intrusive, effectively shortening the data source access cycle and improving system scalability and ease of operation and maintenance.

[0013] In one optional implementation, a unified security semantic model is used to semantically align information security data to obtain standard security events, including: Based on a preset master mapping table, the original field names in information security data from different data sources are mapped to field names in a unified format. Standardize the units of numerical fields in information security data from different data sources to make the units of the same physical quantity the same. Semantic standardization is performed on enumeration fields in information security data from different data sources.

[0014] The information security data collection and preprocessing method provided by this invention performs field mapping, unit normalization and enumeration standardization through a unified security semantic model, and realizes full-domain semantic alignment of multi-source heterogeneous security data. It completely solves the problems of chaotic fields, inconsistent units and inconsistent enumeration values ​​in logs from different manufacturers and devices; effectively eliminates the correlation analysis error caused by semantic fragmentation, and improves the accuracy of security event detection and judgment.

[0015] In one optional implementation, standard security events are classified and de-identified according to the current classification and de-identification parameters, and then encrypted and transmitted to the user database based on the national cryptographic algorithm, including: The sensitivity level of standard security events is labeled based on the sensitive feature library to obtain the hierarchical labels for each field; The fields of the hierarchical label calling strategy mapping table are desensitized. The anonymized data is transmitted to the user database through a transmission channel that supports the national cryptographic security socket protocol.

[0016] The information security data collection and preprocessing method provided by this invention automatically labels security events with security event classification tags through a sensitive feature library, and performs precise field-level desensitization processing according to the tags to achieve classified protection of sensitive data; it uses the national cryptographic secure socket protocol to transmit desensitized data, and constructs a full-link encryption system for collection, desensitization and transmission, automatically meeting the requirements of the Information Security Protection 2.0 and industry compliance; the entire process is automated and requires no manual configuration, ensuring data security without affecting collection efficiency and business continuity.

[0017] In one optional implementation, if the verification result does not meet the preset policy and compliance requirements, the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithms are dynamically orchestrated and reissued, including: Emergency strategies are obtained based on the verification results and approved. The emergency strategies include adjustment strategies for independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithms. The emergency strategy is pushed to the gray nodes, and the gray indexes of the gray nodes are obtained. If all gray indexes meet the standards, the emergency strategy is distributed to all nodes. If not all grayscale metrics are met, a rollback will be triggered to restore the state of the grayscale nodes.

[0018] The information security data collection and preprocessing method provided by this invention drives the generation and approval of emergency strategies through verification results, ensuring that strategy adjustments are compliant and controllable. It adopts a gray-scale release verification mechanism, first running it on a small number of nodes and monitoring indicators, and then distributing it to the whole network after the indicators are met, effectively avoiding the risk of changes. In case of anomalies, it automatically triggers rollback to quickly restore a stable state, realizing a dynamic closed loop and security self-healing of the entire process of collection, parsing, desensitization, and encryption, which greatly improves emergency response efficiency and system operation stability.

[0019] Secondly, this invention provides an information security data acquisition and preprocessing system, the system comprising: The parameter configuration module is used to match independent parsing parameters according to the data source configuration and obtain information security data based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source. The semantic alignment module is used to perform semantic alignment on information security data using a unified security semantic model to obtain standard security events. The graded desensitization and encrypted transmission module is used to perform graded desensitization on standard security events according to the current graded desensitization parameters, and encrypt the transmission to the user database based on the national cryptographic algorithm.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first type of information security data acquisition and preprocessing method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the second process of the information security data acquisition and preprocessing method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an information security data acquisition and preprocessing system according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] This invention provides an information security data acquisition and preprocessing method. It decouples the data source from the acquisition logic by independently parsing parameters, completes semantic alignment by using a unified security semantic model, and achieves automatic protection of sensitive data by relying on hierarchical desensitization and national cryptographic algorithms. This achieves the effects of hot-swappable rapid access, eliminating semantic fragmentation, and meeting industry compliance requirements.

[0028] According to an embodiment of the present invention, an embodiment of an information security data acquisition and preprocessing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides an information security data acquisition and preprocessing method, which can be used in the aforementioned computer system. Figure 1 This is a flowchart of an information security data acquisition and preprocessing method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S201: Match independent parsing parameters according to the data source configuration, and obtain information security data based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source.

[0030] Specifically, the log parsing logic of various security devices and systems can be encapsulated into independent small modules. Each module contains only the independent parsing parameters corresponding to the data source, including but not limited to: dedicated parsing scripts, field mapping tables, and minimal dependency libraries, thereby decoupling the collection logic from the platform's core processes.

[0031] The platform maintains a dynamic index table of data sources and independent small modules, recording the data source type, protocol, port, and corresponding small module file path and version identifier. When a new or changed data source is added, the corresponding independent small module is matched according to the data source configuration, and the information security data of the data source is obtained using the independent small module. The entire process does not require restarting the host collector process or interrupting the collection tasks of other data sources, realizing true plug-in-level hot-swappability and solving the pain point of traditional collectors requiring secondary development and a full restart.

[0032] Step S202: Use the unified security semantic model to perform semantic alignment on information security data to obtain standard security events.

[0033] Specifically, after the modular parsing plugin outputs the original fields of information security data (including source identifier, timestamp, original field, original value, and data type), the asynchronous consumption processing engine based on message queues (such as Kafka / RabbitMQ) pulls the data in real time and calls the Unified Security Semantic Model (USSM) to perform field alignment, unit normalization, and enumeration value translation in sequence. This ensures that the same security event is mapped to the same semantic identity number (ID) in different systems, eliminating subsequent correlation analysis errors caused by semantic mapping fragmentation.

[0034] Step S203: According to the current graded desensitization parameters, the standard security events are graded and desensitized, and then encrypted and transmitted to the user database based on the national cryptographic algorithm.

[0035] Specifically, the hierarchical desensitization parameters have standard templates. The mask is automatically identified and applied according to the templates. After being processed by the hierarchical desensitization engine, the mask is securely written to the backend storage (such as the user database) through an encrypted transmission channel, meeting the compliance requirements of transmission encryption, storage desensitization, and audit trail retention.

[0036] The information security data acquisition and preprocessing method provided in this embodiment decouples the data source and acquisition logic by independently parsing parameters, supports hot-swappable and rapid access, and can complete data source changes without restarting the process; it adopts a unified security semantic model to complete field alignment, unit normalization, and enumeration standardization, eliminating semantic fragmentation and improving the accuracy of security event correlation analysis; it relies on hierarchical desensitization and national cryptographic algorithms to achieve automatic protection of sensitive data, fully meeting the requirements of information security level protection and industry compliance; the overall solution is highly efficient in access, semantically unified, and natively compliant, significantly improving security operation efficiency and emergency response speed, and reducing operation and maintenance and transformation costs.

[0037] In some alternative implementations, such as Figure 2 As shown, the method also includes: Step S204: Verify the information security data, standard security events, hierarchical desensitization and encrypted transmission process, and obtain the verification results.

[0038] Specifically, real-time verification is performed on all aspects of information security data collection, standard security event generation, hierarchical desensitization processing, and national cryptographic encryption transmission to check whether each step complies with preset strategies and compliance requirements, and to generate verification results.

[0039] By comparing strategy template parameters in real time, the system automatically detects whether the data source access matches the configuration, whether the field parsing is complete, whether the semantic alignment meets the standards, whether the desensitization rules are executed, whether national cryptographic encryption is enabled, whether the transmission channel is legal, and whether the data is delivered according to the specified route.

[0040] In step S205, if the verification result does not meet the preset policy and compliance requirements, the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithm are dynamically arranged and redistributed until the verification result meets the preset policy and compliance requirements.

[0041] Specifically, when the validation fails, a lightweight data exchange format (such as JavaScript Object Notation (JSON) format) is provided for the collection strategy configuration template, which supports defining complete data processing pipeline parameters at once. At the same time, the strategy orchestration engine detects strategy changes in real time through a configuration center listening and message queue broadcasting mechanism.

[0042] Define a complete data processing pipeline using JSON format, specifically including: (1) Data source configuration block: data source type (FortiGate / Windows / Linux, etc.), access protocol (Syslog / JSON / API), listening port, collection frequency, filtering rules (e.g., only collect events with severity ≥ Medium).

[0043] (2) Semantic mapping configuration block: Specifies the USSM semantic library version number, field mapping rule set version, and custom field extension items.

[0044] (3) Security policy configuration block: selection of national cryptographic algorithm suite, TLS certificate ID, and desensitization policy template name.

[0045] (4) Target route configuration block: target message queue Topic, backend storage index prefix, tenant isolation identifier.

[0046] Administrators can orchestrate policies by dragging and dropping or importing JSON templates through a visual interface. After approval, the configuration is persisted to the policy database and a globally unique policy version number is generated.

[0047] When the console releases a new policy or updates an existing policy, the engine triggers a closed-loop pipeline of module packaging, gray-scale verification, full distribution, and status tracking, enabling headquarters to issue the policy with one click and the entire network to take effect within minutes.

[0048] The new strategy is sent to the acquisition end, and the system re-executes the acquisition, parsing, alignment, desensitization, and transmission, and performs verification again until it fully complies with the preset strategy and compliance requirements, forming a closed-loop control of verification-orchestration-deployment-verification-re-verification.

[0049] The information security data collection and preprocessing method provided in this embodiment can detect process deviations and compliance risks in real time by performing policy and compliance verification on the entire process of data collection, semantic alignment, and de-identification encryption. Combined with dynamic orchestration capabilities, it automatically adjusts parsing parameters, semantic models, de-identification rules and national cryptographic algorithms to achieve closed-loop policy optimization and immediate effect. Without manual intervention and process restart, it ensures full compliance and controllability, rapid anomaly repair, significantly improves system stability, compliance rigidity and emergency self-healing capabilities, simplifies operation and maintenance and strengthens the closed loop of security operation.

[0050] In some optional implementations, step S205 above includes: Step a1: Obtain emergency strategies based on the verification results and approve the emergency strategies. The emergency strategies include adjustment strategies for independent parsing parameters, unified security semantic models, hierarchical desensitization parameters, and national cryptographic algorithms.

[0051] Specifically, when the verification results do not meet the preset policies and compliance requirements, the system automatically generates an emergency policy based on the anomaly type, or the administrator configures it. The emergency policy includes adjustments to independent parsing parameters, unified security semantic models, hierarchical desensitization parameters, and national cryptographic algorithms. This policy must undergo a compliance approval process, and after approval, it is stored in the policy library and a unique version identifier is generated to ensure that policy adjustments are traceable, controllable, and compliant with security operation specifications.

[0052] Step a2: Push the emergency strategy to the gray nodes and obtain the gray indexes of the gray nodes. If all gray indexes meet the standards, then distribute the emergency strategy to all nodes.

[0053] Specifically, the approved emergency strategy will be pushed to at least a number of gray-scale nodes for trial operation. The system will collect gray-scale indicators in real time, such as module loading status, parsing success rate, de-identification compliance, encryption effectiveness, and data transmission latency. If all gray-scale indicators reach the preset threshold, the verification is deemed successful, and the emergency strategy will be fully distributed to all nodes in the entire network to achieve unified policy effectiveness.

[0054] The strategy orchestration distributes strategies to the three major processing modules: the modular acquisition framework distributes acquisition strategies (data source configuration, acquisition frequency, filtering rules, module version); the USSM semantic alignment module distributes semantic version strategies (semantic library version, field mapping rules, custom extension items); and the national cryptographic desensitization chain distributes desensitization strategies (hierarchical standard version, masking rules, encryption suite, TLS certificate ID).

[0055] Simultaneously, the system receives real-time status feedback streams from each module, including: semantic alignment status (field mapping success rate, unknown field statistics, mapping anomaly alarms); compliance status (tiered statistics (processing volume at each level from L1 to L5), de-identification processing volume, SM4 encryption throughput, audit log integrity); and operational status (module loading time, collection latency, node health). Based on the feedback data, the system automatically adjusts policy parameters to achieve closed-loop control of policy issuance, execution monitoring, and anomaly optimization.

[0056] Step a3: If not all grayscale indicators meet the standards, a rollback is triggered to restore the state of the grayscale nodes.

[0057] Specifically, if all grayscale indicators fail to meet the standards, it indicates that there is an abnormal risk in the strategy. The system will immediately trigger an automatic rollback mechanism to restore the configuration, parameters and running status of the grayscale nodes to the previous stable version, terminate the spread of abnormal strategies, avoid affecting the collection, parsing, desensitization and transmission processes, and ensure the continuous and stable operation of the system.

[0058] In one specific embodiment, a new zero-day vulnerability was discovered in a Virtual Private Network (VPN) device. The headquarters security team urgently needed to collect specific log fields (such as ssl_tunnel_id, auth_method, client_os) from all VPN gateways across the network for threat hunting, but the existing collection module did not cover these fields.

[0059] (1) Policy template configuration: Security personnel create and distribute emergency policies in the console. The policies enter the approval queue and are published to the configuration center after approval.

[0060] (2) Module packaging and dependency resolution: Query the module repository to confirm whether the corresponding resolution module exists. If it exists, pull the latest version; if it does not exist or the fields are missing, trigger the module development process and develop the field resolution module based on the template.

[0061] (3) Gray-scale release and verification: Select 2-3 representative nodes as gray-scale batches and push the strategy package and module files through an HTTPS secure channel. After the gray-scale nodes load the new module, the engine monitors indicators such as module loading success rate and data throughput latency. If all gray-scale indicators meet the standards, full distribution will be automatically initiated. If an anomaly occurs, a rollback will be triggered immediately, and the gray-scale nodes will be restored to the previous stable version, which will then be repaired manually.

[0062] (4) Full distribution and activation: After the gray-scale verification is passed, the policy orchestration broadcasts the policy change instruction to all collectors on the network through the message queue.

[0063] (5) Version Tracking and Rollback: Version changes will record the publisher, approval process, gray-scale results, and change node list for each change. If a new policy is subsequently discovered that causes an alarm storm or field parsing anomalies, the administrator can choose to roll back to any historical version in the console.

[0064] The information security data collection and preprocessing method provided in this embodiment drives the generation and approval of emergency strategies through verification results, ensuring that strategy adjustments are compliant and controllable. It adopts a gray-scale release verification mechanism, first running it on a small number of nodes and monitoring indicators, and then distributing it to the whole network after the indicators meet the standards, effectively avoiding the risk of changes. In case of anomalies, it automatically triggers rollback to quickly restore a stable state, realizing a dynamic closed loop and security self-healing of the entire process of collection, parsing, desensitization, and encryption, which greatly improves emergency response efficiency and system operation stability.

[0065] This embodiment provides an information security data acquisition and preprocessing method, which can be used in the aforementioned computer system. Figure 3 This is a flowchart of an information security data acquisition and preprocessing method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Match independent parsing parameters according to the data source configuration, and obtain information security data based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source.

[0066] Specifically, step S301 includes: Step S3011: If the data source is configured as a firewall, the independently parsed parameters include: the corresponding vendor log parsing script, the security policy field mapping table, and the regular expression dependency library.

[0067] Specifically, when the data source is configured as a firewall device, the system matches firewall-specific independent parsing parameters to specifically parse firewall traffic logs and security events accessed through the corresponding interface. These independent parsing parameters include the vendor's log parsing script, security policy field mapping tables (e.g., SRCI, DSTP, Action, Severity), and a regular expression dependency library. The log parsing script adapts to the vendor's proprietary format, the field mapping table extracts key fields such as source IP, destination IP, access action, and security level, and the regular expression dependency library enables efficient pattern matching and field extraction from the log text.

[0068] Step S3012: If the data source is configured for terminal detection and response, the independently parsed parameters include: terminal parsing, terminal behavior field mapping table, and PowerShell dependency library.

[0069] Specifically, when the data source is configured as an endpoint detection and response device, the endpoint detection and response device forwards endpoint threat detection logs via Syslog. The system matches dedicated independent parsing parameters for Endpoint Detection and Response (EDR), including an endpoint log parsing module, an endpoint behavior field mapping table (e.g., process_name, user_sid, event_id), and PowerShell dependency libraries. The endpoint parsing module processes endpoint data such as Windows event logs, the field mapping table standardizes behavior fields such as process name, user SID, and event ID, and the PowerShell dependency library is used to collect and retrieve system data under the Windows environment.

[0070] Step S3013: If the data source is configured as Linux application logs, the independent parsing parameters include: general log parsing script, application log field mapping table, and Grok pattern library.

[0071] Specifically, when the data source is configured as a Linux application log collection module, the system matches dedicated and independent parsing parameters for Linux application logs, including a general log parsing script, an application log field mapping table (e.g., app_name, timestamp, message), and a Grok pattern library. The general parsing script adapts to middleware logs from Nginx, MySQL, and other applications; the field mapping table standardizes fields such as application name, timestamp, and log information; and the Grok pattern library is used to quickly perform structured parsing of unstructured application logs.

[0072] The information security data collection and preprocessing method provided in this embodiment adopts differentiated and independent parsing parameter configurations for different types of data sources such as firewalls, terminal detection and response, and Linux application logs. It accurately matches dedicated parsing scripts, field mapping tables, and adaptation dependency libraries to decouple the collection logic from the core process. Corresponding components can be loaded on demand to avoid redundant dependencies and improve parsing efficiency and compatibility. It supports independent hot-swappable expansion of data sources, allowing for quick access without restarting, significantly reducing deployment costs and improving system adaptability and stability.

[0073] Step S3014: Detect whether the data source device contains independent parsing parameters. If not, push the independent parsing parameters to the data source device.

[0074] Specifically, the platform pre-maintains a dynamic index table of data sources and modules to record the data source type, access protocol, listening port, and the file path and version identifier of the corresponding parsing module. After the data source configuration and registration are completed, the system first checks whether the data collection agent already has matching independent parsing parameters locally; if not, it pushes the parsing module file to the specified module directory of the target agent via HTTP / HTTPS or an internal message channel to achieve remote completion of parsing capabilities.

[0075] Step S3015: The data source device is updated using a hot update mechanism. Based on the independent parsing parameters, protocol parsing, field extraction, and format standardization are performed to obtain the original information security data.

[0076] Specifically, after the push is completed, the system uses a hot update mechanism to load the independently parsed parameters. The Agent core process does not need to be restarted or the existing collection task is interrupted. Subsequently, the collector routes raw traffic such as Syslog, JSON, SNMP Trap, and API to the corresponding parsing modules. The modules perform protocol parsing, key field extraction, and format standardization processing, and output structured raw information security data containing tenant identifier, data source type, raw fields, timestamp, and data type for use by the downstream semantic alignment module.

[0077] In one specific embodiment, when a customer adds or changes a data source at their site, the complete process includes: (1) Data source configuration registration: Administrators can enter or import data source configurations (device type, IP address, access protocol, port, log format sample) through the console. The console queries the data source-module index table based on protocol characteristics and device type, and intelligently matches the corresponding parsing module.

[0078] (2) Module delivery per minute: If the target data collection agent node does not have the local file of the module, the console will push the module file (usually less than 100KB) to the module directory of the specified agent via the file transfer interface (HTTP / HTTPS or internal message channel). The module adopts a hot update mechanism, and the agent core process does not need to be restarted.

[0079] (3) Traffic routing and parsing: The collector routes the raw traffic (Syslog / JSON / SNMP Trap / API, etc.) of the corresponding data source to the input interface of the module according to the configuration. The module performs protocol parsing, field extraction, and format standardization, and outputs structured raw data packets (including tenant_id, source_type, raw_fields, timestamp, data_type), which are asynchronously transmitted to the downstream unified security semantic model alignment module through a message queue.

[0080] The information security data acquisition and preprocessing method provided in this embodiment automatically detects and remotely pushes independent parsing parameters, and completes data source loading by combining a hot update mechanism. It can achieve rapid distribution of parsing capabilities within minutes without restarting the core process or interrupting the existing acquisition task. It relies on independent parsing parameters to complete protocol parsing, field extraction and format standardization, ensuring stable and reliable structured data output. The overall process is lightweight and non-intrusive, effectively shortening the data source access cycle and improving system scalability and ease of operation and maintenance.

[0081] Step S302: Use the unified security semantic model to perform semantic alignment on the information security data to obtain standard security events.

[0082] Specifically, step S302 includes: Step S3021: Based on the preset master mapping table, map the original field names in the information security data from different data sources to field names in a unified format.

[0083] Specifically, by calling the USSM master mapping table, the original field names of different vendors and data sources are mapped to unified standard fields. For example, FortiGate's srcip, Windows terminal's SourceIP, and Linux application's client_ip, regardless of their original naming differences, are all aligned to source.ip; dstport, DestinationPort, and server_port are uniformly aligned to destination.port. This is just an example and is not a limitation.

[0084] Step S3022: Standardize the units of numerical fields in information security data from different data sources to make the units of the same physical quantity the same.

[0085] Specifically, unit standardization is performed on numerical fields. For example, traffic volume may be reported in KB, MB, or Bytes, and it is uniformly normalized to Bytes; timestamps are uniformly converted to UTC standard time (ISO 8601 format); durations are uniformly converted to millisecond-level integers to eliminate statistical biases and correlation failures caused by unit differences. This is just an example, but not a limitation.

[0086] Step S3023: Perform semantic standardization on enumeration fields in information security data from different data sources.

[0087] Specifically, semantic standardization is applied to enumeration fields such as status, action, and severity level. For example, FortiGate's action=accept / deny, Palo Alto's act=allow / drop, and Windows Firewall's Action=Allow / Block are uniformly translated into the standard enumeration value event_action: allow / block; vendor-defined severity levels (such as severe / high risk) are mapped to standardized levels of event_severity: 1-5, which is only an example and not a limitation.

[0088] Ultimately, a standardized security event, USSM-Event, is generated.

[0089] The information security data collection and preprocessing method provided in this embodiment achieves full-domain semantic alignment of multi-source heterogeneous security data by carrying out field mapping, unit normalization and enumeration standardization through a unified security semantic model. This completely solves the problems of chaotic fields, inconsistent units and inconsistent enumeration values ​​in logs from different manufacturers and devices. It effectively eliminates the correlation analysis errors caused by semantic fragmentation and improves the accuracy of security event detection and judgment.

[0090] Step S303: According to the current graded desensitization parameters, the standard security events are graded and desensitized, and then encrypted and transmitted to the user database based on the national cryptographic algorithm.

[0091] Specifically, step S303 includes: Step S3031: Based on the sensitive feature library, the sensitivity level of standard security events is labeled to obtain the hierarchical labels of each field.

[0092] Specifically, after receiving standardized USSM events, the graded data masking engine needs to be security-hardened. The first step is dynamic data graded identification. The engine has a built-in sensitive feature library (including regular expression patterns for keys, internal IP addresses, etc.) to jointly identify field names (e.g., password, secret_key, user_name) and content context, automatically labeling them with sensitivity levels. The meanings of each sensitivity level are as follows: L1 (Public Level): Network traffic data, non-sensitive system logs.

[0093] L2 (Internal Level): General network configuration.

[0094] L3 (Sensitive Level): Tenant device name, IP address, trigger part mask (e.g., 10.180.***.***).

[0095] L4 (Confidential): Core asset IP addresses and other information, VPN accounts, triggering full-field conformal desensitization (format-preserving encryption FPE) or SM4 reversible encryption.

[0096] L5 (Top Secret): Root key, national cryptographic certificate private key, triggering truncation or tokenization replacement.

[0097] Step S3032: Desensitize each field by calling the strategy mapping table based on the hierarchical label.

[0098] Specifically, field-level processing is performed based on the hierarchical label invocation strategy mapping table, including: shape-preserving desensitization, reversible desensitization, and irreversible processing. Shape-preserving desensitization refers to maintaining the data type and format of fields such as dates (e.g., 2024-01-01 → ****-**-01); reversible desensitization refers to using SM4 encryption on L4-level fields and associating them with a unique key ID to meet the decryption and recovery requirements during audit traceability; irreversible processing refers to truncating or tokenizing L5-level sensitive fields, with the original data only retained in a secure isolation area, and only an irreversible identifier remaining in the final storage.

[0099] Step S3033: Transmit the de-identified data to the user database through a transmission channel that supports the national cryptographic security socket protocol.

[0100] Specifically, the anonymized data packets are sent to the backend storage cluster via a transmission channel supporting the Secure Socket Layer (SSL) protocol (TLCP / GMSSL), employing the ECC-SM2-WITH-SM4-SM3 cipher suite and enabling two-way certificate authentication (SM2 client certificate on the collector side + server certificate verification) to ensure the security of the transmission link. Finally, the data is written to the OpenSearch index and relational database in tenant isolation, achieving end-to-end protection of anonymization during collection, encryption during transmission, and isolation during storage.

[0101] The platform provides a visual rule orchestration interface through a console, allowing administrators to edit the sensitive feature library online (using regular expressions for adding, deleting, and modifying), adjust field hierarchical mapping relationships, or modify desensitization strategy parameters. After configuration changes are approved, they are pushed to all data collection nodes in real time by the configuration center, completing hot-loading of rules within 30 seconds without restarting processes or interrupting the data collection pipeline. For example, when a customer adds a business system involving a customer asset account field, the administrator can add that field name to the L4-level sensitive word library and bind a desensitization strategy in the console. After the rule is issued, this field automatically triggers encryption processing in all tenants' new logs, and historical data is not included in the rollback, ensuring the immediate effectiveness of business continuity and compliance policies.

[0102] The information security data collection and preprocessing method provided in this embodiment automatically labels security events with security event classification tags through a sensitive feature library, and performs precise field-level desensitization processing according to the tags to achieve classified protection of sensitive data; it uses the national cryptographic secure socket protocol to transmit desensitized data, and constructs a full-link encryption system for collection, desensitization and transmission, automatically meeting the requirements of the Information Security Protection 2.0 and industry compliance; the entire process is automated and requires no manual configuration, which ensures data security without affecting collection efficiency and business continuity.

[0103] This embodiment also provides an information security data acquisition and preprocessing system, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] This embodiment provides an information security data acquisition and preprocessing system, such as... Figure 4 As shown, it includes: The parameter configuration module 401 is used to match independent parsing parameters according to the data source configuration and obtain information security data based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source.

[0105] The semantic alignment module 402 is used to perform semantic alignment on information security data using a unified security semantic model to obtain standard security events.

[0106] The graded desensitization and encrypted transmission module 403 is used to perform graded desensitization on standard security events according to the current graded desensitization parameters, and encrypt the transmission to the user database based on the national cryptographic algorithm.

[0107] In some optional implementations, the parameter configuration module 401 includes: The first configuration unit is used to independently parse parameters, including: the corresponding vendor log parsing script, security policy field mapping table, and regular expression dependency library, if the data source is configured as a firewall.

[0108] The second configuration unit is used to independently parse parameters including: terminal parsing, terminal behavior field mapping table, and PowerShell dependency libraries if the data source is configured for terminal detection and response.

[0109] The third configuration unit is used to independently parse parameters, including: general log parsing script, application log field mapping table, and Grok pattern library, if the data source is configured as Linux application logs.

[0110] The parameter detection unit is used to detect whether the data source device contains independent parsing parameters. If not, the independent parsing parameters are pushed to the data source device.

[0111] The parameter update unit is used to update the data source device using a hot update mechanism. It performs protocol parsing, field extraction, and format standardization based on independent parsing parameters to obtain the original information security data.

[0112] In some alternative implementations, the semantic alignment module 402 includes: The field format unification unit is used to map the original field names in information security data from different data sources to field names in a unified format, based on a preset master mapping table.

[0113] Unit unification is used to standardize the units of numerical fields in information security data from different data sources, so that the units of the same physical quantity are the same.

[0114] Semantic standardization unit is used to perform semantic standardization on enumeration class fields in information security data from different data sources.

[0115] In some optional implementations, the hierarchical desensitization and encrypted transmission module 403 includes: The hierarchical unit is used to label the sensitivity level of standard security events based on the sensitive feature library, and obtain the hierarchical label for each field.

[0116] The desensitization processing unit is used to desensitize each field based on the hierarchical label calling strategy mapping table.

[0117] The encrypted transmission unit is used to transmit the de-identified data to the user database through a transmission channel that supports the national cryptographic security socket protocol.

[0118] The information security data acquisition and preprocessing system provided in this embodiment of the invention can execute the information security data acquisition and preprocessing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments, and will not be repeated here.

[0119] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0120] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0121] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0122] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the information security data acquisition and preprocessing method of the embodiments of the present invention.

[0123] Figure 5The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0124] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the information security data acquisition and preprocessing methods shown in the above embodiments are implemented.

[0125] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for information security data acquisition and preprocessing, characterized in that, The method includes: The independent parsing parameters are matched according to the data source configuration, and information security data is obtained based on the independent parsing parameters, wherein the independent parsing parameters include the log parsing logic configured for the corresponding data source; The information security data is semantically aligned using a unified security semantic model to obtain standard security events. According to the current graded desensitization parameters, the standard security events are graded and desensitized, and then encrypted and transmitted to the user database based on the national cryptographic algorithm.

2. The method according to claim 1, characterized in that, The method further includes: The information security data, standard security events, hierarchical desensitization, and encrypted transmission process are verified to obtain the verification results. If the verification result does not meet the preset policy and compliance requirements, the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithm will be dynamically rearranged and reissued until the verification result meets the preset policy and compliance requirements.

3. The method according to claim 1, characterized in that, Match independent parsing parameters based on data source configuration, including: If the data source is configured as a firewall, the independently parsed parameters include: the corresponding vendor log parsing script, the security policy field mapping table, and the regular expression dependency library; If the data source is configured for terminal detection and response, the independently parsed parameters include: terminal parsing, terminal behavior field mapping table, and PowerShell dependency library; If the data source is configured as Linux application logs, the independent parsing parameters include: general log parsing script, application log field mapping table, and Grok schema library.

4. The method according to claim 1 or 3, characterized in that, Information security data is obtained based on the independent parsing parameters, including: The system detects whether the data source device contains independent parsing parameters. If not, it pushes the independent parsing parameters to the data source device. The data source device is updated using a hot update mechanism. Based on the independent parsing parameters, protocol parsing, field extraction, and format standardization are performed to obtain the original information security data.

5. The method according to claim 1, characterized in that, The information security data is semantically aligned using a unified security semantic model to obtain standard security events, including: Based on a preset master mapping table, the original field names in information security data from different data sources are mapped to field names in a unified format. Standardize the units of numerical fields in information security data from different data sources to make the units of the same physical quantity the same. Semantic standardization is performed on enumeration fields in information security data from different data sources.

6. The method according to claim 1, characterized in that, According to the current graded desensitization parameters, the standard security events are graded and desensitized, and then encrypted and transmitted to the user database based on the national cryptographic algorithm, including: The sensitivity level of the standard security events is labeled based on the sensitive feature library to obtain the hierarchical labels for each field; Based on the hierarchical label calling strategy mapping table, each field is desensitized; The anonymized data is transmitted to the user database through a transmission channel that supports the national cryptographic security socket protocol.

7. The method according to claim 2, characterized in that, If the verification result does not meet the preset strategy and compliance requirements, the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithms will be dynamically rearranged and reissued, including: Emergency strategies are obtained based on the verification results and the emergency strategies are approved. The emergency strategies include adjustment strategies for the independent parsing parameters, unified security semantic model, hierarchical desensitization parameters, and national cryptographic algorithms. The emergency strategy is pushed to the gray nodes, and the gray index of the gray nodes is obtained. If all the gray indexes meet the standards, the emergency strategy is distributed to all nodes. If not all of the grayscale metrics are met, a rollback is triggered to restore the state of the grayscale node.

8. An information security data acquisition and preprocessing system, characterized in that, The system includes: The parameter configuration module is used to match independent parsing parameters according to the data source configuration, and to obtain information security data based on the independent parsing parameters. The independent parsing parameters include the log parsing logic configured for the corresponding data source. The semantic alignment module is used to perform semantic alignment on the information security data using a unified security semantic model to obtain standard security events. The graded desensitization and encrypted transmission module is used to perform graded desensitization on the standard security events according to the current graded desensitization parameters, and encrypt the transmission to the user database based on the national cryptographic algorithm.

9. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images