Security log data acquisition method and system and electronic equipment

Abstract

The invention provides a security log data collection method and system, and relates to the technical field of data processing, and the method comprises the steps: obtaining a starting instruction and a target configuration file; selecting a transmission channel based on the target configuration file to obtain a target transmission channel; generating a unique task identifier based on the starting instruction, and sending an associated log file to a target file of a target node; and starting an agent model based on the starting instruction, completing security log data collection through the target transmission channel, synchronously starting a monitoring thread, registering the unique task identifier and the associated log file, and collecting key indexes of the agent model in real time. According to the method, multi-source data access is supported, manual intervention is reduced, deployment and maintenance work of the platform is simplified, and log collection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method, a system and an electronic device for collecting security log data. Background Art

[0002] With the rapid development of cloud computing, the Internet of Things and edge computing, the types of log data generated in enterprise IT environments have become increasingly complex, covering heterogeneous data sources such as server status records, network traffic packets, audio and video streams, and IoT device telemetry data. When facing diverse data sources, traditional log collection systems (such as Flume and Logstash) generally have the following technical bottlenecks:

[0003] Existing solutions mostly rely on static configurations and fixed adapters, making it difficult to dynamically expand new data sources (such as cloud-native event streams). For example, collecting unstructured audio and video logs requires custom development of dedicated plugins, resulting in high access costs and poor flexibility. Traditional systems usually adopt a single transmission channel (such as pure memory or pure disk) and cannot dynamically switch according to scenario requirements. For example, in a scenario of sudden traffic, the pure memory channel is prone to data loss due to buffer overflow; while the pure disk channel cannot meet real-time requirements due to I / O latency. Existing technologies mostly adopt a fixed threshold alarm strategy and lack an anti-fatigue mechanism, resulting in repeated alarms interfering with operation and maintenance decisions. At the same time, resource residues (such as unreleased memory handles and zombie processes) often exist after the task stops, which may pose risks to system stability.

[0004] Therefore, a method, a system and an electronic device for collecting security log data are proposed. Summary of the Invention

[0005] This specification provides a method, a system and an electronic device for collecting security log data, which support multi-source data access, reduce manual intervention, simplify the deployment and maintenance work of the platform, and improve the log collection efficiency.

[0006] This specification provides a method for collecting security log data, including:

[0007] Obtaining a start instruction and a target configuration file;

[0008] Selecting a transmission channel based on the target configuration file to obtain a target transmission channel;

[0009] Generating a unique task identifier based on the start instruction and sending the associated log file to a target file of a target node;

[0010] Starting a proxy model based on the start instruction, completing the collection of security log data through the target transmission channel, synchronously starting a monitoring thread, registering the unique task identifier and its associated log file, and collecting key metrics of the proxy model in real time.

[0011] Optionally, the obtaining of the target configuration file includes:

[0012] Obtain a declarative configuration file;

[0013] Parse the declarative configuration file to obtain a data source type identifier;

[0014] Dynamically load a corresponding pluggable adapter based on the data source type identifier, complete the initialization connection of heterogeneous data sources, and obtain a target configuration file.

[0015] Optionally, the selecting of a transmission channel based on the target configuration file to obtain a target transmission channel includes:

[0016] The target configuration file includes configuration requirements; among them, the configuration requirements include high-performance requirements and persistence guarantee requirements;

[0017] When the configuration requirement is the high-performance requirement, select the Memory Channel to temporarily store data through the memory buffer;

[0018] When the configuration requirement is the persistence guarantee requirement, select the File Channel to record transaction logs through the disk.

[0019] Optionally, the synchronously starting a monitoring thread, registering the unique task identifier and its associated log file, and real-time collecting key metrics of the proxy model includes:

[0020] Continuously parse the log file associated with the unique task identifier and determine whether there is a predefined error;

[0021] When there is a predefined error, determine whether the last warning time is less than a preset time;

[0022] When the last warning time is greater than the preset time, generate a diagnostic log and push it to the operation and maintenance end through an encrypted link.

[0023] Optionally, it further includes:

[0024] When the last warning time is less than the preset time, ignore the current error.

[0025] Optionally, it further includes:

[0026] Obtain a stop instruction;

[0027] Control the proxy model to stop collecting security log data based on the stop instruction, close the monitoring thread, release the memory buffer or the persistence culture handle, unregister the unique task identifier and its associated log file, and complete resource recovery.

[0028] Optionally, the parsing of the declarative configuration file includes:

[0029] When the declarative configuration file is structured data, generate a standardized JSON format through Schema mapping;

[0030] When the declarative configuration file is unstructured data, extract key fields through regular expressions and fill them into a predefined template;

[0031] When the declarative configuration file is audio data, intercept metadata through the FFmpeg plugin and generate a summary log.

[0032] This specification provides a security log data acquisition system, including:

[0033] An acquisition module, used to obtain a startup instruction and a target configuration file;

[0034] A selection module, used to select a transmission channel based on the target configuration file to obtain a target transmission channel;

[0035] A creation module, used to generate a unique task identifier based on the startup instruction and send the associated log file to a target file of a target node;

[0036] An acquisition module, used to start an agent model based on the startup instruction, complete the acquisition of security log data through the target transmission channel, and synchronously start a monitoring thread, register the unique task identifier and its associated log file, and collect key metrics of the agent model in real time.

[0037] Optionally, the acquisition module includes:

[0038] Obtain a declarative configuration file;

[0039] Parse the declarative configuration file to obtain a data source type identifier;

[0040] Dynamically load a corresponding pluggable adapter based on the data source type identifier, complete the initialization connection of heterogeneous data sources, and obtain a target configuration file.

[0041] Optionally, the selection module includes:

[0042] The target configuration file includes configuration requirements; among them, the configuration requirements include high-performance requirements and persistence guarantee requirements;

[0043] When the configuration requirement is the high-performance requirement, select the Memory Channel to implement data caching through the memory buffer;

[0044] When the configuration requirement is a persistence guarantee requirement, select the File Channel and record the transaction log through the disk.

[0045] Optionally, the acquisition module includes:

[0046] Continuously parse the log file associated with the unique task identifier and determine whether there is a predefined error;

[0047] When there is a predefined error, determine whether the most recent alarm time is less than the preset time;

[0048] When the most recent alarm time is greater than the preset time, generate a diagnostic log and push it to the operation and maintenance side through an encrypted link.

[0049] Optionally, it further includes:

[0050] When the most recent alarm time is less than the preset time, ignore the current error.

[0051] Optionally, it further includes:

[0052] Obtain a stop instruction;

[0053] Based on the stop instruction, control the proxy model to stop collecting security log data, close the monitoring thread, release the memory buffer or the persistent culture handle, and unregister the unique task identifier and its associated log file to complete resource recovery.

[0054] Optionally, the parsing of the declarative configuration file includes:

[0055] When the declarative configuration file is structured data, generate a standardized JSON format through Schema mapping;

[0056] When the declarative configuration file is unstructured data, extract keyword fields through regular expressions and fill them into a predefined template;

[0057] When the declarative configuration file is audio data, intercept metadata through the FFmpeg plugin and generate a summary log.

[0058] This specification also provides an electronic device, where the electronic device includes:

[0059] A processor; and,

[0060] A memory storing computer-executable instructions, and the executable instructions, when executed, cause the processor to execute any one of the above methods.

[0061] The present specification also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the above-mentioned method is implemented.

[0062] In the present invention, multi-source data access is supported, manual intervention is reduced, the deployment and maintenance work of the platform is simplified, and the log collection efficiency is improved. The entire log processing process is monitored in real time, and detailed monitoring metrics are provided, enabling operation and maintenance personnel to comprehensively understand the operating status of the platform, promptly discover potential problems, and enhance the comprehensiveness of monitoring. The "alarm fatigue" caused by frequent repeated alarms is avoided, while ensuring the timely transmission of important information, improving the response efficiency and accuracy. The user interface is friendly and the operation is simple, reducing the usage threshold of users and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0064] Figure 1 It is a schematic diagram of the principle of a method for collecting security log data provided by an embodiment of the present specification;

[0065] Figure 2 It is a schematic diagram of the structure of a system for collecting security log data provided by an embodiment of the present specification;

[0066] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present specification;

[0067] Figure 4 It is a schematic diagram of the principle of a computer-readable medium provided by an embodiment of the present specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description can be applied to other implementation schemes, variant schemes, improvement schemes, equivalent schemes, and other technical schemes that do not deviate from the spirit and scope of the present invention.

[0069] The following is combined with the attached Figures 1-4The exemplary embodiments of the present invention are described more fully. However, the exemplary embodiments can be implemented in various forms, and it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments can make the present invention more comprehensive and complete, and it is more convenient to fully convey the inventive concept to those skilled in the art. The same reference numerals in the figures represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0070] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0071] In the description of specific embodiments, the features, structures, characteristics or other details described in the present invention are intended to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can practice the technical solutions of the present invention without one or more of the specific features, structures, characteristics or other details.

[0072] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0073] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0074] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0075] Figure 1 A schematic diagram of the principle of a security log data collection method provided in an embodiment of this specification, the method may include:

[0076] S110: Obtaining a startup instruction and a target configuration file;

[0077] Optionally, the S110 includes:

[0078] Get the declarative configuration file;

[0079] Parsing the declarative configuration file to obtain a data source type identifier;

[0080] Dynamically load the corresponding pluggable adapter based on the data source type identifier, complete the initialization connection of heterogeneous data sources, and obtain the target configuration file.

[0081] In the specific implementation of this specification, read the declarative configuration file provided by the user, and parse the data source type identifier, parsing rules, and transmission protocol fields therein. According to the data source type identifier, dynamically load the corresponding pluggable adapter (such as Avro message queue adapter, incremental file listening adapter), and dynamically expand the acquisition ability for structured or unstructured data sources. It should be noted that the data sources collected include but are not limited to IoT device telemetry data and cloud native event streams.

[0082] S120: Select a transmission channel based on the target configuration file to obtain the target transmission channel;

[0083] Optionally, the S120 includes:

[0084] The target configuration file includes configuration requirements; among them, the configuration requirements include high-performance requirements and persistence guarantee requirements;

[0085] When the configuration requirement is the high-performance requirement, select the Memory Channel to implement data caching through the memory buffer;

[0086] When the configuration requirement is the persistence guarantee requirement, select the File Channel to record transaction logs through the disk.

[0087] In the specific implementation of this specification, before writing data to the channel, start the transaction lock and perform integrity verification, and commit the transaction and release resources after the transmission is completed. Using the transaction mechanism of the Channel during the acquisition and transmission process ensures the reliable transmission of log data from the source to the destination.

[0088] In the specific implementation of this specification, introduce a multi-level caching strategy in the Memory Channel: the first-level cache is a memory circular queue for quickly receiving burst data; the second-level cache is an SSD high-speed cache, which overflows and writes to the SSD when the memory queue is full; automatically clean cold data based on the LRU (Least Recently Used) algorithm, and retain hot data in memory.

[0089] S130: Generate a unique task identifier based on the start instruction, and send the associated log file to the target file of the target node;

[0090] In the specific embodiments of this specification, in response to an external one-key startup instruction, a unique task identifier logID is generated, and the associated log file is sent to the target file of the target node. For example: a log file named logId + logFileSuffix.

[0091] S140: Start the proxy model based on the startup instruction, complete the collection of security log data through the target transmission channel, and synchronously start the monitoring thread, register the unique task identifier and its associated log file, and collect the key metrics of the proxy model in real time.

[0092] In the specific embodiments of this specification, the nohup tool is used to run the Agent command in the background to ensure that the collection task can continue to run even if the user session ends. Once the collection task is successfully started, the monitoring task is immediately triggered, and a new monitoring target is registered with the monitoring component to collect and display relevant performance data in real time. The key metrics include the CPU occupancy rate, memory consumption, and data throughput metrics of the Agent.

[0093] In the specific embodiments of this specification, according to the real-time load status of the data source, the collection frequency and the number of concurrent threads are dynamically adjusted. Specifically, the input traffic rate of the data source and the CPU occupancy rate of the system are monitored; if the CPU occupancy rate exceeds the preset value, the collection frequency is decreased in a gradient manner, and the number of active threads in the thread pool is reduced; when the traffic rate is lower than the preset value and the CPU occupancy rate returns to normal, the collection frequency is gradually increased to the initial value.

[0094] Optionally, S140 includes:

[0095] Continuously parse the log file associated with the unique task identifier, and determine whether there is a predefined error;

[0096] When there is a predefined error, determine whether the last alarm time is less than the preset time;

[0097] When the last alarm time is greater than the preset time, generate a diagnostic log and push it to the operation and maintenance end through an encrypted link.

[0098] Optionally, it further includes:

[0099] When the last alarm time is less than the preset time, ignore the current error.

[0100] In the specific embodiments of this specification, a real-time monitoring status stream is constructed to realize the real-time tracking of the operating parameters of the data input end and the output end, and a real-time exception alarm system for data collection is established. The health of the data channel is reflected in real time through multi-dimensional statuses. When the collection success rate is lower than the threshold, the alarm policy is automatically triggered, and at the same time, a diagnostic log report including the error code, timestamp, and impact range is generated.

[0101] Check the time of the last alarm. If the interval is less than 30 seconds, ignore the current alarm to avoid alarm fatigue. If it exceeds 30 seconds, generate a diagnostic log containing the error code, timestamp, and node IP, and push it to the operation and maintenance terminal through an encrypted link. Update the alarm status database and mark the current alarm as "processed".

[0102] Optionally, it further includes:

[0103] Obtain a stop instruction;

[0104] Based on the stop instruction, control the proxy model to stop collecting security log data, close the monitoring thread, release the memory buffer or persist the culture handle, unregister the unique task identifier and its associated log file, and complete resource recovery.

[0105] In the specific implementation of this specification, if the task is abnormally terminated due to node downtime, after the node recovers, it will automatically re-bind the original logID and resume transmitting the unfinished data. By comparing the offsets of the transaction logs, locate the breakpoint and skip the data blocks that have been successfully transmitted. Send a recovery completion notice and a breakpoint resume transmission statistical report to the operation and maintenance terminal.

[0106] Optionally, the parsing of the declarative configuration file includes:

[0107] When the declarative configuration file is structured data, generate a standardized JSON format through Schema mapping;

[0108] When the declarative configuration file is unstructured data, extract the key fields through regular expressions and fill them into a predefined template;

[0109] When the declarative configuration file is audio data, intercept the metadata through the FFmpeg plugin and generate a summary log.

[0110] In the specific implementation of this specification, the acquisition interface includes a programmable interface, allowing third-party developers to add new data source configurations.

[0111] In the present invention, it supports multi-source data access, reduces manual intervention, simplifies the platform deployment and maintenance work, and improves the log acquisition efficiency. It monitors the entire log processing process in real time and provides detailed monitoring metrics, enabling operation and maintenance personnel to comprehensively understand the platform's operating status, promptly discover potential problems, and enhance the comprehensiveness of monitoring. It avoids "alarm fatigue" caused by frequent repeated alarms, while ensuring the timely transmission of important information, improving the response efficiency and accuracy. The user interface is friendly, the operation is simple, reducing the user's usage threshold and enhancing the user experience.

[0112] Figure 2Schematic diagram of the principle of a security log data acquisition system provided by an embodiment of this specification. The system may include:

[0113] An acquisition module 10, configured to acquire a start instruction and a target configuration file;

[0114] A selection module 20, configured to select a transmission channel based on the target configuration file to obtain a target transmission channel;

[0115] A creation module 30, configured to generate a unique task identifier based on the start instruction and send an associated log file to a target file of a target node;

[0116] An acquisition module 40, configured to start an agent model based on the start instruction, complete security log data acquisition through the target transmission channel, and synchronously start a monitoring thread, register the unique task identifier and its associated log file, and collect key metrics of the agent model in real time.

[0117] Optionally, the acquisition module 10 includes:

[0118] Acquire a declarative configuration file;

[0119] Parse the declarative configuration file to obtain a data source type identifier;

[0120] Dynamically load a corresponding pluggable adapter based on the data source type identifier, complete the initialization connection of heterogeneous data sources, and obtain a target configuration file.

[0121] Optionally, the selection module 20 includes:

[0122] The target configuration file includes configuration requirements; among them, the configuration requirements include high-performance requirements and persistence guarantee requirements;

[0123] When the configuration requirement is the high-performance requirement, select Memory Channel to implement data caching through a memory buffer;

[0124] When the configuration requirement is the persistence guarantee requirement, select File Channel to record transaction logs through a disk.

[0125] Optionally, the acquisition module 40 includes:

[0126] Continuously parse the log file associated with the unique task identifier and determine whether there is a predefined error;

[0127] When there is a predefined error, determine whether the most recent alarm time is less than a preset time;

[0128] When the last alarm time is greater than the preset time, generate a diagnostic log and push it to the operation and maintenance side through an encrypted link.

[0129] Optionally, it further includes:

[0130] When the last alarm time is less than the preset time, ignore the current error.

[0131] Optionally, it further includes:

[0132] Obtain a stop instruction;

[0133] Based on the stop instruction, control the proxy model to stop collecting security log data, close the monitoring thread, release the memory buffer or persist the culture handle, unregister the unique task identifier and its associated log file, and complete resource recycling.

[0134] Optionally, the parsing of the declarative configuration file includes:

[0135] When the declarative configuration file is structured data, generate a standardized JSON format through Schema mapping;

[0136] When the declarative configuration file is unstructured data, extract keyword fields through regular expressions and fill them into a predefined template;

[0137] When the declarative configuration file is audio data, intercept metadata through the FFmpeg plugin and generate a summary log.

[0138] Based on the same inventive concept, an embodiment of this specification further provides an electronic device.

[0139] The following describes an embodiment of the electronic device of the present invention. This electronic device can be regarded as a specific physical implementation manner of the above method and device embodiments of the present invention. For the details described in the embodiment of the electronic device of the present invention, they should be regarded as a supplement to the above method or device embodiments; for the details not disclosed in the embodiment of the electronic device of the present invention, they can be implemented with reference to the above method or device embodiments.

[0140] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of this specification. The following refers to Figure 3 to describe the electronic device 300 according to this embodiment of the present invention. Figure 3 The electronic device 300 shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0141] As Figure 3As shown, the electronic device 300 is presented in the form of a general-purpose computing device. The components of the electronic device 300 may include, but are not limited to: at least one processing unit 310, at least one storage unit 320, a bus 330 connecting different system components (including the storage unit 320 and the processing unit 310), a display unit 340, etc.

[0142] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 310, so that the processing unit 310 executes the steps according to various exemplary embodiments of the present invention described in the above processing method part of this specification. For example, the processing unit 310 can execute as Figure 1 shown in the steps.

[0143] The storage unit 320 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 3201 and / or a cache storage unit 3202, and may further include a read-only storage unit (ROM) 3203.

[0144] The storage unit 320 may also include a program / utilities 3204 having a set (at least one) of program modules 3205. Such program modules 3205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0145] The bus 330 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0146] The electronic device 300 can also communicate with one or more external devices 400 (such as a keyboard, a pointing device, a Bluetooth device, etc.), can also communicate with one or more devices that enable the audience to interact with the electronic device 300, and / or communicate with any device that enables the electronic device 300 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 350. And, the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 360. The network adapter 360 can communicate with other modules of the electronic device 300 through the bus 330. It should be understood that although Figure 3is not shown in the figure, and other hardware and / or software modules can be used in combination with the electronic device 300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0147] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described in the present invention can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on the network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium can implement the above method of the present invention, that is: as Figure 1 the method shown.

[0148] Figure 4 is a schematic diagram of the principle of a computer-readable medium provided by an embodiment of this specification.

[0149] Implement Figure 1 The computer program for implementing the method shown can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0151] Program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the viewer computing device, partially on the viewer device, executed as a stand-alone software package, partially on the viewer computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the viewer computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0152] In summary, the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0153] The specific embodiments described above have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0154] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key point of each embodiment is to illustrate the differences from other embodiments.

[0155] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for collecting security log data, characterized in that, Include: Obtain a start instruction and a target configuration file; Select a transmission channel based on the target configuration file to obtain a target transmission channel; Generate a unique task identifier based on the start instruction and send the associated log file to the target file of the target node; Start the proxy model based on the start instruction, complete the collection of security log data through the target transmission channel, synchronously start a monitoring thread, register the unique task identifier and its associated log file, and collect key metrics of the proxy model in real time.

2. The security log data collection method according to claim 1, wherein The obtaining of the target configuration file includes: Obtain a declarative configuration file; Parse the declarative configuration file to obtain a data source type identifier; Dynamically load the corresponding pluggable adapter based on the data source type identifier, complete the initialization connection of heterogeneous data sources, and obtain a target configuration file.

3. The security log data collection method according to claim 2, characterized in that The selecting of the transmission channel based on the target configuration file to obtain a target transmission channel includes: The target configuration file includes configuration requirements; among them, the configuration requirements include high-performance requirements and persistence guarantee requirements; When the configuration requirement is the high-performance requirement, select Memory Channel to implement data caching through the memory buffer; When the configuration requirement is the persistence guarantee requirement, select File Channel to record transaction logs through the disk.

4. The security log data collection method according to claim 3, wherein, The synchronously starting of the monitoring thread, registering the unique task identifier and its associated log file, and collecting key metrics of the proxy model in real time includes: Continuously parse the log file associated with the unique task identifier and determine whether there are predefined errors; When there are predefined errors, determine whether the last alarm time is less than a preset time; When the last alarm time is greater than the preset time, generate a diagnostic log and push it to the operation and maintenance end through an encrypted link.

5. The security log data collection method according to claim 4, wherein Also include: When the last alarm time is less than the preset time, ignore the current error.

6. The security log data collection method according to claim 5, characterized in that Also include: Obtain a stop instruction; Control the proxy model to stop collecting security log data based on the stop instruction, close the monitoring thread, release the memory buffer or the persistent culture handle, unregister the unique task identifier and its associated log file, and complete resource recovery.

7. The security log data collection method according to claim 5, characterized in that, The parsing of the declarative configuration file includes: When the declarative configuration file is structured data, generate a standardized JSON format through Schema mapping; When the declarative configuration file is unstructured data, extract key fields through regular expressions and fill them into a predefined template; When the declarative configuration file is audio data, intercept metadata through the FFmpeg plugin and generate a summary log.

8. A security log data collection system, characterized in that, Include: An obtaining module for obtaining a start instruction and a target configuration file; A selecting module for selecting a transmission channel based on the target configuration file to obtain a target transmission channel; A creating module for generating a unique task identifier based on the start instruction and sending the associated log file to the target file of the target node; The acquisition module is used to start the proxy model based on the start instruction, complete the acquisition of security log data through the target transmission channel, and simultaneously start a monitoring thread, register the unique task identifier and its associated log file, and collect the key metrics of the proxy model in real time.

9. An electronic device, wherein, The electronic device includes: a processor; and, a memory storing computer-executable instructions, the executable instructions when executed cause the processor to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1-7 is implemented.

Images