Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the steps that a data processing unit obtains first log data; the data processing unit decomposes the first log data into first sub-data based on a minimum unit of decomposition of the first log data; the first sub-data at least comprises partial data in the first log data; the minimum unit represents a minimum component in the first log data; the data processing unit performs data processing on the first sub-data based on the included protocol to obtain second log data corresponding to the protocol; the protocol is pre-stored in the data processing unit; the data processing unit controls the forwarding process of the second log data based on the included protocol configuration item; the protocol configuration item is used for determining a forwarding rule of the second log data.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] The current data processing method involves each protocol module processing log data independently, resulting in duplicate log processing. Different protocols employ different data processing methods, and there is no unified management approach for each protocol. When the log data format is adjusted, the data processing methods for each protocol need to be modified, leading to significant code changes. Furthermore, when importing a new protocol, the corresponding data processing methods for that protocol need to be reconfigured. This current data processing method suffers from resource waste, complex protocol configuration, maintenance difficulties, and poor scalability. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0004] This application provides a data processing method, comprising: a data processing unit obtaining first log data; the data processing unit decomposing the first log data into first sub-data based on the smallest unit of decomposition of the first log data; the first sub-data including at least a portion of the data in the first log data; the smallest unit representing the smallest component in the first log data; the data processing unit performing data processing on the first sub-data based on an included protocol to obtain second log data corresponding to the protocol; the protocol being pre-stored in the data processing unit; the data processing unit controlling the forwarding process of the second log data based on included protocol configuration items; the protocol configuration items being used to determine the forwarding rules of the second log data.

[0005] According to one embodiment of this application, the method further includes: determining the smallest unit for decomposing the first log data; determining the smallest unit for decomposing the first log data includes: determining the log type of the first log data; determining the log splitting rule corresponding to the log type based on the log type; the log splitting rule is preset based on log format and content attributes; different log splitting rules correspond to different smallest units; determining the smallest unit for decomposing the first log data based on the log splitting rule corresponding to the log type.

[0006] According to one embodiment of this application, the step of processing the first sub-data based on the included protocols to obtain the second log data corresponding to the protocols includes: obtaining at least one protocol included in the data processing unit; the protocol includes at least field data rules, log format, and data reorganization rules; the protocol is used to convert the first sub-data into second log data that conforms to the field data rules, log format, and data reorganization rules; determining the data to be processed from the first sub-data based on the field data rules of each protocol; and reorganizing the data to be processed based on the log format and data reorganization rules of each protocol to obtain the second log data corresponding to each protocol.

[0007] According to one embodiment of this application, the data processing unit controls the forwarding process of the second log data based on the included protocol configuration items, including: parsing the protocol configuration items to obtain forwarding rules for the second log data; the forwarding rules include at least a forwarding method, a target object, and forwarding conditions; based on the forwarding rules, determining log data to be forwarded from the second log data that meets the forwarding conditions; and transmitting the log data to be forwarded to the forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object through the forwarding method.

[0008] According to one embodiment of this application, determining the log data to be forwarded from the second log data that meets the forwarding conditions based on the forwarding rules includes: determining a first log type set in the forwarding conditions; the first log type is determined according to the forwarding rules pre-configured in the protocol configuration file; different log types correspond to different forwarding methods and target objects; in response to the log type of the second log data matching the first log type, the corresponding second log data is determined as the log data to be forwarded.

[0009] According to one embodiment of this application, the method further includes: in response to the data processing unit receiving a configuration instruction for the protocol configuration items, obtaining a first configuration file, the first configuration file including at least protocol configuration items corresponding to each protocol; the first configuration file being used to configure the forwarding rules of each protocol for the second log data; updating the protocol configuration items in the first configuration file based on the configuration instruction to obtain an updated first configuration file; the data processing unit loading the updated first configuration file to control the forwarding process of the second log data based on the protocol configuration items included in the updated first configuration file.

[0010] According to one embodiment of this application, the monitoring module of the data processing unit is instantiated using a singleton pattern; the monitoring module includes a first monitoring module and a second monitoring module, the first monitoring module is used to generate the first log data, and the second monitoring module is used to monitor changes in the protocol configuration items.

[0011] This application also provides a data processing apparatus, comprising: an acquisition module for a data processing unit to acquire first log data; a decomposition module for the data processing unit to decompose the first log data into first sub-data based on the smallest unit of decomposition of the first log data; the first sub-data including at least a portion of the data in the first log data; the smallest unit representing the smallest component in the first log data; a processing module for the data processing unit to perform data processing on the first sub-data based on an included protocol to obtain second log data corresponding to the protocol; the protocol being pre-stored in the data processing unit; and a forwarding module for the data processing unit to control the forwarding process of the second log data based on included protocol configuration items; the protocol configuration items being used to determine the forwarding rules of the second log data.

[0012] This application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the above-described embodiments.

[0013] This application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0015] Figure 1 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 1 ; Figure 2 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 2 ; Figure 3 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 3 ; Figure 4 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 4 ; Figure 5 This illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application; Figure 6 This illustration shows another application scenario of the data processing method provided in the embodiments of this application; Figure 7 An optional schematic diagram of the data processing apparatus provided in an embodiment of this application is shown; Figure 8 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0016] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] The processing flow of the data processing method provided in the embodiments of this application will be described. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S104 are explained below.

[0021] In step S101, the data processing unit obtains the first log data.

[0022] In step S102, the data processing unit decomposes the first log data into first sub-data based on the smallest unit of decomposition of the first log data; the first sub-data includes at least some data in the first log data; the smallest unit represents the smallest component in the first log data.

[0023] In some embodiments, the data processing unit can be used to uniformly manage log data, implementing operations such as log monitoring, parsing, reassembly, and transmission. Specifically, the data processing unit can be a middleware used for communication between the BMC (Baseboard Management Controller) and the receiving end. The middleware can also be used for communication between other devices; this application embodiment does not limit the specific device used. The first log data may include: log data generated by the BMC. The smallest unit may include: an independent, indivisible component of the log data, i.e., the smallest constituent part. The first sub-data may include: partial data extracted from the first log data based on the smallest unit. Examples include timestamps, log types, message bodies, severity levels, etc. This application embodiment does not limit the specific first sub-data.

[0024] In step S103, the data processing unit processes the first sub-data based on the included protocol to obtain the second log data corresponding to the protocol; the protocol is pre-stored in the data processing unit.

[0025] In step S104, the data processing unit controls the forwarding process of the second log data based on the included protocol configuration items; the protocol configuration items are used to determine the forwarding rules of the second log data.

[0026] In some embodiments, the protocol may include various communication rules pre-stored in the data processing unit, such as SMTP, RSYSLOG, etc. The second log data may include log data reassembled according to protocol requirements. The second log data can be sent according to predetermined forwarding rules. Protocol configuration items may include various configuration parameters related to the protocol. Protocol configuration items can configure forwarding rules. Forwarding rules may include forwarding method, target object, and forwarding conditions. Forwarding rules can be used to determine whether the second log data is sent, where it is sent, and how it is sent. Configuring forwarding rules ensures that the second log data is transmitted in a set manner. For example, configuring the sending target address, configuring the port number, configuring forwarding conditions, etc.

[0027] As an example, taking the remote sending of server logs by the BMC system as an example, the specific steps are as follows: The data processing unit monitors the first log data generated by the server in real time. When new logs are generated on the server, the data processing unit obtains the corresponding first log data. The data processing unit decomposes the first log data based on predefined log splitting rules, extracting the smallest units, such as timestamp, log type, message body, severity level, etc., to obtain the first sub-data. For example, an original log "2025-12-10 15:30:00, SEL, CPU temperature too high, warning" will be decomposed into the first sub-data such as timestamp "2025-12-10 15:30:00", log type "SEL", message body "CPU temperature too high", and severity level "warning".

[0028] The data processing unit processes the first sub-data according to pre-stored protocols (such as SMTP, RSYSLOG, etc.) to obtain the second log data. Taking the SMTP protocol as an example, the protocol specifies the format for sending emails, including fields such as sender, recipient, subject, and body. The data processing unit extracts the required data from the first sub-data according to the SMTP protocol's field data rules, such as using the message body as the email body and combining the timestamp and log type into the email subject. The data processing unit then reassembles the extracted data into second log data that conforms to SMTP requirements, according to the SMTP protocol's log format and data reassembly rules.

[0029] The data processing unit loads the protocol configuration items and parses the forwarding rules. For example, the forwarding method is SMTP, the target object is aaa@bb.com, and the forwarding condition is that the log type is SEL and the severity level is greater than warning. It determines whether the reconstructed second log data meets the forwarding conditions. If the second log data meets the forwarding conditions, the data processing unit transmits the qualified second log data to the corresponding SMTP forwarding unit. The forwarding unit then sends the second log data to the target object aaa@bb.com according to the general sending method of the SMTP protocol.

[0030] The method in this application embodiment achieves unified monitoring, parsing, and transmission of log data through centralized processing by a data processing unit. Log data is monitored and parsed only once, avoiding the problem of logs being repeatedly processed by multiple protocol modules in traditional distributed architectures, reducing performance resource waste, and improving overall performance. Data processing through a data processing unit simplifies configuration management, eliminating the need to configure the monitoring and parsing processes of multiple protocol modules separately, reducing configuration complexity and the possibility of errors, and making subsequent maintenance more convenient. Controlling log data forwarding through a data processing unit enhances system scalability, allowing for the addition of new protocols or modification of existing protocols without requiring extensive adjustments to the data processing methods of each protocol, thus improving the adaptability and flexibility of the data processing method.

[0031] In some embodiments, the data processing method further includes: determining the smallest unit for decomposing the first log data. Determining the smallest unit for decomposing the first log data includes: determining the log type of the first log data; determining a log splitting rule corresponding to the log type based on the log type; the log splitting rule is pre-defined based on log format and content attributes; different log splitting rules correspond to different smallest units; and determining the smallest unit for decomposing the first log data based on the log splitting rule corresponding to the log type.

[0032] In this embodiment, the log type may include: system logs, application logs, security logs, operation logs, etc. Other log types may also be included; this embodiment does not limit the specific log type. Log splitting rules may include: predefined splitting rules. Log splitting rules can be used to indicate how to decompose different types of log data into the smallest units. Log format may include: the structural components of the log, such as timestamp position, log level representation, message content position, etc. Content attributes may include: the specific information types contained in the log, such as error codes, user IDs, operation types, resource names, etc. This embodiment does not limit the specific log format and content attributes.

[0033] As an example, taking the remote transmission of server logs in the BMC system as an example, the data processing unit obtains the first log data generated by the server. The data processing unit first determines the log type of the first log data, for example, by identifying a specific identifier at the beginning of the log to determine whether the log is a system log, application log, or security log, etc. Then, based on the determined log type, it retrieves the rule corresponding to the log type from predefined log splitting rules. Log splitting rules are pre-defined according to the log format and content attributes; different log types correspond to different splitting rules. For example, system logs can be split based on timestamps, process IDs, message content, etc., while security logs can be split based on user IDs, operation types, result status, etc. Finally, based on the selected log splitting rule, the smallest unit corresponding to the first log data is determined.

[0034] The method in this application, by determining the smallest unit for decomposing the first log data, can extract key information from the log. Log data is only monitored and parsed once, avoiding the problem of logs being repeatedly processed by multiple protocol modules in traditional distributed architectures, reducing performance resource waste, and improving overall performance.

[0035] In some embodiments, the processing flow of the data processing method is illustrated. Figure 2 ,like Figure 2As shown, step S103, based on the included protocol, performs data processing on the first sub-data to obtain the second log data corresponding to the protocol, which may specifically include: Step S201: Obtain at least one protocol included in the data processing unit.

[0036] Step S202: Based on the field data rules of each protocol, determine the data to be processed from the first sub-data.

[0037] Step S203: Based on the log format and data reorganization rules of each protocol, the data to be processed is reorganized to obtain the second log data corresponding to each protocol.

[0038] In this embodiment, field data rules may include: the data type, format, and value range of each field defined in the protocol. Field data rules can be used to determine the data to be reorganized from the first sub-data. Data reorganization rules can be used to combine the determined data to be processed into a complete log according to a set order and method.

[0039] As an example, the data processing unit first obtains at least one pre-stored protocol, which may include SMTP, SNMP, RSYSLOG, etc. Each protocol defines field data rules, log format, and data reorganization rules to indicate how to reorganize the first sub-data into second log data that conforms to the requirements of that protocol. Taking the SMTP protocol as an example, its field data rules can specify that log data includes fields such as sender, recipient, subject, and body, and each field has corresponding format requirements, such as the subject's character count being less than a set character count. Based on the SMTP protocol's field data rules, the data processing unit determines the data to be processed from the first sub-data that meets the requirements. For example, it extracts the log severity level as the email priority, extracts the message body as the email body, and extracts the timestamp and log type to combine into the email subject. The data processing unit then organizes the extracted data into an email format according to the SMTP protocol's log format requirements to obtain the second log data.

[0040] The method in this application embodiment reassembles log data through protocol field data rules, log format, and data reassembly rules. The data processing unit can reassemble the first sub-data into second log data conforming to specific protocol specifications according to different protocol requirements. The reassembled log data can meet the specific requirements of various protocols, improving data processing compatibility and enhancing universality across different protocols. It avoids log data transmission errors and reduces the risk of inconsistencies in log transmission across different protocols.

[0041] In some embodiments, the processing flow of the data processing method is illustrated. Figure 3 ,like Figure 3 As shown, the data processing unit in step S104 controls the forwarding process of the second log data based on the included protocol configuration items, which may specifically include: Step S301: Parse the protocol configuration items to obtain the forwarding rules for the second log data.

[0042] Step S302: Based on the forwarding rules, determine the log data to be forwarded that meets the forwarding conditions from the second log data.

[0043] In some embodiments, step S302 may include: determining a first log type set in the forwarding conditions; the first log type is determined according to the forwarding rules pre-configured in the protocol configuration file; different log types correspond to different forwarding methods and target objects; in response to the second log data's log type matching the first log type, the corresponding second log data is determined as log data to be forwarded.

[0044] In this embodiment, the protocol configuration file may include at least one protocol configuration item corresponding to at least one protocol. Each protocol configuration item can define the forwarding rules for the corresponding protocol, including the destination address, port number, log type, and severity level. When controlling the forwarding process of log data, the data processing unit loads the protocol configuration file and parses it to obtain the protocol configuration items.

[0045] As an example, if SMTP requires forwarding only event logs, while RSYSLOG requires forwarding both event logs and audit logs, then the protocol configuration file could look like this: { “SMTP”: [ “SEL” ], “RSYSLOG”:[ “SEL”, "Login", “Operation”, Maintain ], … } As an example, taking the remote transmission of server logs in the BMC system as an example, after generating the second log data, the data processing unit parses the protocol configuration file to obtain the first log type set in the forwarding conditions. Specifically, the protocol configuration file may include: for the SMTP protocol, only event-type log data is forwarded; for the RSYSLOG protocol, both event-type and audit-type log data are forwarded. The data processing unit iterates through the second log data, comparing the corresponding log type of each second log data with the first log type set in the forwarding conditions, and determining the second log data that matches the first log type as the log data to be forwarded. For example, for the SMTP protocol, only event-type log data is determined as the log data to be forwarded; for the RSYSLOG protocol, both event-type and audit-type log data are determined as the log data to be forwarded.

[0046] Step S303: The log data to be forwarded is transmitted to the forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object through forwarding.

[0047] In this embodiment, the first log type may include: log types pre-defined in the protocol configuration file, such as operation logs, audit logs, etc. This application embodiment does not limit the specific first log type. The forwarding unit can be used to: receive log data to be forwarded, and send the log data to be forwarded to the target object through the corresponding forwarding method.

[0048] As an example, taking the remote transmission of server logs in the BMC system, after generating the second log data, the data processing unit parses the protocol configuration items to obtain the first log type set in the forwarding conditions. For example, the protocol configuration file may include log data of the operation log type that is only forwarded. For each piece of second log data, the data processing unit compares the corresponding log type with the first log type set in the forwarding conditions. When it is determined that the log type of the second log data matches the first log type, the corresponding second log data is identified as log data to be forwarded. For example, if a piece of second log data is marked as an operation log type, this data is identified as log data to be forwarded. The data processing unit transmits all log data to be forwarded to the corresponding forwarding unit. The forwarding unit sends the log data to be forwarded to the predetermined target object according to the protocol's general transmission method.

[0049] The method in this embodiment obtains forwarding rules by parsing protocol configuration items, thereby controlling the log forwarding process. The data processing unit can determine the forwarding method, target object, and forwarding conditions of the second log data, ensuring that the log data is sent according to predetermined rules. This avoids log data transmission errors and reduces the risk of inconsistencies in log transmission across different protocols. Furthermore, the log forwarding process can be flexibly adjusted according to actual needs to meet different management requirements.

[0050] In some embodiments, the processing flow of the data processing method is illustrated. Figure 4 ,like Figure 4 As shown, the data processing method may also include: Step S401: In response to the data processing unit receiving a configuration instruction for the protocol configuration items, the first configuration file is obtained.

[0051] Step S402: Update the protocol configuration items in the first configuration file based on the configuration instructions to obtain the updated first configuration file.

[0052] In step S403, the data processing unit loads the updated first configuration file to control the forwarding process of the second log data based on the protocol configuration items included in the updated first configuration file.

[0053] In this embodiment, the configuration instructions may include instructions received through a management interface or configuration file modification. The configuration instructions can be used to update protocol configuration items. The first configuration file may include files corresponding to the protocol configuration items for each protocol. The first configuration file can be used to configure the forwarding rules for the second log data for each protocol.

[0054] As an example, the user enters the new forwarding target address and port number for RSYSLOG in the management interface and clicks the save button. At this time, the data processing unit receives the corresponding configuration instruction. The data processing unit identifies the protocol (such as RSYSLOG) corresponding to the configuration instruction and loads the current first configuration file from the storage location. The first configuration file contains the current configuration items for each protocol, such as target address, port number, log type, etc. Based on the configuration instruction, the RSYSLOG protocol configuration items in the first configuration file are updated, that is, the target address of the RSYSLOG protocol is updated from aa@bb.com to aa@cc.com, and the port number is updated from 0 to 1. After completing the update of the protocol configuration items, the data processing unit saves the modified configuration file and generates the updated first configuration file. The data processing unit reloads the updated first configuration file. In the subsequent log forwarding process, the data processing unit controls the forwarding of the second log data according to the updated protocol configuration items. That is, when processing log data of a new event type, the data processing unit will send the log to aa@cc.com according to the updated RSYSLOG protocol configuration items and send it using the new port number 1.

[0055] The method in this application embodiment achieves real-time adjustment of log forwarding rules by dynamically updating protocol configuration items. When a configuration command is received for a protocol configuration item, the data processing unit can respond and update the corresponding configuration item, quickly adapting to new forwarding requirements. The design of independently monitoring protocol configuration items enhances module cohesion, making the system highly portable and easy to deploy in different environments. Log forwarding rules can be adjusted at any time according to actual conditions without redeployment, improving the flexibility of data processing.

[0056] In some embodiments, the monitoring module of the data processing unit is instantiated using the singleton pattern; the monitoring module includes a first monitoring module and a second monitoring module, the first monitoring module is used to generate the first log data, and the second monitoring module is used to monitor changes in protocol configuration items.

[0057] In this embodiment, the singleton pattern can be used to ensure the global uniqueness of the monitoring module of the data processing unit. The first monitoring module can be a component used to monitor the generation of the first log data. The second monitoring module can be a component used to monitor changes in protocol configuration items. Specifically, the first monitoring module can be a RemoteMonitor module, which can be instantiated using the singleton pattern. It is used to monitor log generation from the start of program startup and, once logs are generated, decompose the log content into the smallest elements (timestamp, log type, message body, severity level, etc.). Specifically, the second monitoring module can be a protocol Config module, which can be instantiated using the singleton pattern. It is used to monitor changes in the configuration files of each protocol from the start of program startup and synchronize them to the program in a timely manner.

[0058] As an example, the data processing unit includes the RemoteMonitor module, the LogRedo (log reassembly) module, various protocol Config modules, and various protocol Trap (forwarding) modules. During initialization, the RemoteMonitor module and each protocol Config module are instantiated using the singleton pattern and begin operation. The RemoteMonitor monitors log generation in real time, capturing logs and breaking them down into the smallest units such as timestamps and log types. The LogRedo module then reads the protocol configuration items and reassembles the log format according to the protocol requirements. For example, it reassembles SEL type logs into an email format conforming to the SMTP protocol, or a syslog message format conforming to the RSYSLOG protocol. Each protocol Config module continuously monitors for protocol configuration changes; when it detects user modifications to the protocol configuration file, it promptly updates the internal system, and subsequent log processing applies the updated protocol configuration items. Finally, each protocol Trap module receives the reassembled logs transmitted by the LogRedo module and forwards them directly according to the protocol requirements, such as sending emails via SMTP or sending them to a remote server via RSYSLOG.

[0059] The method in this application, through a unified log processing framework, achieves centralized monitoring and parsing of logs, avoiding performance waste caused by redundant processing. The design of independent monitoring protocol configuration items enhances module cohesion, making the system highly portable and easy to deploy in different environments. The framework supports extending to new protocols; existing log parsing code can be reused simply by importing the corresponding Trap interface, simplifying the development process. The standardization of the log sending process not only facilitates problem localization and troubleshooting but also reduces the risk of inconsistencies in log sending across different protocols, enhancing the stability of log data processing. Furthermore, configurable parameters are provided externally, allowing for rapid customization of log data processing according to project needs.

[0060] refer to Figure 5The application scenario diagram of the data processing method provided in this application embodiment is applied to the remote log forwarding program framework.

[0061] The remote log forwarding program framework includes: listening to logs, parsing logs, parsing various protocol configuration items, and protocol forwarding.

[0062] Log monitoring: After the system starts, the RemoteMonitor module begins to monitor the logs generated by the server.

[0063] Log parsing: After capturing logs, the RemoteMonitor module breaks them down into the smallest units, such as timestamp "2025-12-10 15:30:00", log type "SEL", message body "CPU temperature too high", and severity level "Warning".

[0064] Parsing the configuration items of each protocol: The LogRedo module reads the configuration items of the SMTP protocol, including the sender's email address "aaa@bbb.com", the recipient's email address "ccc@ddd.com", the SMTP server address "smtp.111.com", and the port number "0", etc.

[0065] Protocol forwarding: The LogRedo module reassembles the smallest unit into an email format according to the requirements of the SMTP protocol. The email subject is set to "Warning-SEL-2025-12-10 15:30:00", and the body is set to "CPU temperature too high". Then, this email is sent to the Trap module of each protocol, and the Trap module sends the email to the specified recipient email address "ccc@ddd.com" via the SMTP protocol.

[0066] refer to Figure 6 The application scenario diagram of the data processing method provided in this application embodiment is applied to log processing and protocol configuration item monitoring in the remote log forwarding program framework.

[0067] After the program starts, the RemoteMonitor module and the protocol Config module begin working simultaneously: Log generation monitoring: The RemoteMonitor module monitors the logs generated by the server in real time.

[0068] Breaking logs down into their smallest units: After capturing logs, the RemoteMonitor module breaks them down into their smallest units, such as timestamp "2025-12-10 15:30:00", log type "SEL", message body "CPU temperature too high", and severity level "Warning".

[0069] Monitoring Protocol Configuration Items: Each protocol Config module monitors changes to the protocol configuration files in real time. When a user modifies the SMTP protocol configuration items through the management interface, the protocol Config module detects the change.

[0070] Timely update of variable information: The protocol Config module synchronizes new protocol configuration items to the program, updates the relevant protocol configuration items, and uses the updated protocol configuration items for subsequent log processing.

[0071] The following continues to describe exemplary structures of the data processing apparatus 90 provided in the embodiments of this application as software modules. In some embodiments, such as Figure 7 As shown, the data processing device 90 may include: an acquisition module 901, used by the data processing unit to acquire first log data; a decomposition module 902, used by the data processing unit to decompose the first log data into first sub-data based on the smallest unit of decomposition of the first log data; the first sub-data includes at least some data in the first log data; the smallest unit represents the smallest component in the first log data; a processing module 903, used by the data processing unit to process the first sub-data based on the included protocol to obtain second log data corresponding to the protocol; the protocol is pre-stored in the data processing unit; and a forwarding module 904, used by the data processing unit to control the forwarding process of the second log data based on the included protocol configuration items; the protocol configuration items are used to determine the forwarding rules of the second log data.

[0072] In some embodiments, the obtaining module 901 is further configured to: determine the smallest unit for decomposing the first log data; determining the smallest unit for decomposing the first log data includes: determining the log type of the first log data; determining the log splitting rule corresponding to the log type based on the log type; the log splitting rule is preset based on log format and content attributes; different log splitting rules correspond to different smallest units; and determining the smallest unit for decomposing the first log data based on the log splitting rule corresponding to the log type. In some embodiments, the processing module 903 is configured to: obtain at least one protocol included in the data processing unit; the protocol includes at least field data rules, log format, and data reorganization rules; the protocol is used to convert first sub-data into second log data that conforms to the field data rules, log format, and data reorganization rules; determine the data to be processed from the first sub-data based on the field data rules of each protocol; and reorganize the data to be processed based on the log format and data reorganization rules of each protocol to obtain the second log data corresponding to each protocol.

[0073] In some embodiments, the forwarding module 904 is configured to: parse the protocol configuration items to obtain the forwarding rules for the second log data; the forwarding rules include at least a forwarding method, a target object, and forwarding conditions; based on the forwarding rules, determine the log data to be forwarded that meets the forwarding conditions from the second log data; and transmit the log data to be forwarded to the forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object through the forwarding method.

[0074] In some embodiments, the forwarding module 904 is configured to: determine a first log type set in the forwarding conditions; the first log type is determined according to the forwarding rules pre-configured in the protocol configuration file; different log types correspond to different forwarding methods and target objects; in response to the second log data's log type matching the first log type, determine the corresponding second log data as log data to be forwarded.

[0075] In some embodiments, the data processing apparatus 90 may further include an update module, which is configured to: obtain a first configuration file in response to the data processing unit receiving a configuration instruction for protocol configuration items, the first configuration file including at least protocol configuration items corresponding to each protocol; configure the forwarding rules of each protocol for the second log data; update the protocol configuration items in the first configuration file based on the configuration instruction to obtain an updated first configuration file; and load the updated first configuration file to control the forwarding process of the second log data based on the protocol configuration items included in the updated first configuration file.

[0076] In some embodiments, the monitoring module of the data processing unit is instantiated using the singleton pattern; the monitoring module includes a first monitoring module and a second monitoring module, the first monitoring module is used to generate first log data, and the second monitoring module is used to monitor changes in protocol configuration items.

[0077] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figures 1 to 7 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0078] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0079] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0080] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0081] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0082] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0083] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0087] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0088] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0089] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0090] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, comprising: The data processing unit obtains the first log data; The data processing unit decomposes the first log data into a first sub-data based on the smallest unit of decomposition of the first log data; the first sub-data includes at least a portion of the data in the first log data; the smallest unit represents the smallest component in the first log data. The data processing unit processes the first sub-data based on the included protocol to obtain the second log data corresponding to the protocol; the protocol is pre-stored in the data processing unit. The data processing unit controls the forwarding process of the second log data based on the included protocol configuration items; the protocol configuration items are used to determine the forwarding rules of the second log data.

2. The method according to claim 1, further comprising: Determine the smallest unit for decomposing the first log data; The smallest unit for determining the first log data to be decomposed includes: Determine the log type of the first log data; Based on the log type, the corresponding log splitting rule is determined; the log splitting rule is based on the log format and content attributes that are pre-defined; different log splitting rules correspond to different minimum units; Based on the log splitting rules corresponding to the log type, the smallest unit for decomposing the first log data is determined.

3. The method according to claim 1, wherein processing the first sub-data based on the included protocol to obtain the second log data corresponding to the protocol includes: Obtain at least one protocol included in the data processing unit; The protocol includes at least field data rules, log format, and data reorganization rules; The protocol is used to convert the first sub-data into second log data that conforms to field data rules, log format, and data reorganization rules; Based on the field data rules of each protocol, the data to be processed is determined from the first sub-data. Based on the log format and data reorganization rules of each protocol, the data to be processed is reorganized to obtain the second log data corresponding to each protocol.

4. The method according to claim 1, wherein the data processing unit controls the forwarding process of the second log data based on the included protocol configuration items, comprising: The protocol configuration items are parsed to obtain the forwarding rules for the second log data; The forwarding rules include at least the forwarding method, the target object, and the forwarding conditions; Based on the forwarding rules, log data to be forwarded that meets the forwarding conditions is determined from the second log data; The log data to be forwarded is transmitted to the forwarding unit corresponding to the protocol, so that the forwarding unit forwards the log data to be forwarded to the target object through the forwarding method.

5. The method according to claim 4, wherein determining the log data to be forwarded from the second log data that satisfies the forwarding conditions based on the forwarding rules comprises: Determine the first log type set in the forwarding conditions; The first log type is determined according to the forwarding rules pre-configured in the protocol configuration file; different log types correspond to different forwarding methods and target objects; In response to the second log data matching the first log type, the corresponding second log data is determined as the log data to be forwarded.

6. The method according to claim 1, further comprising: In response to the data processing unit receiving a configuration instruction for the protocol configuration items, a first configuration file is obtained, the first configuration file including at least the protocol configuration items corresponding to each protocol; the first configuration file is used to configure the forwarding rules of each protocol for the second log data; The protocol configuration items in the first configuration file are updated based on the configuration instructions to obtain the updated first configuration file; The data processing unit loads the updated first configuration file to control the forwarding process of the second log data based on the protocol configuration items included in the updated first configuration file.

7. The method according to claim 1, wherein the monitoring module of the data processing unit is instantiated using a singleton pattern; the monitoring module includes a first monitoring module and a second monitoring module, wherein the first monitoring module is used to generate the first log data, and the second monitoring module is used to monitor changes in the protocol configuration items.

8. A data processing apparatus, comprising: The acquisition module is used by the data processing unit to acquire the first log data; The decomposition module is used by the data processing unit to decompose the first log data into first sub-data, which is the smallest unit for decomposing the first log data into first sub-data. The first sub-data includes at least a portion of the data in the first log data. The smallest unit represents the smallest component in the first log data. A processing module is used by the data processing unit to process the first sub-data based on the included protocol to obtain the second log data corresponding to the protocol; the protocol is pre-stored in the data processing unit. The forwarding module is used by the data processing unit to control the forwarding process of the second log data based on the included protocol configuration items; the protocol configuration items are used to determine the forwarding rules of the second log data.

9. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-7.

Images