Log field processing method and system based on description model and pluggable policy

Abstract

The present application relates to the technical field of log field processing, and provides a log field processing method and system based on a description model and a pluggable strategy to solve the problems of inflexible configuration, insufficient performance and difficulty in dynamic updating commonly faced by log field processing technologies.The log field processing method based on the description model and the pluggable strategy comprises receiving current log data and structurally analyzing the same to obtain a log data structure description model; reading pre-generated structured field processing configuration data and analyzing the same to determine a field path to be processed in the current log data, locate a target field to be processed in the log data structure description model, call a corresponding independently encapsulated pluggable field processing strategy unit, and then execute a corresponding field processing operation, write the processed field value back to a corresponding field position in the log data structure to update the log data content, and meet the log field processing requirements in modern high-concurrency and complex business scenarios.

Description

Technical Field

[0001] This invention relates to the field of log field processing technology, and in particular to a log field processing method and system based on a description model and a pluggable strategy. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In modern distributed systems and microservice architectures, logging has become a key tool for system monitoring, troubleshooting, and performance tuning. Especially in scenarios such as API gateways, service call chains, anomaly tracing, and user behavior collection, system logs often contain a large amount of request and response data, some of which may involve user privacy or sensitive information; these are the log fields that need to be processed.

[0004] Traditional log field processing methods fail to perform deep parsing of complex nested JSON and XML structures, making it difficult to accurately locate deep target fields. This results in incomplete coverage or the wrong fields being affected. For a large amount of log text without a fixed format, they often rely on inefficient regular expression matching, which cannot guarantee the accuracy of matching and execution performance, and can easily cause system bottlenecks. In addition, since log field processing rules need to be dynamically adjusted, frequent changes can lead to cache invalidation, complex hot updates, and some log field processing strategies have a large computational load, which may cause system response delays. Different business scenarios have different requirements for the intensity and method of log field processing, and existing technologies mostly adopt a single strategy, which cannot meet the complex and ever-changing compliance and security requirements. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a log field processing method and system based on a description model and pluggable strategies. This method enables log data processing schemes with different strategies configured for different fields, and can be adapted to different business scenarios.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a log field processing method based on a description model and a pluggable strategy.

[0007] In one or more embodiments, a log field processing method based on a description model and a pluggable strategy is provided. When the system generates new log data and is ready to output the log data, before the log data is written to the log storage medium, the method includes: Receive the current log data and perform structured parsing to obtain a log data structure description model; Read the pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data. Locate the target field to be processed in the log data structure description model and then call the corresponding independently encapsulated pluggable field processing strategy unit. Based on the retrieved corresponding field processing strategy unit, the corresponding field processing operation is executed, and the field value of each target field is processed according to the strategy parameter information; The processed field values ​​are written back to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

[0008] As one implementation method, the pre-generation process for structured field processing configuration data is as follows: Based on log data samples and their structured parsing results, a log data structure description model is constructed, which in turn forms a hierarchical field structure representation and generates a field configuration view accordingly. Upon receiving a field node selection operation, determine and record the complete field path and field type that match the field node selection operation in the field configuration view; The recorded complete field paths and field types are associated with independently encapsulated field processing strategy units to form field processing configuration items, and structured field processing configuration data is generated through structured storage.

[0009] As one implementation, when a user performs a selection operation on any field node in the field configuration view, the selection operation is responded to and the following processing is performed: Get the node identifier corresponding to the selected field; Determine the complete hierarchical path of this field in the log data structure description model based on the node identifier; Record the complete field path information of the field to uniquely identify the target field in the log data.

[0010] As one implementation method, during the structured parsing of log data, the log data is converted into an internally traversable data structure representation, the top-level fields in the log data are identified, and each top-level field is used as the parsing starting point, i.e., the root node. The log data is then traversed hierarchically and recursively according to the nesting relationship of the fields.

[0011] As one implementation method, during the traversal, for each field at the current level, the following operation is performed: Identify the field name of the field; Record the complete hierarchical path information of the field in the log data. The hierarchical path is used to uniquely identify the position of the field in the nested structure. Determine the data type of the field.

[0012] As one implementation method, different parsing strategies are executed based on the data type of the fields during the traversal of log data: When a field is of string, numeric, or boolean type, mark the field as a basic data field and record its field path and field type information; When a field is an object type, mark the field as the parent node field, and use the field as the new hierarchical starting point to continue the recursive traversal operation on its contained child fields; When the field is an array type, the data structure characteristics of the elements in the array are further analyzed, the field structure and data type contained in the array elements are identified, and the structural description corresponding to the array elements is abstracted to form a unified model of the overall array structure.

[0013] As one implementation method, the field information collected during the hierarchical recursive traversal of log data is summarized to form a complete log data structure information set; the log data structure information set includes at least: the field names of all fields in the log data; the hierarchical path relationship of each field in the log data; the data type corresponding to each field; and the hierarchical relationship between object fields and array fields; Based on the aggregated log data structure information set, a log data structure description model is constructed.

[0014] As one implementation, each field processing strategy unit includes at least the following: Strategy type identifier, used to distinguish different types of field processing strategies; Strategy parameter information is used to describe the processing rules or constraints required during field processing; The execution method description indicates how the policy will be processed.

[0015] A second aspect of the present invention provides a log field processing system based on a description model and a pluggable strategy.

[0016] In one or more embodiments, a log field processing system based on a description model and a pluggable strategy includes: The log data structure description model construction module is used to receive the current log data and perform structured parsing to obtain the log data structure description model; The field processing strategy unit retrieval module is used to read pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data, locate the target field to be processed in the log data structure description model, and then retrieve the corresponding independently encapsulated pluggable field processing strategy unit. The field processing strategy unit execution module is used to execute the corresponding field processing operation based on the invoked corresponding field processing strategy unit, and to process the field value of each target field according to the strategy parameter information. The log data content update module is used to write back the processed field values ​​to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

[0017] A third aspect of the present invention provides an electronic device.

[0018] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the log field processing method based on the description model and pluggable strategy described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention models log data to form a log data structure description model, obtaining the field hierarchy and type information of the logs. Then, it combines pre-generated structured field processing configuration data to generate standardized configuration rules, avoiding errors from manually writing expressions. Furthermore, it locates the target fields to be processed in the log data structure description model and matches and calls the corresponding independently encapsulated pluggable field processing strategy units. This supports log field processing for multi-format logs, expression-driven processing, runtime dynamic configuration, and flexible combinations of multiple strategies, meeting the log field processing needs of modern high-concurrency and complex business scenarios. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of a log field processing method based on a description model and a pluggable strategy according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the log field processing system structure based on a description model and a pluggable strategy according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] Figure 1 A schematic diagram of the log field processing method based on a description model and a pluggable strategy according to an embodiment of the present invention is provided. Figure 1 The log field processing method based on the description model and pluggable strategy in this embodiment may include the following steps S101 to S104 before the log data is written to the log storage medium when the system generates new log data and is ready to output the log data.

[0026] The specific implementation process of steps S101 to S104 is as follows: Step S101: Receive the current log data and perform structured parsing to obtain the log data structure description model.

[0027] In the specific implementation process, during the structured parsing of log data, the log data is converted into an internally traversable data structure representation, the top-level fields in the log data are identified, and each top-level field is used as the parsing starting point, i.e., the root node. The log data is then traversed hierarchically and recursively according to the nesting relationship of the fields.

[0028] During the traversal, for each field at the current level, the following operations are performed: Identify the field name of the field; Record the complete hierarchical path information of the field in the log data. The hierarchical path is used to uniquely identify the position of the field in the nested structure. Determine the data type of the field.

[0029] The data types of the fields must include at least string, numeric, boolean, object, and array types.

[0030] During the process of traversing log data, different parsing strategies are executed based on the data type of the fields: When a field is of string, numeric, or boolean type, mark the field as a basic data field and record its field path and field type information; When a field is an object type, mark the field as the parent node field, and use the field as the new hierarchical starting point to continue the recursive traversal operation on its contained child fields; When the field is an array type, the data structure characteristics of the elements in the array are further analyzed, the field structure and data type contained in the array elements are identified, and the structural description corresponding to the array elements is abstracted to form a unified model of the overall array structure.

[0031] Summarize the field information collected during the hierarchical recursive traversal of log data to form a complete log data structure information set; the log data structure information set should include at least: the field names of all fields in the log data; the hierarchical path relationship of each field in the log data; the data type of each field; and the hierarchical relationship between object fields and array fields; Based on the aggregated log data structure information set, a log data structure description model is constructed.

[0032] The log data structure description model uses JSON Schema to uniformly describe the overall field structure, field hierarchy, and field type information of the log data. This provides a unified data foundation for subsequent field selection and configuration generation.

[0033] When generating the log data structure description model, the field hierarchy path is used as the basis for structure organization to ensure that the structure in the JSON Schema is consistent with the actual nesting relationship of the log data; the log data structure description model is explicitly used as an intermediate structure carrier in the configuration and execution process of log desensitization rules.

[0034] The log data structure description model is not used for validating the validity or format of log data. Instead, it supports subsequent operations such as field selection, field path binding, de-identification expression association, and field processing strategy matching. Through this approach, log de-identification rules can be dynamically configured and automatically executed based on the actual structure of the log data.

[0035] Step S102: Read the pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data. Locate the target field to be processed in the log data structure description model, and then call the corresponding independently encapsulated pluggable field processing strategy unit.

[0036] Based on the field path information recorded in the field processing configuration data, the target field to be processed is located in the parsed log data structure. This field path-based location method accurately matches fields in nested structures, avoiding matching errors caused by duplicate field names or unclear hierarchical levels.

[0037] In step S102, the pre-generation process of structured field processing configuration data includes steps a-c; Specifically, step a: Based on log data samples and their structured parsing results, construct a log data structure description model, thereby forming a hierarchical field structure representation, and generating a field configuration view accordingly.

[0038] First, obtain one or more actual generated log data samples from the log generation source.

[0039] The log data sample consists of business logs or call logs generated during the actual operation of the system. The data content is organized in a structured data format, preferably JSON format.

[0040] Log data samples may contain multi-level nested data structures, including nested objects, nested arrays, and fields of different data types, to reflect the complex structural characteristics of log data in real business scenarios.

[0041] For example, using the top-level field of the log data as the root node, the field nodes in the schema are organized into a tree structure according to the nesting relationship of the field paths. In the tree structure, each node corresponds to a field in the log data, and the parent-child relationship between nodes reflects the actual nesting hierarchy of the fields in the log data.

[0042] Step b: Receive the field node selection operation, and determine and record the complete field path and field type that match the field node selection operation in the field configuration view.

[0043] The field configuration view is used to present the hierarchical distribution of fields in log data, displaying the nesting relationship of fields in a visual hierarchy, thus providing an intuitive basis for field selection operations.

[0044] It should be noted that the field configuration view serves only as the entry point for field selection and does not participate in the log data processing logic itself.

[0045] When a user selects any field node in the field configuration view, the selection operation is responded to, and the following processing is performed: Get the node identifier corresponding to the selected field; Determine the complete hierarchical path of this field in the log data structure description model based on the node identifier; Record the complete field path information of the field to uniquely identify the target field in the log data.

[0046] By recording complete field path information, the system can accurately locate fields in nested structures, avoiding ambiguity caused by relying solely on field names.

[0047] Step c: Associate the recorded complete field paths and field types with the independently encapsulated field processing strategy units to form field processing configuration items and generate structured field processing configuration data through structured storage. This is more intuitive and accurate than traditional regular expression matching or manual expression configuration; it generates standardized configuration rules and avoids errors from manually writing expressions.

[0048] After recording the field path information, the field path is associated with the field processing strategy to form a field processing configuration item.

[0049] The field processing strategy can be a predefined system strategy or a user-configured strategy based on actual needs.

[0050] Each field processing configuration item includes at least the field path information, field type information, and the corresponding field processing strategy identifier.

[0051] When a user selects a parent node field in a tree structure, the system automatically identifies the child node structure contained within that parent node field. Based on preset configuration rules, it determines whether to apply the field processing strategy associated with the parent node to all its child fields, or only to some of them.

[0052] When different processing strategies are required for different sub-fields, it is supported to configure the sub-fields under the parent node independently, thereby reducing repetitive configuration operations while ensuring flexibility.

[0053] After completing the field selection and strategy association operations, the generated field processing configuration items are summarized and processed.

[0054] Each field processing configuration item is stored in a structured manner according to field path, field type, and field processing strategy, forming unified field processing configuration data. This field processing configuration data serves as the configuration basis for the log processing process, and is used to dynamically match and process log data during the log output phase.

[0055] During the log output phase, based on the field processing configuration data, each field in the log data is matched one by one, and the corresponding field processing strategy is invoked according to the matched field path information to achieve automated processing of log fields. In this way, the field selection operation can directly affect the log processing behavior, thereby enabling flexible configuration and dynamic execution of log anonymization rules.

[0056] Step S103: Execute the corresponding field processing operation based on the retrieved corresponding field processing strategy unit, and process the field values ​​of each target field according to the strategy parameter information.

[0057] Multiple field processing strategies are pre-configured in the memory to form a set of field processing strategies.

[0058] Field processing strategies are used to perform different types of processing operations on field values ​​in log data. Different field processing strategies correspond to different log field processing methods. Field processing strategies here include, but are not limited to, field value replacement, field value truncation, field value obfuscation, field value encryption, field value deletion, and keeping field values ​​unchanged.

[0059] For example: desensitization (such as masking mobile phone numbers); Hide (delete field); Field length pruning (keeping or hiding based on threshold); Abstract (defaults to the first 100 characters); In one or more embodiments, users can configure different strategies for different fields.

[0060] Each field processing strategy is encapsulated as an independent strategy unit and defined in a configurable manner. Each field processing strategy unit includes at least the following: Strategy type identifier, used to distinguish different types of field processing strategies; Strategy parameter information is used to describe the processing rules or constraints required during field processing; The execution method description indicates how the policy will be processed.

[0061] By using the above method, the field processing strategy exists in the form of a standardized strategy unit, which facilitates subsequent unified invocation and management.

[0062] After completing the field selection and policy association operations, the field path information is bound to the corresponding policy unit to generate field processing configuration data. This field processing configuration data describes which fields need to be processed during log processing and the corresponding field processing strategies. The field processing configuration data is stored in memory as the basis for configuration during the log output phase.

[0063] It should be noted that the execution of the field processing strategy unit is handled by the processor, which performs log data parsing, field path matching, strategy unit scheduling, and field value rewriting. Field processing configuration data and the set of field processing strategies are stored in memory to support strategy loading and execution. Log data after field processing (such as de-identification) is completed is written to the log storage medium for persistent storage. Through the collaborative work of the processor and memory, the automated execution of log field processing strategies is achieved.

[0064] Step S104: Write the processed field values ​​back to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

[0065] This embodiment models log data to form a log data structure description model, obtains the field hierarchy and type information of the log, and then combines it with pre-generated structured field processing configuration data to generate standardized configuration rules, avoiding errors from manually writing expressions. Moreover, it locates the target field to be processed in the log data structure description model and matches and calls the corresponding independently encapsulated pluggable field processing strategy unit, realizing a log data processing scheme with different strategies for different fields, which can be adapted to different business scenarios.

[0066] like Figure 2 As shown, the log field processing system based on the description model and pluggable strategy provided in this embodiment of the invention can be implemented in software. The log field processing system based on the description model and pluggable strategy includes the following software modules: log data structure description model construction module 201, field processing strategy unit retrieval module 202, field processing strategy unit execution module 203, and log data content update module 204.

[0067] The following section describes the functions of each software module in the log field processing system based on a description model and a pluggable strategy: Log data structure description model construction module 201 is used to receive the current log data and perform structured parsing to obtain the log data structure description model; The field processing strategy unit retrieval module 202 is used to read pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data, locate the target field to be processed in the log data structure description model, and then retrieve the corresponding independently encapsulated pluggable field processing strategy unit. The field processing strategy unit execution module 203 is used to execute the corresponding field processing operation based on the invoked corresponding field processing strategy unit, and to process the field value of each target field according to the strategy parameter information. The log data content update module 204 is used to write back the processed field values ​​to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

[0068] It should be noted that each module in the log field processing system based on description model and pluggable strategy in the embodiments of the present invention corresponds one-to-one with each step in the log field processing method based on description model and pluggable strategy in the above embodiments, and their specific implementation processes are the same, so they will not be repeated here.

[0069] This embodiment models log data to form a log data structure description model, obtains the field hierarchy and type information of the log, and then combines it with pre-generated structured field processing configuration data to generate standardized configuration rules, avoiding errors from manually writing expressions. Moreover, it locates the target field to be processed in the log data structure description model and matches and calls the corresponding independently encapsulated pluggable field processing strategy unit. It can support log field processing of multi-format logs, expression-driven, runtime dynamic configuration, and flexible combination of multiple strategies to meet the log field processing needs of modern high-concurrency and complex business scenarios.

[0070] The structure of the electronic device according to an embodiment of the present invention will be described in detail below. Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 3 The diagram shows only an exemplary structure of the electronic device, not the entire structure. Some or all of the structures shown may be implemented as needed.

[0071] The electronic device provided in this embodiment of the invention includes: at least one processor 301, a memory 302, a user interface 303, and at least one network interface 304. The various components in the log field processing system based on a description model and a pluggable strategy are coupled together via a bus system 305. It is understood that the bus system 305 is used to implement communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general designated all buses as Bus System 305.

[0072] The user interface 303 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0073] It is understood that memory 302 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 302 is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as framework layers, core library layers, driver layers, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0074] In some embodiments, the log field processing system based on a description model and a pluggable strategy provided in this invention can be implemented using a combination of hardware and software. For example, the log field processing system based on a description model and a pluggable strategy provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the log field processing method based on a description model and a pluggable strategy provided in this invention. For instance, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] As an example, processor 301 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] As an example of the hardware implementation of the log field processing system based on the description model and pluggable strategy provided in the embodiments of the present invention, the device provided in the embodiments of the present invention can be directly executed by a processor 301 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the log field processing method based on the description model and pluggable strategy provided in the embodiments of the present invention.

[0077] The memory 302 in this embodiment of the invention is used to store various types of data to support the operation of a log field processing system based on a description model and a pluggable strategy, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operating on a log field processing system based on a description model and a pluggable strategy, such as executable instructions that can be included in a program implementing the log field processing method based on a description model and a pluggable strategy according to embodiments of the present invention.

[0078] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0079] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A log field processing method based on a description model and a pluggable strategy, characterized in that, When the system generates new log data and prepares to output it, before the log data is written to the log storage medium, the following steps are taken: Receive the current log data and perform structured parsing to obtain a log data structure description model; Read the pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data. Locate the target field to be processed in the log data structure description model and then call the corresponding independently encapsulated pluggable field processing strategy unit. Based on the retrieved corresponding field processing strategy unit, the corresponding field processing operation is executed, and the field value of each target field is processed according to the strategy parameter information; The processed field values ​​are written back to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

2. The log field processing method based on a description model and a pluggable strategy as described in claim 1, characterized in that, The pre-generation process for structured field processing configuration data is as follows: Based on log data samples and their structured parsing results, a log data structure description model is constructed, which in turn forms a hierarchical field structure representation and generates a field configuration view accordingly. Upon receiving a field node selection operation, determine and record the complete field path and field type that match the field node selection operation in the field configuration view; The recorded complete field paths and field types are associated with independently encapsulated field processing strategy units to form field processing configuration items, and structured field processing configuration data is generated through structured storage.

3. The log field processing method based on a description model and a pluggable strategy as described in claim 2, characterized in that, When a user selects any field node in the field configuration view, the selection operation is responded to, and the following processing is performed: Get the node identifier corresponding to the selected field; Determine the complete hierarchical path of this field in the log data structure description model based on the node identifier; Record the complete field path information of the field to uniquely identify the target field in the log data.

4. The log field processing method based on a description model and a pluggable strategy as described in claim 1, characterized in that, In the process of structured parsing of log data, the log data is converted into an internally traversable data structure representation, the top-level fields in the log data are identified, and each top-level field is used as the starting point for parsing, i.e., the root node. The log data is then traversed hierarchically and recursively according to the nesting relationship of the fields.

5. The log field processing method based on a description model and a pluggable strategy as described in claim 4, characterized in that, During the traversal, for each field at the current level, the following operations are performed: Identify the field name of the field; Record the complete hierarchical path information of the field in the log data. The hierarchical path is used to uniquely identify the position of the field in the nested structure. Determine the data type of the field.

6. The log field processing method based on a description model and a pluggable strategy as described in claim 1, characterized in that, During the process of traversing log data, different parsing strategies are executed based on the data type of the fields: When a field is of string, numeric, or boolean type, mark the field as a basic data field and record its field path and field type information; When a field is an object type, mark the field as the parent node field, and use the field as the new hierarchical starting point to continue the recursive traversal operation on its contained child fields; When the field is an array type, the data structure characteristics of the elements in the array are further analyzed, the field structure and data type contained in the array elements are identified, and the structural description corresponding to the array elements is abstracted to form a unified model of the overall array structure.

7. The log field processing method based on a description model and a pluggable strategy as described in claim 1, characterized in that, The field information collected during the hierarchical recursive traversal of log data is summarized to form a complete set of log data structure information; The log data structure information set should include at least: the field names of all fields in the log data; the hierarchical path relationship of each field in the log data; the data type of each field; and the hierarchical relationship between object fields and array fields. Based on the aggregated log data structure information set, a log data structure description model is constructed.

8. The log field processing method based on a description model and a pluggable strategy as described in claim 1, characterized in that, Each field processing strategy unit should include at least the following: Strategy type identifier, used to distinguish different types of field processing strategies; Strategy parameter information is used to describe the processing rules or constraints required during field processing; The execution method description indicates how the policy will be processed.

9. A log field processing system based on a description model and a pluggable strategy, characterized in that, The log field processing method based on the description model and pluggable strategy as described in any one of claims 1-8 includes: The log data structure description model construction module is used to receive the current log data and perform structured parsing to obtain the log data structure description model; The field processing strategy unit retrieval module is used to read pre-generated structured field processing configuration data from the memory and parse it to determine the field path to be processed in the current log data, locate the target field to be processed in the log data structure description model, and then retrieve the corresponding independently encapsulated pluggable field processing strategy unit. The field processing strategy unit execution module is used to execute the corresponding field processing operation based on the invoked corresponding field processing strategy unit, and to process the field value of each target field according to the strategy parameter information. The log data content update module is used to write back the processed field values ​​to the corresponding field positions in the log data structure to update the log data content and output it to the log storage medium.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the log field processing method based on the description model and pluggable strategy as described in any one of claims 1-8.

Images