Log field mapping method and device based on large language model, equipment and medium

Abstract

The invention discloses a log field mapping method and device based on a large language model, equipment and a medium. The method comprises the steps of performing data preprocessing on an original log data set corresponding to a target management system based on a target large language model to obtain a basic log data set; based on the target large language model, the target context information and the historical field mapping rule set, carrying out normalized mapping on the basic log data set to obtain a first mapping field set and a to-be-selected log data set; determining a sub-field set corresponding to each piece of to-be-selected log data in the to-be-selected log data set and a corresponding dynamic mapping rule based on the target large language model, and generating a second mapping field set corresponding to the to-be-selected log data set based on the dynamic mapping rule, and performing combined processing on the first mapping field set and the second mapping field set to generate a target log field mapping result. By means of the technical scheme, field mapping can be automatically conducted on the log data, and the efficiency and accuracy of field mapping are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a log field mapping method, device, equipment and medium based on a large language model. Background Art

[0002] With the rapid development of computer technology, event management systems are gradually being applied to various business scenarios to manage the log data of various business scenarios. However, since the log formats and field contents of event management systems and various business scenarios are often different, it is very important to perform field mapping on the log data of various business scenarios and obtain standardized field mapping results to ensure that the fields of log data can correctly and completely participate in the subsequent event management process.

[0003] In the prior art, a set of parsing rules is usually created for each log data type by manually writing parsing rules, such as regular expressions or keyword mapping rules. Then, the log data of each business scenario is mapped to fields through regular expressions or keyword mapping rules to obtain mapped standard fields.

[0004] However, if we use the method of manually writing parsing rules, it is not only time-consuming and labor-intensive, but also limits the flexibility of the event management system to deal with new log data types or field changes, and reduces the efficiency and accuracy of field mapping. Therefore, how to automatically map the fields of log data and improve the efficiency and accuracy of field mapping is an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a log field mapping method, device, equipment and medium based on a large language model, which can solve the problem of low efficiency and accuracy of field mapping of log data.

[0006] According to one aspect of the present invention, a log field mapping method based on a large language model is provided, comprising:

[0007] Obtain the original log data set and target context information corresponding to the target management system;

[0008] Performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set;

[0009] Based on the target large language model, the target context information and the historical field mapping rule set, the basic log data set is normalized and mapped to obtain a first mapping field set and a to-be-selected log data set corresponding to the basic log data set;

[0010] Based on the target large language model, determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, combine and process the first mapping field set and the second mapping field set to generate the target log field mapping result corresponding to the original log data set.

[0011] According to another aspect of the present invention, a log field mapping device based on a large language model is provided, comprising:

[0012] A data acquisition module is used to obtain the original log data set and target context information corresponding to the target management system;

[0013] A data preprocessing module, configured to perform data preprocessing on each original log data in the original log data set based on a target large language model to obtain a basic log data set corresponding to the original log data set;

[0014] A first mapping module, configured to perform normalized mapping on the basic log data set based on a target large language model, target context information, and a historical field mapping rule set, to obtain a first mapping field set and a candidate log data set corresponding to the basic log data set;

[0015] The second mapping module is used to determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data based on the target large language model, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, and combine and process the first mapping field set and the second mapping field set to generate a target log field mapping result corresponding to the original log data set.

[0016] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log field mapping method based on a large language model described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log field mapping method based on a large language model described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the log field mapping method based on a large language model according to any embodiment of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention obtains the original log data set and target context information corresponding to the target management system. Then, the original log data in the original log data set is preprocessed based on the target large language model to obtain the basic log data set corresponding to the original log data set, and the basic log data set is normalized and mapped based on the target large language model, the target context information and the historical field mapping rule set to obtain the first mapping field set and the selected log data set corresponding to the basic log data set. Finally, the sub-field set corresponding to each selected log data in the selected log data set and the dynamic mapping rule corresponding to each selected log data are determined based on the target large language model, and the second mapping field set corresponding to the selected log data set is generated based on the dynamic mapping rule, and the first mapping field set and the second mapping field set are combined and processed to generate the target log field mapping result corresponding to the original log data set. Due to the automatic processing capability of the large language model, the tedious steps of manually writing rules are omitted, the configuration time of field mapping is greatly shortened, the problem of low efficiency and accuracy of field mapping of log data is solved, and the field mapping of log data can be automatically performed, which improves the efficiency and accuracy of field mapping.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a log field mapping method based on a large language model provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a log field mapping method based on a large language model provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a flowchart of an optional log field mapping method based on a large language model provided according to Embodiment 2 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of a log field mapping device based on a large language model provided according to Embodiment 3 of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device for implementing the log field mapping method based on a large language model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", "target", "original", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 The flowchart of a log field mapping method based on a large language model provided in the first embodiment of the present invention is applicable to the case of automatically mapping fields of log data. The method can be executed by a log field mapping device based on a large language model. The log field mapping device based on a large language model can be implemented in the form of hardware and / or software. The log field mapping device based on a large language model can be configured in an electronic device. Figure 1As shown, the method includes:

[0034] S110: Acquire an original log data set and target context information corresponding to the target management system.

[0035] The management system may refer to a system that manages various security event data. Generally, the management system may be used to collect and analyze various log data. The target management system may refer to a management system selected to manage log data.

[0036] Among them, raw log data can refer to unprocessed log data received by the target management system. Usually, raw log data can be generated by multiple sources, such as a protection system, a production system, or a detection system, and can also be generated in multiple language formats. A raw log data set can refer to a set of raw log data received by the same target management system at the same time. Usually, each raw log data set can contain raw log data generated by multiple sources.

[0037] The context information may refer to requirement information for evaluating the mapping process of log data. Exemplarily, the context information may be the requirement for converting A to B. The target context information may refer to the context information selected for log data mapping. Generally, the corresponding target context information may be determined according to the actual needs of the user.

[0038] S120 . Preprocess each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set.

[0039] Among them, data preprocessing may refer to the operation of performing preliminary data processing on each original log data in the original log data set. Exemplarily, data preprocessing may include data cleaning operations and data basic mapping operations. Basic log data may refer to log data generated after data preprocessing of the original log data. The basic log data set may refer to a set composed of each basic log data corresponding to the same original log data set.

[0040] The large language model may refer to a natural language processing model based on deep learning technology. Generally, the large language model can be trained through self-supervision with a large amount of unlabeled text data to obtain a deep understanding and expression ability of the language. The target large language model may refer to a large language model selected for field parsing of log data.

[0041] In an optional embodiment, before performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set, it also includes: obtaining a basic large language model and a historical log data set; wherein the historical log data set includes historical log data; performing model training on the basic large language model based on the historical log data set to obtain a trained target large language model. The basic large language model may refer to an original large language model that has not been trained with data. Historical log data may refer to log data received within a historical time period. The historical log data set may refer to a set consisting of each historical log data within the same historical time period. Exemplarily, the historical log data set may be a set consisting of each historical log data received within the past month.

[0042] Specifically, before performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain the basic log data set corresponding to the original log data set, the basic large language model can be pre-trained using each historical log data in the historical log data set to obtain a trained target large language model. This enables the target large language model to perform context analysis on the input log data, automatically extract key fields and preliminarily identify their semantic attributes, providing an effective basis for subsequent field mapping.

[0043] S130 , performing normalized mapping on the basic log data set based on the target large language model, target context information, and a historical field mapping rule set to obtain a first mapping field set and a candidate log data set corresponding to the basic log data set.

[0044] Among them, the historical field mapping rule may refer to an existing field mapping rule. Exemplarily, the historical field mapping rule may be a field mapping rule that has been stored in a preset rule library. The historical field mapping rule set may refer to a set composed of various historical field mapping rules. Normalized mapping may refer to the operation of performing preliminary field mapping on the basic log data set according to the historical field mapping rule set.

[0045] Among them, the first mapping field may refer to a mapping field generated after normalized mapping. The first mapping field set may refer to a set consisting of various first mapping fields corresponding to the same basic log data set. The log data to be selected may refer to basic log data that has not been successfully normalized and mapped. Exemplarily, the log data to be selected may be basic log data for which there is no corresponding historical field mapping rule in the historical field mapping rule set. The log data set to be selected may refer to a set consisting of various log data to be selected corresponding to the same basic log data set.

[0046] S140. Determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data based on the target large language model, generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, combine and process the first mapping field set and the second mapping field set, and generate the target log field mapping result corresponding to the original log data set.

[0047] Among them, the subfield may refer to each subfield in the selected log data. Exemplarily, each subfield may be identified by an underscore. Specifically, if the selected log data is "user_ip", the corresponding subfields may be user and ip. The subfield set may refer to a set consisting of each subfield corresponding to the same selected log data. Usually, one selected log data corresponds to one subfield set.

[0048] Among them, the dynamic mapping rule may refer to the field mapping rule regenerated for the selected log data. Exemplarily, the dynamic mapping rule corresponding to the selected log data can be determined by the sub-field set corresponding to the selected log data. The second mapping field may refer to the mapping field generated after field mapping of the selected log data based on the dynamic mapping rule. The second mapping field set may refer to a set composed of the second mapping fields corresponding to the same set of selected log data. The target log field mapping result may refer to the full mapping field set corresponding to the original log data set. Exemplarily, the target log field mapping result may include a first mapping field set and a second mapping field set.

[0049] In an optional embodiment, after determining the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data based on the target large language model, it also includes: storing the dynamic mapping rule in a preset rule base, and optimizing and adjusting the preset rule base based on preset optimization rules. Among them, the preset rule base may refer to a preset database for storing field mapping rules. Usually, after the field mapping rule is stored in the preset rule base, the field mapping rule can be used as a historical field mapping rule in a historical field mapping rule set to perform the next field mapping. The preset optimization rule may refer to a preset method for evaluating the field mapping rules in the preset rule base. Exemplarily, the preset optimization rule can be to adjust and optimize the field mapping rules in the preset rule base using the historical log data set.

[0050] Specifically, after determining the subfield set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data in the candidate log data set based on the target large language model, the dynamic mapping rule can be stored in the preset rule base. At the same time, the field mapping rules in the preset rule base are tested and verified using the historical log data set, and the verification result is compared with the original field mapping result. Based on the comparison result, the field mapping rules in the preset rule base are optimized and adjusted. In this way, the accuracy of the field mapping rules in the preset rule base can be improved, providing an effective basis for improving the accuracy of field mapping.

[0051] The technical solution of the embodiment of the present invention obtains the original log data set and target context information corresponding to the target management system. Then, the original log data in the original log data set is preprocessed based on the target large language model to obtain the basic log data set corresponding to the original log data set, and the basic log data set is normalized and mapped based on the target large language model, the target context information and the historical field mapping rule set to obtain the first mapping field set and the selected log data set corresponding to the basic log data set. Finally, the sub-field set corresponding to each selected log data in the selected log data set and the dynamic mapping rule corresponding to each selected log data are determined based on the target large language model, and the second mapping field set corresponding to the selected log data set is generated based on the dynamic mapping rule, and the first mapping field set and the second mapping field set are combined and processed to generate the target log field mapping result corresponding to the original log data set. Due to the automatic processing capability of the large language model, the tedious steps of manually writing rules are omitted, the configuration time of field mapping is greatly shortened, the problem of low efficiency and accuracy of field mapping of log data is solved, and the field mapping of log data can be automatically performed, which improves the efficiency and accuracy of field mapping.

[0052] Embodiment 2

[0053] Figure 2 A flowchart of a log field mapping method based on a large language model is provided for the second embodiment of the present invention. This embodiment is refined based on the above embodiment. In this embodiment, the operation of performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set is specifically refined. Specifically, it may include: formatting the original log data set based on preset formatting rules to obtain an intermediate log data set that meets the preset formatting rules; performing basic field mapping on each intermediate log data in the intermediate log data set based on the target large language model and the basic field rule set to obtain a basic log data set corresponding to the original log data set. Figure 2 As shown, the method includes:

[0054] S210: Obtain an original log data set and target context information corresponding to the target management system.

[0055] S220: Format the original log data set based on a preset formatting rule to obtain an intermediate log data set that meets the preset formatting rule.

[0056] Among them, the preset formatting rules may refer to pre-set rules for standardizing formatting processing operations. Exemplarily, the preset formatting rules may be rules for removing redundant characters or special symbols in log data, and for normalizing common fields. Formatting processing may refer to related data processing operations performed according to preset formatting rules. The integrity and consistency of log data can be ensured through formatting processing. Intermediate log data may refer to log data obtained after formatting original log data. An intermediate log data set may refer to a set consisting of intermediate log data corresponding to the same original log data set.

[0057] S230 , performing basic field mapping on each intermediate log data in the intermediate log data set based on the target large language model and the basic field rule set to obtain a basic log data set corresponding to the original log data set.

[0058] Among them, the basic field rule may refer to a rule for preliminarily mapping the general fields in the log data. Exemplarily, the basic field rule may be a rule for mapping the general field Internet Protocol (IP) to the field src_ip. Generally, the basic field rules corresponding to each general field may be pre-built according to user needs. The basic field rule set may refer to a set composed of each basic field rule. The basic field mapping may refer to an operation of preliminarily mapping the general fields.

[0059] Specifically, after obtaining the original log data set and target context information corresponding to the target management system, the original log data set can be formatted using the preset formatting rules to obtain an intermediate log data set that meets the preset formatting rules, and then, based on the target large language model, the common fields corresponding to each intermediate log data in the intermediate log data set are parsed, and the basic field rules in the basic field rule set are used to perform basic field mapping on the common fields corresponding to each intermediate log data. In this way, the basic log data set corresponding to the original log data set is obtained, which can simplify the processing complexity of subsequent steps and improve the efficiency of field mapping.

[0060] S240: Determine a target historical field mapping rule corresponding to the target context information based on the target context information and a historical field mapping rule set.

[0061] The target historical field mapping rule may refer to a historical field mapping rule in the historical field mapping rule set that matches the target context information. For example, if the target context information is to map src_ip to source_ip, the corresponding target historical field mapping rule may be a rule to map src_ip to source_ip.

[0062] S250. Determine the target log field corresponding to each basic log data in the basic log data set based on the target large language model, and normalize and map the target log field based on the target historical field mapping rule to obtain a first mapping field corresponding to the basic log data.

[0063] Among them, the target log field can refer to the key field corresponding to each basic log data in the basic log data set. Exemplarily, the target log field can be the source IP, destination IP, event type, user name, etc. Generally, the basic log data can be deeply semantically analyzed by the target large language model to identify the core fields in the basic log data.

[0064] S260: Combine and process the first mapping fields corresponding to the respective basic log data in the basic log data set to obtain a first mapping field set corresponding to the basic log data set.

[0065] Specifically, after obtaining the basic log data set corresponding to the original log data set, the target context information can be used to match the target historical field mapping rule corresponding to the target context information in the historical field mapping rule set. Then, the target large language model is used to perform deep semantic analysis on each basic log data in the basic log data set to determine the target log field corresponding to each basic log data. Further, the target log field is normalized and mapped based on the target historical field mapping rule to obtain the first mapping field corresponding to the basic log data. Finally, the first mapping fields corresponding to each basic log data in the basic log data set are combined and processed. Thus, the first mapping field set corresponding to the basic log data set is obtained.

[0066] S270: Determine a candidate log data set corresponding to the basic log data set based on each basic log data corresponding to the first mapping field set.

[0067] Specifically, after obtaining the first mapping field set corresponding to the basic log data set, the basic log data in the basic log data set that has not generated the corresponding first mapping field can be screened out as the selected log data, and the various selected log data corresponding to the same basic log data set can be combined to obtain the selected log data set corresponding to the basic log data set, providing an effective basis for subsequent operations.

[0068] S280: Determine, based on the target large language model, each subfield corresponding to each candidate log data in the candidate log data set, and combine and process each subfield corresponding to the same candidate log data to generate a subfield set corresponding to the candidate log data.

[0069] Specifically, after obtaining the candidate log data set corresponding to the basic log data set, the target large language model can be used to perform secondary deep semantic analysis on each candidate log data in the candidate log data set to obtain each subfield corresponding to each candidate log data, and then combine and process each subfield corresponding to the same candidate log data. In this way, a subfield set corresponding to the candidate log data is obtained, providing an effective basis for subsequent operations.

[0070] S290: Determine a dynamic mapping sub-rule corresponding to the target sub-field based on the target sub-field in the sub-field set and the historical field mapping rule set.

[0071] The target subfield may refer to a subfield selected in a subfield set for rule determination. The dynamic mapping subrule may refer to a field mapping rule generated based on the target subfield. Exemplarily, data matching may be performed in a historical field mapping rule set by the target subfield to determine the field mapping rule corresponding to the target subfield as the dynamic mapping subrule.

[0072] Specifically, if the log data to be selected is user_ip, the corresponding subfield set is (user,ip). Then the target subfield user can be used to match data in the historical field mapping rule set, determine the field mapping rule A corresponding to the target subfield user, and use field mapping rule A as the dynamic mapping subrule corresponding to the target subfield user. Similarly, the target subfield ip can be used to match data in the historical field mapping rule set, determine the field mapping rule B corresponding to the target subfield ip, and use field mapping rule B as the dynamic mapping subrule corresponding to the target subfield ip.

[0073] S2100 , combining and processing the dynamic mapping sub-rules corresponding to the target sub-fields in the sub-field set to generate dynamic mapping rules corresponding to the log data to be selected.

[0074] Specifically, after obtaining the dynamic mapping sub-rules corresponding to each target sub-field in the sub-field set, the dynamic mapping sub-rules corresponding to the same selected log data can be combined and processed to generate the dynamic mapping rules corresponding to the selected log data. In this way, new mapping rules are generated for unrecognized fields in new log data, which can adapt to changes in log formats and provide an effective basis for subsequent operations.

[0075] S2110. Generate a second mapping field set corresponding to the selected log data set based on the dynamic mapping rule, combine and process the first mapping field set and the second mapping field set, and generate a target log field mapping result corresponding to the original log data set.

[0076] Specifically, after generating the dynamic mapping rules corresponding to the selected log data, the dynamic mapping rules can be used to perform field mapping on the matching selected log data to obtain the second mapping field corresponding to the selected log data, and all the second mapping fields corresponding to the selected log data set are combined to obtain the second mapping field set corresponding to the selected log data set. Further, the first mapping field set and the second mapping field set are combined to generate the target log field mapping result corresponding to the original log data set. In this way, the field mapping of all log data is completed, providing an effective basis for subsequent event management.

[0077] S2120: Send the target log field mapping result to the target management system, so that the target management system performs corresponding data management operations on the target log field mapping result based on preset data management requirements.

[0078] The preset data management requirement may refer to a pre-set rule for regulating the event management process. For example, the preset data management requirement may be to integrate and process the various log data in the target log field mapping result. The data management operation may refer to a data processing operation performed according to the preset data management requirement.

[0079] Specifically, after generating the target log field mapping result corresponding to the original log data set, the target log field mapping result can be sent to the target management system, and then, the target management system can perform corresponding data management operations on the target log field mapping result based on preset data management requirements to realize event management.

[0080] The technical solution of the embodiment of the present invention is to obtain the original log data set and target context information corresponding to the target management system. Then, the original log data set is formatted based on the preset formatting rules to obtain the intermediate log data set that meets the preset formatting rules, and the basic fields are mapped to each intermediate log data in the intermediate log data set based on the target large language model and the basic field rule set to obtain the basic log data set corresponding to the original log data set. Further, the target historical field mapping rules corresponding to the target context information are determined based on the target context information and the historical field mapping rule set, the target log field corresponding to each basic log data in the basic log data set is determined based on the target large language model, the target log field is normalized and mapped based on the target historical field mapping rules to obtain the first mapping field corresponding to the basic log data, the first mapping field corresponding to each basic log data in the basic log data set is combined and processed to obtain the first mapping field set corresponding to the basic log data set, and based on each basic log data corresponding to the first mapping field set, the candidate log data set corresponding to the basic log data set is determined. Further, based on the target large language model, the subfields corresponding to each of the selected log data in the selected log data set are determined, and the subfields corresponding to the same selected log data are processed in combination to generate the subfield set corresponding to the selected log data, and the dynamic mapping subrules corresponding to the target subfields and the historical field mapping rule set in the subfield set are determined, and the dynamic mapping subrules corresponding to each of the target subfields in the subfield set are processed in combination to generate the dynamic mapping rules corresponding to the selected log data, and the second mapping field set corresponding to the selected log data set is generated based on the dynamic mapping rules, and the first mapping field set and the second mapping field set are processed in combination to generate the target log field mapping result corresponding to the original log data set. Finally, the target log field mapping result is sent to the target management system so that the target management system performs the corresponding data management operation on the target log field mapping result based on the preset data management requirements. Due to the automatic processing capability of the large language model, the tedious steps of manually writing rules are omitted, the configuration time of field mapping is greatly shortened, and the problem of low efficiency and accuracy of field mapping of log data is solved. The field mapping of log data can be automatically performed, and the efficiency and accuracy of field mapping are improved.

[0081] Figure 3The flowchart of an optional log field mapping method based on a large language model provided by an embodiment of the present invention is shown. Specifically, first, the original log data set and the target context information corresponding to the target management system are obtained. After that, the original log data set is formatted based on the preset formatting rules to obtain an intermediate log data set that meets the preset formatting rules, and the basic fields are mapped to each intermediate log data in the intermediate log data set based on the target large language model and the basic field rule set to obtain the basic log data set corresponding to the original log data set, and the data preprocessing is completed. Then, the target historical field mapping rule corresponding to the target context information is determined based on the target context information and the historical field mapping rule set, and the target log field corresponding to each basic log data in the basic log data set is determined based on the target large language model, and the field identification and context semantic analysis are completed, and the target log field is normalized and mapped based on the target historical field mapping rule to obtain the first mapping field corresponding to the basic log data. The first mapping field corresponding to each basic log data in the basic log data set is combined and processed to obtain the first mapping field set corresponding to the basic log data set, and the candidate log data set corresponding to the basic log data set is determined based on each basic log data corresponding to the first mapping field set. Further, the subfields corresponding to each candidate log data in the candidate log data set are determined based on the target large language model, and the subfields corresponding to the same candidate log data are processed in combination to generate a subfield set corresponding to the candidate log data, and the dynamic mapping subrules corresponding to the target subfields and the historical field mapping rule set in the subfield set are determined, and the dynamic mapping subrules corresponding to each target subfield in the subfield set are processed in combination to generate dynamic mapping rules corresponding to the candidate log data, and a second mapping field set corresponding to the candidate log data set is generated based on the dynamic mapping rules to achieve dynamic mapping. Finally, the dynamic mapping rules are stored in a preset rule library, and the preset rule library is optimized and adjusted based on the preset optimization rules, so that subsequent field mappings can directly use the rules in the preset rule library.

[0082] Embodiment 3

[0083] Figure 4 The structural diagram of a log field mapping device based on a large language model provided in the third embodiment of the present invention. Figure 4 As shown, the device includes: a data acquisition module 310, a data preprocessing module 320, a first mapping module 330 and a second mapping module 340;

[0084] The data acquisition module 310 is used to acquire the original log data set and target context information corresponding to the target management system;

[0085] A data preprocessing module 320 is used to perform data preprocessing on each original log data in the original log data set based on a target large language model to obtain a basic log data set corresponding to the original log data set;

[0086] A first mapping module 330 is used to perform normalization mapping on the basic log data set based on the target large language model, target context information and a historical field mapping rule set to obtain a first mapping field set and a candidate log data set corresponding to the basic log data set;

[0087] The second mapping module 340 is used to determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data based on the target large language model, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, and combine and process the first mapping field set and the second mapping field set to generate a target log field mapping result corresponding to the original log data set.

[0088] The technical solution of the embodiment of the present invention obtains the original log data set and target context information corresponding to the target management system. Then, the original log data in the original log data set is preprocessed based on the target large language model to obtain the basic log data set corresponding to the original log data set, and the basic log data set is normalized and mapped based on the target large language model, the target context information and the historical field mapping rule set to obtain the first mapping field set and the selected log data set corresponding to the basic log data set. Finally, the sub-field set corresponding to each selected log data in the selected log data set and the dynamic mapping rule corresponding to each selected log data are determined based on the target large language model, and the second mapping field set corresponding to the selected log data set is generated based on the dynamic mapping rule, and the first mapping field set and the second mapping field set are combined and processed to generate the target log field mapping result corresponding to the original log data set. Due to the automatic processing capability of the large language model, the tedious steps of manually writing rules are omitted, the configuration time of field mapping is greatly shortened, the problem of low efficiency and accuracy of field mapping of log data is solved, and the field mapping of log data can be automatically performed, which improves the efficiency and accuracy of field mapping.

[0089] Optionally, the data preprocessing module 320 may be specifically used for:

[0090] Formatting the original log data set based on a preset formatting rule to obtain an intermediate log data set that meets the preset formatting rule;

[0091] Based on the target large language model and the basic field rule set, basic field mapping is performed on each intermediate log data in the intermediate log data set to obtain a basic log data set corresponding to the original log data set.

[0092] Optionally, the log field mapping device based on the large language model also includes: a model training module, which is used to obtain a basic large language model and a historical log data set before performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set; wherein the historical log data set contains historical log data; and model training is performed on the basic large language model based on the historical log data set to obtain a trained target large language model.

[0093] Optionally, the first mapping module 330 may be specifically configured to:

[0094] Determine a target historical field mapping rule corresponding to the target context information based on the target context information and a historical field mapping rule set;

[0095] Determine a target log field corresponding to each basic log data in the basic log data set based on the target large language model, and perform normalization mapping on the target log field based on the target historical field mapping rule to obtain a first mapping field corresponding to the basic log data;

[0096] Combining and processing the first mapping fields corresponding to the respective basic log data in the basic log data set to obtain a first mapping field set corresponding to the basic log data set;

[0097] Based on each basic log data corresponding to the first mapping field set, a candidate log data set corresponding to the basic log data set is determined.

[0098] Optionally, the second mapping module 340 may be specifically configured to:

[0099] Determine, based on the target large language model, each subfield corresponding to each to-be-selected log data in the to-be-selected log data set, and combine and process each subfield corresponding to the same to-be-selected log data to generate a subfield set corresponding to the to-be-selected log data;

[0100] Determine a dynamic mapping sub-rule corresponding to the target sub-field based on the target sub-field in the sub-field set and the historical field mapping rule set;

[0101] The dynamic mapping sub-rules corresponding to the target sub-fields in the sub-field set are combined and processed to generate dynamic mapping rules corresponding to the log data to be selected.

[0102] Optionally, the log field mapping device based on the large language model also includes: a rule storage module, which is used to store the dynamic mapping rules in a preset rule base after determining the sub-field set corresponding to each candidate log data in the candidate log data set based on the target large language model and the dynamic mapping rules corresponding to each candidate log data, and optimize and adjust the preset rule base based on preset optimization rules.

[0103] Optionally, the log field mapping device based on a large language model also includes: a post-processing module, which is used for sending the target log field mapping result to a target management system after the combined processing of the first mapping field set and the second mapping field set generates a target log field mapping result corresponding to the original log data set, so that the target management system performs corresponding data management operations on the target log field mapping result based on preset data management requirements.

[0104] The log field mapping device based on a large language model provided in the embodiment of the present invention can execute the log field mapping method based on a large language model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0105] Embodiment 4

[0106] Figure 5 A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 5As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0108] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 420 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 420 performs the various methods and processes described above, such as a log field mapping method based on a large language model.

[0110] The method includes:

[0111] Obtain the original log data set and target context information corresponding to the target management system;

[0112] Performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set;

[0113] Based on the target large language model, the target context information and the historical field mapping rule set, the basic log data set is normalized and mapped to obtain a first mapping field set and a to-be-selected log data set corresponding to the basic log data set;

[0114] Based on the target large language model, determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, combine and process the first mapping field set and the second mapping field set to generate the target log field mapping result corresponding to the original log data set.

[0115] In some embodiments, the log field mapping method based on the large language model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the log field mapping method based on the large language model described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the log field mapping method based on the large language model in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0122] The present application also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the log field mapping method based on a large language model provided in any embodiment of the present application. The program product and the log field mapping method based on a large language model disclosed in each embodiment of the present application belong to the same inventive concept, so they are not described here.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log field mapping method based on a large language model, characterized in that: include: Obtain the original log data set and target context information corresponding to the target management system; Performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set; Based on the target large language model, the target context information and the historical field mapping rule set, the basic log data set is normalized and mapped to obtain a first mapping field set and a to-be-selected log data set corresponding to the basic log data set; Based on the target large language model, determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, combine and process the first mapping field set and the second mapping field set to generate the target log field mapping result corresponding to the original log data set.

2. The method according to claim 1, characterized in that The method of performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set includes: Formatting the original log data set based on a preset formatting rule to obtain an intermediate log data set that meets the preset formatting rule; Based on the target large language model and the basic field rule set, basic field mapping is performed on each intermediate log data in the intermediate log data set to obtain a basic log data set corresponding to the original log data set.

3. The method according to claim 1, characterized in that Before performing data preprocessing on each original log data in the original log data set based on the target large language model to obtain a basic log data set corresponding to the original log data set, the method further includes: Acquire a basic large language model and a historical log data set; wherein the historical log data set includes historical log data; The basic large language model is trained based on the historical log data set to obtain a trained target large language model.

4. The method according to claim 1, characterized in that The method of normalizing and mapping the basic log data set based on the target large language model, the target context information and the historical field mapping rule set to obtain a first mapping field set and a to-be-selected log data set corresponding to the basic log data set includes: Determine a target historical field mapping rule corresponding to the target context information based on the target context information and a historical field mapping rule set; Determine the target log field corresponding to each basic log data in the basic log data set based on the target large language model, and perform normalization mapping on the target log field based on the target historical field mapping rule to obtain a first mapping field corresponding to the basic log data; Combining and processing the first mapping fields corresponding to the respective basic log data in the basic log data set to obtain a first mapping field set corresponding to the basic log data set; Based on each basic log data corresponding to the first mapping field set, a candidate log data set corresponding to the basic log data set is determined.

5. The method according to claim 1, characterized in that: The determining, based on the target large language model, a subfield set corresponding to each log data to be selected in the log data set to be selected and a dynamic mapping rule corresponding to each log data to be selected, comprises: Determine, based on the target large language model, each subfield corresponding to each to-be-selected log data in the to-be-selected log data set, and combine and process each subfield corresponding to the same to-be-selected log data to generate a subfield set corresponding to the to-be-selected log data; Determine a dynamic mapping sub-rule corresponding to the target sub-field based on the target sub-field in the sub-field set and the historical field mapping rule set; The dynamic mapping sub-rules corresponding to the target sub-fields in the sub-field set are combined and processed to generate dynamic mapping rules corresponding to the log data to be selected.

6. The method according to claim 1, characterized in that After determining the subfield set corresponding to each to-be-selected log data in the to-be-selected log data set and the dynamic mapping rule corresponding to each to-be-selected log data based on the target large language model, the method further includes: The dynamic mapping rules are stored in a preset rule base, and the preset rule base is optimized and adjusted based on preset optimization rules.

7. The method according to claim 1, characterized in that After the combination processes the first mapping field set and the second mapping field set to generate a target log field mapping result corresponding to the original log data set, the method further includes: The target log field mapping result is sent to a target management system, so that the target management system performs corresponding data management operations on the target log field mapping result based on preset data management requirements.

8. A log field mapping device based on a large language model, characterized in that: include: A data acquisition module is used to obtain the original log data set and target context information corresponding to the target management system; A data preprocessing module, used for performing data preprocessing on each original log data in the original log data set based on a target large language model to obtain a basic log data set corresponding to the original log data set; A first mapping module, configured to perform normalized mapping on the basic log data set based on a target large language model, target context information, and a historical field mapping rule set, to obtain a first mapping field set and a candidate log data set corresponding to the basic log data set; The second mapping module is used to determine the sub-field set corresponding to each candidate log data in the candidate log data set and the dynamic mapping rule corresponding to each candidate log data based on the target large language model, and generate the second mapping field set corresponding to the candidate log data set based on the dynamic mapping rule, and combine and process the first mapping field set and the second mapping field set to generate a target log field mapping result corresponding to the original log data set.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log field mapping method based on a large language model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log field mapping method based on a large language model according to any one of claims 1 to 7 when executed.

Images