Log file processing method and device

Abstract

The present invention provides a log file processing method and device, the method comprising: when receiving the log configuration information corresponding to each application system sent by CMBD, determining each data dimension; based on the log configuration information corresponding to each application system, checking whether each log file of each application system meets each data dimension, if so, collecting each log file; reading the log data of each log file, and checking whether the log data contains sensitive fields; if so, desensitizing the sensitive fields; determining the storage strategy corresponding to each log file, and based on each storage strategy, storing each log file, completing the processing process of each log file in each application system. Applying the method provided by the present invention, log files can be collected through each data dimension, and the log files can be stored according to the storage strategy after desensitizing the log files, so as to prevent the leakage of sensitive data in the log files.

Description

Technical Field

[0001] The present invention relates to the technical field of log file processing, and in particular to a log file processing method and device. Background Art

[0002] In the transaction system of a bank, there are many application systems, and each application system will generate different types of log files when executing different application businesses. With the application and promotion of new technologies such as distributed, microservices, and containers, the centralized collection, dynamic storage, and real-time retrieval of application logs are indispensable monitoring services for large enterprises, especially banking transaction systems.

[0003] In the prior art, a log sending and receiving mechanism is usually set in each application system to collect various log files in different application systems. However, the prior art requires that a corresponding sending and receiving mechanism be set for each application system. When the number of application systems increases, the number of log files also increases, and it is impossible to collect each log file on demand, and each log file usually contains a lot of user-related sensitive data. If the collection of log files is achieved through the sending and receiving mechanism, not only can the log files not be collected according to actual needs, but also the sensitive data in the log files cannot be processed. Summary of the invention

[0004] In view of this, the present invention provides a log file processing method, through which log files can be collected through various data dimensions, and the log files can be desensitized and then stored according to the storage strategy to prevent the leakage of sensitive data in the log files.

[0005] The present invention also provides a log file processing device to ensure the implementation and application of the above method in practice.

[0006] A log file processing method, comprising:

[0007] When receiving the log configuration information corresponding to each application system sent by the preset configuration management database CMBD, determining each preset data dimension;

[0008] Based on the log configuration information corresponding to each application system, check whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, collect each log file in the application system;

[0009] Read the log data in each of the log files, and check whether each of the log data contains sensitive fields;

[0010] When any log data contains sensitive fields, desensitize the sensitive fields in the log data according to the pre-set desensitization rules, and set the log file to which the desensitized log data belongs as the first log file;

[0011] When any log data does not contain a sensitive field, the log file to which the log data belongs is set as the second log file;

[0012] Determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

[0013] The above method, optionally, based on the log configuration information corresponding to each application system, checks whether each log file of each application system satisfies each data dimension, and when it is determined that each log file of any application system satisfies each data dimension, collects each log file in the application system, including:

[0014] Setting the log collection order corresponding to each of the application systems;

[0015] Based on the log collection sequence and the log configuration information corresponding to each of the application systems, the log collection process corresponding to each of the application systems is sequentially executed to obtain each log file corresponding to each of the application systems based on the log collection process corresponding to each of the application systems;

[0016] For each of the application systems, the log collection process corresponding to the application system includes: based on the configuration information corresponding to the application system, determining the number of log files, the total file size, the file transfer flow and the transfer time corresponding to each log file to be collected in the application system, and judging whether the log collection switch corresponding to the application system is in the on state; if so, judging whether the number of log files exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transfer time exceeds the preset duration threshold; if the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transfer time does not exceed the preset duration threshold, then collecting each log file in the application system and ending the log collection process corresponding to the application system; if the number of log files exceeds the preset number threshold, or the total file size exceeds the preset capacity threshold, or the file transfer flow exceeds the preset flow threshold, or the transmission time exceeds the preset duration threshold, then ending the log collection process corresponding to the application system.

[0017] In the above method, optionally, the log collection process corresponding to the application system further includes:

[0018] When the log collection process corresponding to the application system starts to execute, a preset application component lock is obtained;

[0019] Apply the application component lock to lock the log collection process corresponding to the application system;

[0020] When the log collection process corresponding to the application system is completed, the application component lock is released.

[0021] In the above method, optionally, performing desensitization processing on sensitive fields in the log data according to a preset desensitization rule includes:

[0022] Extracting sensitive fields from the log data;

[0023] Determine the data type corresponding to the sensitive field;

[0024] Determine the security level corresponding to the data type and the desensitization method corresponding to the security level;

[0025] Based on the desensitization method, the sensitive field is desensitized.

[0026] The above method, optionally, determining a storage strategy corresponding to each of the first log files and the second log files, and parsing and storing each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, includes:

[0027] Based on the log configuration information, determine the CMBD tag corresponding to each of the first log file and the second log file set by the CMBD;

[0028] For each CMBD tag, determine whether the CMBD tag contains a storage identification number; the storage identification number is a cluster flag of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

[0029] A log file processing device, comprising:

[0030] A determination unit, configured to determine each preset data dimension when receiving log configuration information corresponding to each application system sent by a preset configuration management database CMBD;

[0031] A collection unit, configured to check whether each log file of each application system satisfies each data dimension based on the log configuration information corresponding to each application system, and collect each log file in the application system when it is determined that each log file of any application system satisfies each data dimension;

[0032] A checking unit, used for reading the log data in each of the log files, and checking whether each of the log data contains sensitive fields;

[0033] A desensitization unit, for desensitizing the sensitive fields in the log data according to a preset desensitization rule when any log data contains sensitive fields, and setting the log file to which the desensitized log data belongs as the first log file;

[0034] A setting unit, configured to set the log file to which any log data belongs as a second log file when the log data does not contain a sensitive field;

[0035] The storage unit is used to determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

[0036] In the above device, optionally, the acquisition unit comprises:

[0037] A setting subunit, used to set the log collection order corresponding to each of the application systems;

[0038] A collection subunit, configured to sequentially execute the log collection process corresponding to each of the application systems based on the log collection sequence and the log configuration information corresponding to each of the application systems, so as to obtain each log file corresponding to each of the application systems based on the log collection process corresponding to each of the application systems;

[0039] The first execution subunit is used to determine the number of log files, the total file size, the file transfer flow and the transmission time corresponding to each log file to be collected in the application system based on the configuration information corresponding to the application system, and judge whether the log collection switch corresponding to the application system is in the on state; if so, judge whether the number of log files exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transmission time exceeds the preset duration threshold; if the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transmission time does not exceed the preset duration threshold, then collect each log file in the application system and end the log collection process corresponding to the application system; if the number of log files exceeds the preset number threshold, or the total file size exceeds the preset capacity threshold, or the file transfer flow exceeds the preset flow threshold, or the transmission time exceeds the preset duration threshold, then end the log collection process corresponding to the application system.

[0040] The above device may optionally further include:

[0041] The second execution subunit is used to obtain a preset application component lock when the log collection process corresponding to the application system starts to be executed; apply the application component lock to lock the log collection process corresponding to the application system; and release the application component lock when the log collection process corresponding to the application system ends.

[0042] In the above device, optionally, the desensitization unit comprises:

[0043] An extraction subunit, used to extract sensitive fields from the log data;

[0044] A first determining subunit, used to determine the data type corresponding to the sensitive field;

[0045] A second determination subunit is used to determine the security level corresponding to the data type and the desensitization method corresponding to the security level;

[0046] The desensitizing subunit is used to perform desensitization processing on the sensitive field based on the desensitization method.

[0047] In the above device, optionally, the storage unit comprises:

[0048] A third determining subunit is used to determine, based on the log configuration information, a CMBD tag corresponding to each of the first log file and the second log file set by the CMBD;

[0049] The storage subunit is used to determine, for each CMBD tag, whether the CMBD tag contains a storage identification number; the storage identification number is a cluster mark of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

[0050] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned log file processing method.

[0051] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the above-mentioned log file processing method.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] The present invention provides a log file processing method, comprising: when receiving log configuration information corresponding to each application system sent by a preset configuration management database CMBD, determining each preset data dimension; based on the log configuration information corresponding to each application system, checking whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, collecting each log file in the application system; reading log data in each log file, and checking whether each log data contains sensitive fields; when any log data contains sensitive fields, desensitizing the sensitive fields in the log data according to a preset desensitization rule, and setting the log file to which the desensitized log data belongs as a first log file; when any log data does not contain sensitive fields, setting the log file to which the log data belongs as a second log file; determining the storage strategy corresponding to each of the first log file and the second log file, and storing each of the first log file and the second log file according to the storage strategy corresponding to each of the first log file and the second log file, and completing the processing process of each log file in each of the application systems. By applying the method provided by the present invention, log files can be collected through various data dimensions, and the log files can be stored according to the storage strategy after being desensitized, thereby preventing the leakage of sensitive data in the log files. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0055] Figure 1 A method flow chart of a log file processing method provided by an embodiment of the present invention;

[0056] Figure 2 Another method flow chart of a log file processing method provided by an embodiment of the present invention;

[0057] Figure 3 A flow chart of another method of a log file processing method provided by an embodiment of the present invention;

[0058] Figure 4 Another method flow chart of a log file processing method provided by an embodiment of the present invention;

[0059] Figure 5 A device structure diagram of a log file processing device provided by an embodiment of the present invention;

[0060] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0063] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0064] The embodiment of the present invention provides a log file processing method, which can be applied to various system platforms, and its execution subject can be a processor of a computer terminal or various mobile devices. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0065] S101: When log configuration information corresponding to each application system sent by a preset configuration management database CMBD is received, each preset data dimension is determined.

[0066] Among them, each of the data dimensions includes quantity, size, flow and duration.

[0067] In the present invention, CMBD is a data management library for data management, which can realize the synchronous transmission of data between the processor and each application system. The log configuration information corresponding to each application system includes the log flag information, file name, log file path and log directory of each log file in the application system, and the log directory records the file name, file size, number of log files, etc. of each log file.

[0068] S102: Based on the log configuration information corresponding to each application system, check whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, collect each log file in the application system.

[0069] It is understandable that before collecting log files, it is necessary to check multiple data dimensions such as duration, flow, quantity, collection time, etc. of the log files to be collected to determine whether each log file in the application system meets each data dimension.

[0070] S103: Read the log data in each of the log files, and check whether each of the log data contains a sensitive field.

[0071] In the present invention, the log data is the field information and log content in the log file. The log data contains multiple log fields, and the log data of each collected log file is checked to see whether it contains sensitive fields. When any log data contains sensitive fields, S104 is executed; when any log data does not contain sensitive fields, S105 is executed.

[0072] It should be noted that sensitive fields may include user account, mobile phone number, bank card number, etc.

[0073] S104: Desensitizing the sensitive fields in the log data according to the preset desensitizing rules, and setting the log file to which the desensitized log data belongs as the first log file.

[0074] It is understandable that when the log data contains sensitive fields, desensitization processing is performed. Among them, the desensitization rules of the desensitization processing can be to replace the sensitive field with a field of other content according to the set desensitization algorithm, and when the field needs to be identified, the original field content is reversely calculated; or the sensitive field is covered by applying pre-set characters for desensitization processing, and the log file after desensitization processing is the first log file.

[0075] Specifically, for each log data containing sensitive fields, different sensitivity levels can be set and different sensitivity rules can be determined according to different sensitivity levels. For example, sensitive fields with high sensitivity levels are processed by applying sensitive algorithms; sensitive fields with low sensitivity levels are covered by set characters.

[0076] S105: Setting the log file to which the log data belongs as a second log file.

[0077] It is understandable that there is no log data with sensitive fields, and no desensitization processing is required, and the log file is set as the second log file.

[0078] S106: Determine the storage strategy corresponding to each of the first log files and the second log files, and parse and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

[0079] In the present invention, the storage strategy corresponding to each log file includes storage location, storage method and storage format, etc. The storage strategy corresponding to each first log file and the second log file is determined, and the log files are parsed and stored according to the storage strategy corresponding to each log file.

[0080] In the log file processing method provided by the embodiment of the present invention, when the log configuration information corresponding to each application system sent by CMBD is received, the currently set data dimensions are determined. Each data dimension can be changed, added or deleted according to business needs. Based on the log configuration information sent by CMBD, it is checked whether each log file in each application system meets each data dimension. Under the condition that each data dimension is met, each log file of the application system can be collected. The log data of each read log file is read, and it is checked whether each log data contains sensitive fields. If sensitive fields are contained, desensitization processing is performed according to the desensitization rules; if sensitive fields are not contained, desensitization processing is not required. Finally, the storage strategy corresponding to each log file is determined, and each log file is stored according to the storage strategy corresponding to each log file. The processing process of each log file in each application system is completed.

[0081] By applying the method provided in the embodiment of the present invention, log files can be collected through various data dimensions, and the log files can be stored according to the storage policy after being desensitized, thereby preventing the leakage of sensitive data in the log files.

[0082] In the method provided by the embodiment of the present invention, based on the content of S102 above, the log configuration information corresponding to each application system is used to check whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, each log file in the application system is collected, such as Figure 2 As shown, specifically including:

[0083] S201: Setting the log collection order corresponding to each of the application systems.

[0084] In the present invention, a log collection sequence corresponding to each application system is set, and each log file of each application system is collected in sequence.

[0085] It should be noted that a collection agent can be installed on the processor. The collection agent is a log collection device on the processor. The collection agent is connected to the CMDB and can obtain the log configuration information sent by the CMDB in real time. The CMDB sends each data dimension to the processor, and the processor collects data by installing the collection agent.

[0086] S202: Based on the log collection order and the log configuration information corresponding to each of the application systems, the log collection process corresponding to each of the application systems is sequentially executed to obtain the log files corresponding to each of the application systems based on the log collection process corresponding to each of the application systems.

[0087] Among them, reference Figure 3 For each of the application systems, the log collection process corresponding to the application system includes the following processes from S301 to S308.

[0088] S301: Based on the configuration information corresponding to the application system, determine the number of log files, the total file size, the file transfer flow, and the transfer time corresponding to each log file to be collected in the application system.

[0089] It should be noted that by reading the log directory in the log configuration information, the number of log files corresponding to each log file included in the application system, the size of each file, the flow rate for transmitting each log file, and the time for transmitting each log file are determined.

[0090] Among them, the number of log files described in S301 is the number of all log files currently contained in the application system, the total file size is the sum of the sizes of each log file in the application system, the file transfer traffic is the total traffic required to transfer all log files in the application system, and the transmission time is the total time required to transfer all log files in the application system.

[0091] S302: Determine whether the log collection switch corresponding to the application system is in an on state.

[0092] It is understandable that each application system can choose whether to turn on the log collection switch according to its own configuration. If the log collection switch of the application system is turned on, it indicates that each log file in the application system can be collected, and S303 is continued.

[0093] S303: Determine whether the number of log files exceeds a preset number threshold.

[0094] When the number of log files does not exceed the preset number threshold, S304 is executed; otherwise, S308 is executed.

[0095] S304: Determine whether the total file size exceeds a preset capacity threshold.

[0096] When the total file size does not exceed the preset capacity threshold, S305 is executed; otherwise, S308 is executed.

[0097] S305: Determine whether the file transfer traffic exceeds a preset traffic threshold.

[0098] When the file transfer flow rate does not exceed the preset flow rate threshold, S306 is executed; otherwise, S308 is executed.

[0099] S306: Determine whether the transmission time exceeds a preset duration threshold.

[0100] When the transmission time does not exceed the preset duration threshold, S307 is executed; otherwise, S308 is executed.

[0101] S307: Collect various log files in the application system.

[0102] It can be understood that after all the conditions from S302 to S306 are met, it is confirmed that the various log files of the application system can be collected.

[0103] S308: End the log collection process corresponding to the application system, and return to execute S303 to enter the log collection process corresponding to the next application system.

[0104] It should be noted that when the log collection process corresponding to the current application system is terminated, it is necessary to confirm whether there is a log collection process corresponding to the next application system. If so, the log collection process corresponding to the next application system is continued and the above S303 is returned. If not, the collection of all log files is terminated.

[0105] It is further explained that the execution process of steps S302 to S306 in the embodiment of the present invention is only a specific implementation process. In actual applications, the execution order of S302 to S306 is not limited.

[0106] In the log file processing method provided by the embodiment of the present invention, the log collection order corresponding to each application system is set, and each application system executes the log collection process according to the log collection order. Among them, the log collection process corresponding to each application system is: determine the number of log files, the total file size, the file transfer flow and the transmission time corresponding to each log file in the application system. First, determine whether the log collection switch corresponding to the application system is in the on state. If so, determine whether the number of log files in the application system exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transmission time exceeds the preset duration threshold. If all the above conditions are met, that is, the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transmission time does not exceed the preset duration threshold, then collect all the log files in the application system; otherwise, skip the application system and execute the log collection process corresponding to the next application system.

[0107] It should be noted that the preset quantity threshold, total file size, file transfer flow and transfer time all belong to various data dimensions.

[0108] The embodiment of the present invention limits the file collection conditions for each log file in the application system through various data dimensions, thereby avoiding damage to the parser and storage caused by collecting a large number of logs, and forming a current limiting protection.

[0109] Specifically, the log collection process for each application system also includes:

[0110] When the log collection process corresponding to the application system starts to execute, a preset application component lock is obtained;

[0111] Apply the application component lock to lock the log collection process corresponding to the application system;

[0112] When the log collection process corresponding to the application system is completed, the application component lock is released.

[0113] In the method provided by the embodiment of the present invention, when the log collection process corresponding to each application system is started, it is first checked whether the preset application component lock is in a released state. If so, the application component lock is acquired, and the application component lock is used to lock the log collection process that needs to be executed currently, and the application component lock is released when the current log collection process ends.

[0114] It should be noted that the application component lock is used to lock the execution process of the current log collection process to prevent the concurrency of multiple threads, which may lead to an excessive number of collected log files, and ensure that each log collection process can be executed from beginning to end by one thread.

[0115] In the method provided by the embodiment of the present invention, based on the content of S104, the process of desensitizing the sensitive fields in the log data according to the pre-set desensitization rules is as follows: Figure 4 As shown, it may specifically include:

[0116] S401: Extract sensitive fields from the log data.

[0117] S402: Determine the data type corresponding to the sensitive field.

[0118] It should be noted that the log files are first classified, and the sensitive fields in the classified log files correspond to the data types corresponding to the log files.

[0119] S403: Determine the security level corresponding to the data type and the desensitization method corresponding to the security level.

[0120] It should be noted that the processor will set a security equivalent label for each data type, and each security level corresponds to a desensitization method. After determining the data type corresponding to the sensitive field, the security level is determined according to the data type, and the desensitization method is determined according to the security level.

[0121] S404: Based on the desensitization method, desensitize the sensitive field.

[0122] It should be noted that after the sensitive fields are desensitized, the desensitized fields will replace the sensitive fields in the log data.

[0123] In the log file processing method provided by the embodiment of the present invention, the sensitive fields in the log data are extracted, and the field type corresponding to the sensitive field is determined, and then the corresponding security level and the desensitization method corresponding to the security level are determined. The sensitive fields are desensitized according to the desensitization method, and the desensitized fields cover the original sensitive fields in the log data. The embodiment of the present invention sets the corresponding desensitization level according to different data types, and desensitizes different desensitization levels according to different desensitization methods to ensure the security of the log file.

[0124] In the method provided by the embodiment of the present invention, based on the content of S106 above, determining the storage strategy corresponding to each of the first log files and the second log files, and parsing and storing each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, includes:

[0125] Based on the log configuration information, determine the CMBD tag corresponding to each of the first log file and the second log file set by the CMBD;

[0126] For each CMBD tag, determine whether the CMBD tag contains a storage identification number; the storage identification number is a cluster flag of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

[0127] It is understandable that, in the process of synchronously sending the configuration information corresponding to each application system, the CMDB configures the corresponding CMDB tag according to the data type of each log file in each application system, and the CMDB tag is used to mark the log cluster where the log file needs to be stored. When the CMDB sets the storage policy corresponding to any log file, the storage identification number corresponding to the log cluster storing the log file will be set in the CMDB tag of the log file. After the processor implements the desensitization processing of each log file, based on the log configuration information, it determines the CMDB tag of each log file and determines whether the CMDB tag of the log file has a storage identification number. If so, the log file is stored in the log cluster corresponding to the storage identification number; otherwise, any log cluster is randomly selected to store the log file.

[0128] By applying the method provided in the embodiment of the present invention, different storage strategies can be set in advance to implement the storage of different log files, and the log files can be quickly searched when they are subsequently applied.

[0129] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0130] and Figure 1 Corresponding to the method described above, the embodiment of the present invention further provides a log file processing device for Figure 1The specific implementation of the method in the embodiment of the present invention is that the log file processing device provided by the embodiment of the present invention can be applied to a computer terminal or various mobile devices, and its structural diagram is as follows Figure 5 As shown, specifically including:

[0131] The determination unit 501 is used to determine each preset data dimension when receiving log configuration information corresponding to each application system sent by the preset configuration management database CMBD;

[0132] The collection unit 502 is used to check whether each log file of each application system meets each data dimension based on the log configuration information corresponding to each application system, and collect each log file in the application system when it is determined that each log file of any application system meets each data dimension;

[0133] The checking unit 503 is used to read the log data in each of the log files and check whether each of the log data contains a sensitive field;

[0134] The desensitization unit 504 is used for desensitizing the sensitive fields in the log data according to a preset desensitization rule when any log data contains sensitive fields, and setting the log file to which the desensitized log data belongs as the first log file;

[0135] A setting unit 505, configured to set the log file to which the log data belongs as a second log file when any log data does not contain a sensitive field;

[0136] The storage unit 506 is used to determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

[0137] In the log file processing device provided by the embodiment of the present invention, when the log configuration information corresponding to each application system sent by CMBD is received, the currently set data dimensions are determined. Each data dimension can be changed, added or deleted according to business needs. Based on the log configuration information sent by CMBD, check whether each log file in each application system meets each data dimension. Under the condition that each data dimension is met, each log file of the application system can be collected. Read the log data of each read log file, and check whether each log data contains sensitive fields. If sensitive fields are contained, desensitization processing is performed according to the desensitization rules; if sensitive fields are not contained, desensitization processing is not required. Finally, determine the storage strategy corresponding to each log file, and store each log file according to the storage strategy corresponding to each log file. Complete the processing process of each log file in each application system.

[0138] By using the device provided by the embodiment of the present invention, log files can be collected through various data dimensions, and the log files can be stored according to the storage policy after being desensitized, thereby preventing the leakage of sensitive data in the log files.

[0139] In the device provided by the embodiment of the present invention, the collection unit includes:

[0140] A setting subunit, used to set the log collection order corresponding to each of the application systems;

[0141] A collection subunit, configured to sequentially execute the log collection process corresponding to each of the application systems based on the log collection sequence and the log configuration information corresponding to each of the application systems, so as to obtain each log file corresponding to each of the application systems based on the log collection process corresponding to each of the application systems;

[0142] The first execution subunit is used to determine the number of log files, the total file size, the file transfer flow and the transmission time corresponding to each log file to be collected in the application system based on the configuration information corresponding to the application system, and judge whether the log collection switch corresponding to the application system is in the on state; if so, judge whether the number of log files exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transmission time exceeds the preset duration threshold; if the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transmission time does not exceed the preset duration threshold, then collect each log file in the application system and end the log collection process corresponding to the application system; if the number of log files exceeds the preset number threshold, or the total file size exceeds the preset capacity threshold, or the file transfer flow exceeds the preset flow threshold, or the transmission time exceeds the preset duration threshold, then end the log collection process corresponding to the application system.

[0143] The device provided in the embodiment of the present invention further includes:

[0144] The second execution subunit is used to obtain a preset application component lock when the log collection process corresponding to the application system starts to be executed; apply the application component lock to lock the log collection process corresponding to the application system; and release the application component lock when the log collection process corresponding to the application system ends.

[0145] In the device provided in the embodiment of the present invention, the desensitization unit includes:

[0146] An extraction subunit, used to extract sensitive fields from the log data;

[0147] A first determining subunit, used to determine the data type corresponding to the sensitive field;

[0148] A second determination subunit is used to determine the security level corresponding to the data type and the desensitization method corresponding to the security level;

[0149] The desensitizing subunit is used to perform desensitization processing on the sensitive field based on the desensitization method.

[0150] In the device provided by the embodiment of the present invention, the storage unit includes:

[0151] A third determining subunit is used to determine, based on the log configuration information, a CMBD tag corresponding to each of the first log file and the second log file set by the CMBD;

[0152] The storage subunit is used to determine, for each CMBD tag, whether the CMBD tag contains a storage identification number; the storage identification number is a cluster mark of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

[0153] For the specific working process of each unit and sub-unit in the log file processing device disclosed in the above embodiment of the present invention, reference may be made to the corresponding content in the log file processing method disclosed in the above embodiment of the present invention, and will not be described in detail here.

[0154] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned log file processing method.

[0155] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 6 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to perform the following operations:

[0156] When receiving the log configuration information corresponding to each application system sent by the preset configuration management database CMBD, determining each preset data dimension;

[0157] Based on the log configuration information corresponding to each application system, check whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, collect each log file in the application system;

[0158] Read the log data in each of the log files, and check whether each of the log data contains sensitive fields;

[0159] When any log data contains sensitive fields, desensitize the sensitive fields in the log data according to the pre-set desensitization rules, and set the log file to which the desensitized log data belongs as the first log file;

[0160] When any log data does not contain a sensitive field, the log file to which the log data belongs is set as the second log file;

[0161] Determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

[0162] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0163] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0164] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0165] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log file processing method, characterized in that: include: When receiving the log configuration information corresponding to each application system sent by the preset configuration management database CMBD, determining each preset data dimension; The various data dimensions include a preset quantity threshold, a total file size, a file transfer flow, and a transfer time; Based on the log configuration information corresponding to each application system, check whether each log file of each application system meets each data dimension, and when it is determined that each log file of any application system meets each data dimension, collect each log file in the application system; wherein, if the number of log files of each log file of the application system does not exceed a preset number threshold, the total file size does not exceed a preset capacity threshold, the file transfer flow does not exceed a preset flow threshold, and the transfer time does not exceed a preset duration threshold; then it is determined that each log file of the application system meets each data dimension; Read the log data in each of the log files, and check whether each of the log data contains sensitive fields; When any log data contains sensitive fields, desensitize the sensitive fields in the log data according to the pre-set desensitization rules, and set the log file to which the desensitized log data belongs as the first log file; When any log data does not contain a sensitive field, the log file to which the log data belongs is set as the second log file; Determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

2. The method according to claim 1, characterized in that The step of checking whether each log file of each application system satisfies each data dimension based on the log configuration information corresponding to each application system, and collecting each log file in the application system when it is determined that each log file of any application system satisfies each data dimension, includes: Setting the log collection order corresponding to each of the application systems; Based on the log collection sequence and the log configuration information corresponding to each of the application systems, the log collection process corresponding to each of the application systems is sequentially executed to obtain each log file corresponding to each of the application systems based on the log collection process corresponding to each of the application systems; For each of the application systems, the log collection process corresponding to the application system includes: based on the configuration information corresponding to the application system, determining the number of log files, the total file size, the file transfer flow and the transfer time corresponding to each log file to be collected in the application system, and judging whether the log collection switch corresponding to the application system is in the on state; if so, judging whether the number of log files exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transfer time exceeds the preset duration threshold; if the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transfer time does not exceed the preset duration threshold, then collecting each log file in the application system and ending the log collection process corresponding to the application system; if the number of log files exceeds the preset number threshold, or the total file size exceeds the preset capacity threshold, or the file transfer flow exceeds the preset flow threshold, or the transmission time exceeds the preset duration threshold, then ending the log collection process corresponding to the application system.

3. The method according to claim 2, characterized in that The log collection process corresponding to the application system also includes: When the log collection process corresponding to the application system starts to execute, a preset application component lock is obtained; Apply the application component lock to lock the log collection process corresponding to the application system; When the log collection process corresponding to the application system is completed, the application component lock is released.

4. The method according to claim 1, characterized in that The desensitizing the sensitive fields in the log data according to the preset desensitization rules includes: Extracting sensitive fields from the log data; Determine the data type corresponding to the sensitive field; Determine the security level corresponding to the data type and the desensitization method corresponding to the security level; Based on the desensitization method, the sensitive field is desensitized.

5. The method according to claim 1, characterized in that: The determining of the storage strategy corresponding to each of the first log files and the second log files, and parsing and storing each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, includes: Based on the log configuration information, determine the CMBD tag corresponding to each of the first log file and the second log file set by the CMBD; For each CMBD tag, determine whether the CMBD tag contains a storage identification number; the storage identification number is a cluster flag of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain a storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

6. A log file processing device, characterized in that: include: A determination unit, configured to determine each preset data dimension when receiving log configuration information corresponding to each application system sent by a preset configuration management database CMBD; each data dimension includes a preset quantity threshold, a total file size, a file transfer flow, and a transfer time; A collection unit is used to check whether each log file of each application system meets each data dimension based on the log configuration information corresponding to each application system, and collect each log file in the application system when it is determined that each log file of any application system meets each data dimension; wherein, if the number of log files of each log file of the application system does not exceed a preset number threshold, the total file size does not exceed a preset capacity threshold, the file transfer flow does not exceed a preset flow threshold, and the transfer time does not exceed a preset duration threshold; then it is determined that each log file of the application system meets each data dimension; A checking unit, used for reading the log data in each of the log files, and checking whether each of the log data contains sensitive fields; A desensitization unit, for desensitizing the sensitive fields in the log data according to a preset desensitization rule when any log data contains sensitive fields, and setting the log file to which the desensitized log data belongs as the first log file; A setting unit, configured to set the log file to which any log data belongs as a second log file when any log data does not contain a sensitive field; The storage unit is used to determine the storage strategy corresponding to each of the first log files and the second log files, and store each of the first log files and the second log files according to the storage strategy corresponding to each of the first log files and the second log files, so as to complete the processing process of each log file in each of the application systems.

7. The device according to claim 6, characterized in that The acquisition unit comprises: A setting subunit, used to set the log collection order corresponding to each of the application systems; A collection subunit, configured to sequentially execute the log collection process corresponding to each of the application systems based on the log collection sequence and the log configuration information corresponding to each of the application systems, so as to obtain each log file corresponding to each of the application systems based on the log collection process corresponding to each of the application systems; The first execution subunit is used to determine the number of log files, the total file size, the file transfer flow and the transmission time corresponding to each log file to be collected in the application system based on the configuration information corresponding to the application system, and judge whether the log collection switch corresponding to the application system is in the on state; if so, judge whether the number of log files exceeds the preset number threshold, whether the total file size exceeds the preset capacity threshold, whether the file transfer flow exceeds the preset flow threshold, and whether the transmission time exceeds the preset duration threshold; if the number of log files does not exceed the preset number threshold, the total file size does not exceed the preset capacity threshold, the file transfer flow does not exceed the preset flow threshold, and the transmission time does not exceed the preset duration threshold, then collect each log file in the application system and end the log collection process corresponding to the application system; if the number of log files exceeds the preset number threshold, or the total file size exceeds the preset capacity threshold, or the file transfer flow exceeds the preset flow threshold, or the transmission time exceeds the preset duration threshold, then end the log collection process corresponding to the application system.

8. The device according to claim 7, characterized in that Also includes: A second execution subunit is used to obtain a preset application component lock when the log collection process corresponding to the application system starts to execute; The application component lock is applied to lock the log collection process corresponding to the application system; when the log collection process corresponding to the application system is completed, the application component lock is released.

9. The device according to claim 6, characterized in that The desensitization unit comprises: An extraction subunit, used to extract sensitive fields from the log data; A first determining subunit, used to determine the data type corresponding to the sensitive field; A second determination subunit is used to determine the security level corresponding to the data type and the desensitization method corresponding to the security level; The desensitizing subunit is used to perform desensitization processing on the sensitive field based on the desensitization method.

10. The device according to claim 6, characterized in that The storage unit comprises: A third determining subunit is used to determine, based on the log configuration information, a CMBD tag corresponding to each of the first log file and the second log file set by the CMBD; The storage subunit is used to determine, for each CMBD tag, whether the CMBD tag contains a storage identification number; the storage identification number is a cluster mark of a log cluster set by the CMBD to store the log file corresponding to the CMBD tag; when the CMBD tag contains the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to the log cluster corresponding to the CMBD tag; when the CMBD tag does not contain the storage identification number, obtain the log data in the log file corresponding to the CMBD tag, and store the log data in the log file corresponding to the CMBD tag to any log cluster.

Images