Log processing method and device, equipment and storage medium
By automatically processing log encryption, storage, decryption and writing processes, the problems of high security risks and high costs in manual migration are solved, and efficient and secure log migration is achieved.
Patent Information
- Application Number
- CN202510017472.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-06
- Publication Date
- 2025-05-06
AI Technical Summary
In the prior art, manual execution of the log migration process leads to problems of high security risks and high migration costs.
The log monitoring program automatically handles the encryption, storage, decryption and writing process of the original log to be transmitted through the log monitoring program. The original log to be transmitted is encrypted using the log encryption parameters, and automatically obtains and decrypts the latest encrypted log file when the target log path changes, and finally writes the decrypted log to the data table of the target database.
It significantly reduces human intervention, simplifies the migration process, ensures data security and privacy protection, and prevents the risk of data breaches or tampering.
Smart Images
Figure CN119945751A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a log processing method, device, equipment and storage medium. Background Art
[0002] The process of switching logs to a domestic database usually includes two main steps: preliminary preparation and data migration. First, in the preliminary preparation stage, it is necessary to evaluate the migration requirements, such as improving performance, reducing costs, or meeting compliance requirements, and evaluate the existing MySQL database in terms of data volume, data type, and table structure. Then, a complete backup is performed through MySQL backup tools (such as mysqldump) or third-party tools to ensure data security and recoverability. Next, in the data migration stage, select a suitable migration tool (such as DM Database Transfer System (DM DTS) of DM Database), configure the information of the source and target databases, and select the migration method (such as full or incremental migration). During the migration process, it is necessary to select migration objects (such as tables, views, stored procedures, etc.), execute tasks and monitor the progress of the migration, and handle any errors or warnings that occur. After the migration is completed, the integrity and accuracy of the data must be verified. The above steps are usually implemented manually, but this method will lead to problems such as high security risks and high migration costs.
[0003] Therefore, how to reduce the high security risks and high migration costs caused by manual operations is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0004] Based on the above problems, the present application provides a log processing method, device, equipment and storage medium, which can reduce the high security risks and high migration costs caused by manual operations, thereby realizing automated, efficient and secure log migration.
[0005] The embodiments of the present application disclose the following technical solutions:
[0006] A log processing method is applied to a log monitoring program, the method comprising:
[0007] In response to receiving the original log to be transmitted, obtaining a log encryption parameter;
[0008] Using the log encryption parameter to encrypt the original log to be transmitted to obtain an encrypted log file;
[0009] Storing the encrypted log file in a target log path;
[0010] When the target log path changes, the latest encrypted log file stored in the target log path is obtained, and the latest encrypted log file is decrypted using the log encryption parameter to obtain a decrypted log;
[0011] The decryption log is written into a data table of a target database.
[0012] In a possible implementation manner, writing the decryption log into a data table of a target database includes:
[0013] Identify the data storage location recorded in the decryption log;
[0014] Determine a data table in the target database corresponding to the data storage location, and obtain a target data table;
[0015] The decryption log is written into the target data table.
[0016] In a possible implementation, storing the encrypted log file in a target log path includes:
[0017] Identify whether a log file is stored in the target log path. If so, use the encrypted log file to overwrite the log file stored in the target log path; otherwise, directly store the encrypted log in the target log path.
[0018] In a possible implementation, the target log path changes, including:
[0019] The target log path changes from a state without log files to a state with log files; or,
[0020] The original log file in the target log path is overwritten by the new log file.
[0021] In a possible implementation, the method further includes:
[0022] Create the log encryption parameters and create a key file;
[0023] The log encryption parameters are stored in the key file.
[0024] In a possible implementation, the log encryption parameters include: a password and an initialization vector.
[0025] A log processing device, the device comprising:
[0026] an encryption parameter acquisition unit, in response to receiving the original log to be transmitted, for acquiring the log encryption parameter;
[0027] A log encryption unit, used to encrypt the original log to be transmitted using the log encryption parameter to obtain an encrypted log file;
[0028] A log file storage unit, used to store the encrypted log file in a target log path;
[0029] A log file acquisition unit, used for acquiring the latest encrypted log file stored in the target log path when the target log path changes;
[0030] A log decryption unit, used to decrypt the latest encrypted log file using the log encryption parameter to obtain a decrypted log;
[0031] The first log writing unit is used to write the decryption log into a data table of a target database.
[0032] In a possible implementation manner, writing the first log into the unit specifically includes:
[0033] An identification unit, used to identify a data storage location recorded in the decryption log;
[0034] A target data table determining unit, configured to determine a data table in the target database corresponding to the data storage location, and obtain a target data table;
[0035] The second log writing unit is used to write the decryption log into the target data table.
[0036] A log processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the log processing method described above is implemented.
[0037] A computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the log processing method as described above.
[0038] Compared with the prior art, this application has the following beneficial effects:
[0039] The present application provides a method, device, equipment and storage medium for processing logs. Specifically, when executing the method for processing logs provided by the embodiments of the present application, first, the log monitoring program can obtain log encryption parameters when receiving the original log to be transmitted. Then, the original log to be transmitted is encrypted using the log encryption parameters to obtain an encrypted log file, and the encrypted log file is stored in the target log path. When the log monitoring program detects that the target log path has changed, the latest encrypted log file stored in the target log path is obtained, and the latest encrypted log file is decrypted using the log encryption parameters to obtain a decrypted log. The decrypted log is then written to the data table of the target database. The present application greatly reduces human intervention and simplifies the migration process by automating the encryption, storage, decryption and writing of the original log to be transmitted. At the same time, encryption technology is applied during the log transmission process to ensure the security and privacy protection of data and prevent the risk of data leakage or tampering during the migration process. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 A flow chart of a log processing method provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of the structure of a log processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.
[0044] In the process of "switching logs to a domestic database", manually performing each step usually leads to problems such as high security risks and high migration costs for the following reasons:
[0045] High security risk: Manual migration involves database backup, configuration, migration and other operations. If there are no strict security measures, there may be a risk of data leakage or loss. In particular, sensitive information such as database credentials and keys required during the migration process may cause security risks if they are not properly protected or managed. In addition, some potential security vulnerabilities may be missed when manually handling errors and warning information.
[0046] High migration cost: Manually performing the entire migration process requires a lot of human intervention and technical input, especially when constant debugging, verification, and compatibility issues need to be resolved during the migration process, which significantly increases the time and labor costs. In addition, manual processing may lead to migration errors, requiring repeated operations, further increasing costs.
[0047] In summary, manual migration methods usually have problems such as low efficiency, many errors, high risks and high costs.
[0048] In order to solve this problem, a method, device, equipment and storage medium for processing logs are provided in an embodiment of the present application. First, the log monitoring program obtains the log encryption parameters when receiving the original log to be transmitted. Then, the original log is encrypted using the encryption parameters to generate an encrypted log file, and the encrypted log file is stored in the target log path. When a change in the target log path is detected, the log monitoring program obtains the latest encrypted log file from the path, and decrypts it using the same encryption parameters to obtain the decrypted log content. Finally, the decrypted log is written into the data table of the target database. In this way, the present application realizes the automated encryption, storage, decryption and writing process of the original log to be transmitted, significantly reduces human intervention, and simplifies the migration process. At the same time, the application of encryption technology ensures the security and privacy protection of the data, and effectively prevents the risk of data leakage or tampering that may occur during the migration process.
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0050] See also Figure 1 , which is a flow chart of a log processing method provided by an embodiment of the present application, and is applied to a log monitoring program (Log Monitor Program), such as Figure 1 As shown, the log processing method may include steps S101-S105:
[0051] S101: In response to receiving an original log to be transmitted, obtaining a log encryption parameter.
[0052] After the log monitoring program receives the log data to be transmitted, it first needs to extract the necessary parameters for encrypting the log from the relevant files. These parameters are used to ensure that the log data can be correctly and securely encrypted during subsequent processing. In other words, this is a step before the encryption operation, the purpose is to ensure that the encryption strategy used meets security requirements and avoid data leakage or tampering.
[0053] In a possible implementation, the log encryption parameters include: a password and an initialization vector (Initialization Vector, IV).
[0054] In a possible implementation, the method further includes:
[0055] Create the log encryption parameters and create a key file named logiv.txt. Then store the log encryption parameters in the key file.
[0056] First, create the key parameters for encrypting log files, namely the encryption key (password) and initialization vector (IV). These parameters are the core of the encryption operation, and they are used to ensure that the log data can be securely encrypted and restored to its original state when decrypted. Next, save these encryption parameters (keys and IV) into a file called logiv.txt, which will be used as a key file to store these important encryption information. The purpose of saving these parameters is to extract the corresponding key and IV from the file when encrypting and decrypting logs, so as to encrypt the log content or restore the log content before encryption. In this way, the key file provides the key information required for encryption and decryption.
[0057] It should be noted that in encryption, the password (Key) is the core secret information used to encrypt and decrypt data. Depending on the encryption algorithm, the password can have different functions and forms:
[0058] Symmetric encryption: The same cipher is used for encryption and decryption, such as the Advanced Encryption Standard (AES) and Data Encryption Standard (DES) algorithms;
[0059] Asymmetric encryption: Encryption and decryption use different ciphers (public key and private key), such as the Rivest-Shamir-Adleman (RSA) algorithm.
[0060] The purpose of the password:
[0061] In encryption, a password is used to convert plaintext data into ciphertext.
[0062] When decrypting, the cipher is used to convert the ciphertext back into plaintext.
[0063] The password is the core of the encrypted log monitoring program, ensuring the confidentiality of the data. The password needs to be kept secret and only authorized personnel can access it.
[0064] An initialization vector (IV) is a random number or random data used in the encryption process. It is used together with the password to enhance the security of encryption. IV is not used to encrypt the data itself, but to increase the unpredictability and randomness of the encryption process.
[0065] The role of IV is to ensure that even if the same password and the same plaintext are used, the result of each encryption (i.e. ciphertext) is different.
[0066] IV Features:
[0067] The IV is usually public, transmitted along with the ciphertext, and does not need to be kept secret.
[0068] The IV must be the same during encryption and decryption.
[0069] Each time encryption is performed, the IV should be randomly generated to ensure encryption diversity.
[0070] S102: Encrypt the original log to be transmitted using the log encryption parameter to obtain an encrypted log file.
[0071] In the process of encrypting logs using log encryption parameters such as passwords and initialization vectors (IVs), you first need to select a symmetric encryption algorithm such as AES to ensure the symmetry of encryption and decryption operations, and then use the key, IV, and Cipher Block Chaining (CBC) mode to encrypt through the encryption algorithm. The CBC mode can effectively handle block-level encryption of data. The encrypted content is usually saved in the form of a ciphertext file, and the IV is stored at the beginning of the file to ensure that the original data can be correctly restored when decrypted. In short, the combined use of keys and IVs ensures the confidentiality and security of log data, and only those who hold the correct key can decrypt and read the log content.
[0072] It should be noted that CBC mode is a common block encryption mode used to increase the security of encryption algorithms. It is usually used in conjunction with symmetric encryption algorithms (such as AES and DES) to ensure that each ciphertext block in the encryption process depends on the previous ciphertext block, thereby increasing the complexity of encryption and anti-attack capabilities.
[0073] For example: The original log to be transmitted:
[0074] "INSERT login_log(USER,timestamp)("admin","2025-01-023:4:5")".
[0075] The format of the encrypted log file is as follows:
[0076] “==ABCDEFD==”.
[0077] S103: storing the encrypted log file in a target log path.
[0078] It takes a long time to transfer log files directly to the database, especially when the log volume is large, the network bandwidth is limited, or the database is under great pressure to write. In this case, direct transmission will cause delays in the processing process, affecting the performance and response speed of the log monitoring program. In order to avoid the impact on the performance of the log monitoring program during direct transmission, the encrypted log files are temporarily stored locally (such as the target log path) for caching. This allows the log monitoring program to save the log files first, and then batch process or upload them to the database at a scheduled time, thereby reducing the impact on real-time performance.
[0079] In a possible implementation, storing the encrypted log file in a target log path includes:
[0080] Identify whether a log file is stored in the target log path. If so, use the encrypted log file to overwrite the log file stored in the target log path; otherwise, directly store the encrypted log in the target log path.
[0081] Specifically, to identify whether a log file is stored in the target log path: first, it is necessary to check whether a log file already exists in the target log path. If a log file already exists in the target path (for example, a file named "X" already exists in the log folder), the subsequent operations will be different.
[0082] If a log file exists, overwrite it with the encrypted log file:
[0083] If a log file already exists in the target log path, the new encrypted log file will directly overwrite the original log file. This operation is to ensure that only the latest log data is in the log path and that all log data is encrypted.
[0084] If the log file does not exist, directly store the encrypted log to the target log path:
[0085] If there is no existing log file in the target path, the encrypted log file is directly stored in the path. In this case, no overwriting operation is required because there is no file in the path.
[0086] S104: When the target log path changes, the latest encrypted log file stored in the target log path is obtained, and the latest encrypted log file is decrypted using the log encryption parameter to obtain a decrypted log.
[0087] When the storage path of the target log file changes, the log monitoring program will automatically locate the new path and obtain the latest encrypted log file, and then decrypt the file using the specified encryption parameters to obtain the plaintext log content.
[0088] In a possible implementation, the target log path changes, including:
[0089] The target log path changes from a state without log files to a state with log files; or,
[0090] The original log file in the target log path is overwritten by the new log file.
[0091] From the state without log files to the state with log files:
[0092] This situation refers to the situation where there was no log file in the target log path originally, but now a log file has been added. In other words, the target log path has changed from being empty or not having any log files to containing at least one log file.
[0093] The old log file is overwritten by the new log file:
[0094] This situation refers to the existing log file in the target log path being replaced by a new log file. In other words, the original log file or files are replaced by the new log file.
[0095] Both of these situations indicate that the content in the target log path has changed, which may be a new file or an existing file replaced. Log monitoring programs usually monitor these changes so that they can process new log files or the latest log files in a timely manner.
[0096] S105: Writing the decryption log into a data table of a target database.
[0097] Writing the decrypted log to the data table of the target database means that the decrypted log data (i.e., the decrypted log) will be stored in the specified database. Specifically, the decrypted log content will be inserted into a table in the database according to a certain structure or format for subsequent query, analysis, or backup. This process is usually used to convert encrypted log files into structured data that can be further processed, so that they can be stored and managed in the database.
[0098] In a possible implementation manner, writing the decryption log into a data table of a target database includes:
[0099] Identify the data storage location recorded in the decryption log, then determine the data table corresponding to the data storage location in the target database to obtain a target data table, and then write the decryption log into the target data table.
[0100] Specifically, first, the log monitoring program will analyze the log file of the decrypted log and identify the data storage location recorded therein. The data storage location may refer to a certain identifier or field contained in the log file, indicating which specific location or table in the database the data should be stored in. Next, the log monitoring program determines the corresponding data table in the target database based on this data storage location. For example, the target database may have multiple tables, each table is used to store different types of data, and the log monitoring program needs to find a table that matches the log data storage location. Finally, after determining the target data table, the log monitoring program writes the data in the decrypted log into this target data table, that is, stores the information in the log in the database for subsequent use or analysis.
[0101] In a possible implementation, the target database may be, but is not limited to, any domestic database.
[0102] Based on the contents of S101-S105, it can be seen that the log monitoring program receives the original log to be transmitted and extracts the log encryption parameters, then encrypts the log to be transmitted to generate an encrypted log file, and stores it in the target log path. Once a change in the target log path is detected, the program will automatically obtain the latest encrypted log file and decrypt it, and finally write the decrypted log into the target database data table. This application significantly reduces human intervention and simplifies the migration process by automating the encryption, storage, decryption and writing processes of the log. At the same time, the use of encryption technology ensures data security and privacy protection, effectively preventing the risk of data leakage or tampering.
[0103] See also Figure 2 , Figure 2 A schematic diagram of the structure of a log processing device provided in an embodiment of the present application. Figure 2 As shown, the log processing device includes:
[0104] The encryption parameter acquisition unit 201 is used to acquire the log encryption parameter in response to receiving the original log to be transmitted;
[0105] The log encryption unit 202 is used to encrypt the original log to be transmitted using the log encryption parameter to obtain an encrypted log file;
[0106] The log file storage unit 203 is used to store the encrypted log file in the target log path;
[0107] The log file acquisition unit 204 is used to acquire the latest encrypted log file stored in the target log path when the target log path changes;
[0108] The log decryption unit 205 is used to decrypt the latest encrypted log file using the log encryption parameter to obtain a decrypted log;
[0109] The first log writing unit 206 is used to write the decryption log into a data table of a target database.
[0110] In a possible implementation, the writing the first log into the unit 206 specifically includes:
[0111] An identification unit, used to identify a data storage location recorded in the decryption log;
[0112] A target data table determining unit, configured to determine a data table in the target database corresponding to the data storage location, and obtain a target data table;
[0113] The second log writing unit is used to write the decryption log into the target data table.
[0114] In a possible implementation, the log file storage unit 203 is specifically used for:
[0115] Identify whether a log file is stored in the target log path. If so, use the encrypted log file to overwrite the log file stored in the target log path; otherwise, directly store the encrypted log in the target log path.
[0116] In a possible implementation, the target log path changes, including:
[0117] The target log path changes from a state without log files to a state with log files; or,
[0118] The original log file in the target log path is overwritten by the new log file.
[0119] In a possible implementation manner, the device further includes:
[0120] A key file creation unit, used to create the log encryption parameters and create a key file;
[0121] The encryption parameter storage unit is used to store the log encryption parameter in the key file.
[0122] In a possible implementation, the log encryption parameters include: a password and an initialization vector.
[0123] In addition, an embodiment of the present application also provides a log processing device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the log processing method described above is implemented.
[0124] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the log processing method as described above.
[0125] The embodiment of the present application provides a log processing device. When receiving the original log to be transmitted, the encryption parameter acquisition unit 201 is used to obtain the log encryption parameter, and the log encryption unit 202 uses the log encryption parameter to encrypt the original log to be transmitted to obtain an encrypted log file. The log file storage unit 203 stores the encrypted log file in the target log path, and the log file acquisition unit 204 obtains the latest encrypted log file stored in the target log path when the log monitoring program detects that the target log path has changed. Then the log decryption unit 205 uses the log encryption parameter to decrypt the latest encrypted log file to obtain a decrypted log, so that the first log writing unit 206 can write the decrypted log into the data table of the target database. The present application automates the encryption, storage, decryption and writing operations of the original log to be transmitted, effectively reducing the need for manual intervention, thereby simplifying the entire migration process. At the same time, encryption technology is used in the log transmission process to ensure the security and privacy protection of data, and effectively prevent the risk of data leakage or tampering during the migration process.
[0126] The above is a detailed introduction to a log processing method, device, equipment and storage medium provided by the present application. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.
[0127] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0128] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
Claims
1. A log processing method, characterized in that: Applied to a log monitoring program, the method comprises: In response to receiving the original log to be transmitted, obtaining a log encryption parameter; Using the log encryption parameter to encrypt the original log to be transmitted to obtain an encrypted log file; Storing the encrypted log file in a target log path; When the target log path changes, the latest encrypted log file stored in the target log path is obtained, and the latest encrypted log file is decrypted using the log encryption parameter to obtain a decrypted log; The decryption log is written into a data table of a target database.
2. The method according to claim 1, characterized in that Writing the decryption log into a data table of a target database includes: Identify the data storage location recorded in the decryption log; Determine a data table in the target database corresponding to the data storage location, and obtain a target data table; The decryption log is written into the target data table.
3. The method according to claim 1, characterized in that The step of storing the encrypted log file in a target log path includes: Identify whether a log file is stored in the target log path. If so, use the encrypted log file to overwrite the log file stored in the target log path; otherwise, directly store the encrypted log in the target log path.
4. The method according to claim 1, characterized in that: The target log path changes, including: The target log path changes from a state without log files to a state with log files; or, The original log file in the target log path is overwritten by the new log file.
5. The method according to claim 1, characterized in that The method further comprises: Create the log encryption parameters and create a key file; The log encryption parameters are stored in the key file.
6. The method according to claim 1 or 5, characterized in that: The log encryption parameters include: a password and an initialization vector.
7. A log processing device, characterized in that: The device comprises: An encryption parameter acquisition unit, in response to receiving the original log to be transmitted, is used to acquire the log encryption parameter; A log encryption unit, used to encrypt the original log to be transmitted using the log encryption parameter to obtain an encrypted log file; A log file storage unit, used to store the encrypted log file in a target log path; A log file acquisition unit, used for acquiring the latest encrypted log file stored in the target log path when the target log path changes; A log decryption unit, used to decrypt the latest encrypted log file using the log encryption parameter to obtain a decrypted log; The first log writing unit is used to write the decryption log into a data table of a target database.
8. The device according to claim 7, characterized in that The step of writing the first log into the unit specifically includes: An identification unit, used to identify a data storage location recorded in the decryption log; A target data table determining unit, configured to determine a data table in the target database corresponding to the data storage location, and obtain a target data table; The second log writing unit is used to write the decryption log into the target data table.
9. A log processing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the log processing method according to any one of claims 1 to 6 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the log processing method according to any one of claims 1 to 6.