Method for automatically generating operation logs on the lake-warehouse integrated platform

Abstract

The present invention relates to the technical field of data warehouses, and in particular to a method for automatically generating operation logs on a lake-warehouse integrated platform, the method comprising: capturing user operation behaviors in real time, extracting operation parameters in the operation behaviors, generating target tags based on preset tag rules and operation parameters; associating the target tags with the timestamps of the operation behaviors to form initial operation logs; identifying sensitive information in the initial operation logs, analyzing the sensitive information, encrypting the initial operation logs based on the analysis results, and obtaining encrypted logs; storing the encrypted logs based on the number of storage nodes and the analysis results of the sensitive information, and obtaining initial storage logs; real-time monitoring of access records of the initial storage logs, adjusting the initial storage logs according to the monitoring results of the access records, and updating the storage of the initial storage logs according to the adjustment results, and obtaining updated storage logs. The present invention improves the accuracy of the operation log generation results.

Description

Technical Field

[0001] The present invention relates to the technical field of data warehouses, and in particular to a method for automatically generating operation logs on a lake-warehouse integrated platform. Background Art

[0002] With the development of big data technology, the Lake-Warehouse Integration Platform, as a new data platform that integrates the advantages of data lakes and data warehouses, has been widely used in enterprise-level data management and analysis. During the operation and maintenance of the Lake-Warehouse Integration Platform, detailed operation logs are usually required to monitor and analyze the operating status of the system.

[0003] The patent document with Chinese patent application number CN112199335A discloses a method for generating an operation log, comprising the following steps: Step 1: obtaining operation log annotation information, wherein the operation log annotation information includes various fields necessary for generating an operation log; Step 2: obtaining the language type used by the user to perform operations on the front end; Step 3: converting the fields in step 1 according to the language type of step 2; Step 4: generating an operation log with the same language type as step 2 according to the operation log generation logic for the display value of the field after the language conversion in step 3; wherein the fields are pre-configured for different languages ​​to obtain display values ​​for different languages.

[0004] The prior art does not mention the processing of sensitive information. If it is not properly processed, the sensitive information in the operation log may be tampered with by an attacker, thereby making the operation log generated inaccurate. Summary of the invention

[0005] To this end, the present invention provides a method for automatically generating operation logs for a lake-warehouse integrated platform, which can solve the problem of inaccurate operation log generation.

[0006] To achieve the above object, the present invention provides a method for automatically generating an operation log for a lake-warehouse integrated platform, the method comprising:

[0007] Capture user operation behaviors in real time, extract operation parameters in the operation behaviors, and generate target tags based on preset tag rules and operation parameters;

[0008] Associating the target tag with the timestamp of the operation behavior to form an initial operation log;

[0009] Identify sensitive information in the initial operation log, analyze the sensitive information, encrypt the initial operation log based on the analysis result, and obtain an encrypted log;

[0010] Determine the total number of storage nodes based on the physical architecture and storage capacity of the lake-warehouse integrated platform, store the encrypted logs based on the analysis results of the total number and sensitive information, and obtain the initial storage logs;

[0011] The access record of the initial storage log is monitored in real time, and the initial storage log is adjusted according to the monitoring result of the access record, so as to update the storage of the initial storage log according to the adjustment result, and obtain the updated storage log.

[0012] Furthermore, the step of generating a target tag based on preset tag rules and operation parameters includes:

[0013] Determine operation parameters based on the operation behavior, wherein the operation parameters include operation type, operation time and operation object;

[0014] Setting an initial label based on the operation type, the operation time, the operation object and a preset label rule;

[0015] The uniqueness and accuracy of the initial label are verified, and the initial label is adjusted based on the verification result to obtain the target label.

[0016] Furthermore, the step of verifying the uniqueness and accuracy of the initial tag includes:

[0017] Initialize an empty tag set, add a number of the initial tags to the tag set according to the generation order of the initial tags, compare the initial tags with the initial tags in the tag set, and obtain the tag comparison result. If there is no identical tag in the tag comparison result, the initial tag is unique.

[0018] The actual lengths corresponding to the initial labels are compared with the preset lengths, the actual formats corresponding to the initial labels are compared with the preset formats, and the accuracy of the initial labels is determined based on the length comparison results and the format comparison results.

[0019] Furthermore, the step of adjusting the initial label based on the verification result includes:

[0020] Determine the number of repetitions of the repeated initial labels based on the label comparison result, and obtain the number of repetitions;

[0021] Obtaining the number of inaccurate initial labels based on the determination result of the accuracy of the initial labels, obtaining an inaccurate number;

[0022] Analyze the initial labels corresponding to the repeated quantity and the initial labels corresponding to the inaccurate quantity, and determine the final quantity according to the analysis results;

[0023] The final number is compared with the preset number. If the final number is less than the preset number, the single initial label is readjusted. If the final number is greater than or equal to the preset number, the preset label rule is adjusted.

[0024] Furthermore, the step of associating the target tag with the timestamp of the operation behavior includes:

[0025] Encoding the target tag according to a preset encoding rule to form a target code;

[0026] Obtaining a timestamp of the operation behavior, and converting the timestamp into a time code;

[0027] Combining the target code with the time code to form an operation log code;

[0028] The initial operation log is marked based on the operation log code to form a marked initial operation log.

[0029] Furthermore, the step of identifying sensitive information in the initial operation log includes:

[0030] Scanning the initial operation log based on a preset sensitive word library to identify whether the initial operation log contains preset sensitive information in the preset sensitive word library;

[0031] If the initial operation log contains the preset sensitive information, marking the information words corresponding to the preset sensitive information in the operation log to obtain first marked sensitive information;

[0032] If the initial operation log does not contain the preset sensitive information, performing semantic analysis on the initial operation log, obtaining a semantic analysis result, and determining whether the initial operation log contains sensitive information based on the semantic analysis result;

[0033] If the semantic analysis result indicates that the initial operation log contains sensitive information, the sensitive information in the initial operation log is marked to obtain second marked sensitive information.

[0034] Furthermore, the step of analyzing the sensitive information includes:

[0035] Determining a first sensitivity level based on a comparison result of an actual proportion of the first-marked sensitive information in the initial operation log and a preset proportion;

[0036] Determining a second sensitivity level based on a comparison result of an actual proportion of the second-marked sensitive information in the operation log and a preset proportion;

[0037] A final sensitivity level is determined based on the first sensitivity level and the second sensitivity level.

[0038] Furthermore, the step of encrypting the initial operation log based on the analysis result includes:

[0039] When the final sensitivity level is a high sensitivity level, performing data desensitization processing on the first sensitive information or the second sensitive information, encrypting the initial operation log after the data desensitization processing by using a first encryption algorithm, and obtaining an encrypted log;

[0040] When the final sensitivity level is a low sensitivity level, the initial operation log is encrypted using a second encryption algorithm to obtain an encrypted log.

[0041] Furthermore, the step of storing the encrypted log based on the analysis result of the total quantity and sensitive information includes:

[0042] Determine the total number of storage nodes;

[0043] Determining a storage location for the encrypted log based on the final sensitivity level and the total number;

[0044] According to the storage location, the encrypted log is stored in a corresponding storage node to obtain an initial storage log.

[0045] Furthermore, the step of adjusting the initial storage log according to the monitoring result of the access record includes:

[0046] Identifying an access timestamp in the access record;

[0047] determining a frequency of access based on a number of access timestamps;

[0048] Comparing the access frequency with a preset frequency, and adjusting the final sensitivity level corresponding to the initial storage log based on the comparison result to obtain an adjusted sensitivity level;

[0049] The initial storage log is adjusted based on the adjustment sensitivity level, and the updated storage log is obtained.

[0050] Compared with the prior art, the beneficial effect of the present invention is that by capturing user operation behaviors and extracting operation parameters in real time, the real-time and accuracy of operation logs can be ensured, which is helpful to timely discover potential security issues and quickly take corresponding processing measures. The target label is generated based on the preset label rules and operation parameters, and the label is associated with the timestamp of the operation behavior to form an initial operation log, so that the log is easier to manage and query, and the availability and readability of the log are improved. By identifying sensitive information in the initial operation log, analyzing and encrypting it, user privacy and system security can be effectively protected. The encryption process ensures the security of sensitive information during storage and transmission, and prevents the risks of data leakage and unauthorized access. The encrypted log is stored based on the number of storage nodes and the analysis results of sensitive information, and intelligent storage management can be realized. According to the load of the storage node and the level of sensitive information, storage resources are reasonably allocated, and storage efficiency and security are improved. By real-time monitoring of the access records of the initial storage log and adjusting the storage strategy according to the monitoring results, dynamic adjustment and optimization of the storage log is realized, potential security issues or performance bottlenecks are discovered and solved in time, the stability and reliability of the system are improved, and the integrity and consistency of the storage log are guaranteed.

[0051] In particular, by initializing an empty tag set and adding the generated initial tags to the set one by one, and comparing them with the existing tags in the set, the uniqueness of each initial tag is ensured, confusion and errors caused by repeated tags are avoided, and the accuracy and reliability of log management are improved. By comparing the actual length and format of the initial tag with the preset length and format, the accuracy and standardization of the tag are ensured, which helps to identify and correct potential errors or inconsistencies, thereby improving the quality and availability of log data. When it is found that the actual length or format does not match the preset, it is determined that the initial tag is inaccurate, so that timely measures are taken to correct it, which helps to maintain the integrity and consistency of the data and ensure that the log data can accurately reflect the behavior and operations of the system or user. Through a strict tag verification process, it can be ensured that only tags that meet the specifications and requirements can be added to the system, which helps to prevent the injection and attack of malicious tags and improve the security and stability of the system.

[0052] In particular, by verifying the uniqueness and accuracy of the initial labels, identifying duplicate labels and inaccurate labels, and making corresponding adjustments, it helps to reduce data confusion and misleading caused by duplicate and erroneous labels, and improve the quality and accuracy of log data. Analyzing the initial labels corresponding to the number of duplicates and inaccurate numbers and determining the final number based on the analysis results helps to understand the problems and bottlenecks in the label generation process. Based on the comparison result between the final number and the preset number, it is decided whether to readjust the single initial label or adjust the preset label rules, thereby optimizing the label generation and management process. When the final number is less than the preset number, only the single initial label is readjusted, avoiding large-scale label modification work and improving the operating efficiency of the system. When the final number is greater than or equal to the preset number, the preset label rules are adjusted to fundamentally solve the problems in the label generation process and improve the accuracy and efficiency of subsequent label generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flow chart of a method for automatically generating an operation log on a lake-warehouse integrated platform provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a process for forming an initial operation log in a method for automatically generating an operation log on a lake-warehouse integrated platform provided in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a process for identifying sensitive information in a method for automatically generating operation logs on a lake-warehouse integrated platform provided in an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of the process of obtaining storage logs in the method for automatically generating operation logs on the lake-warehouse integrated platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0059] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present invention.

[0060] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] See also Figure 1 As shown, an embodiment of the present invention provides a method for automatically generating an operation log on a lake-warehouse integrated platform, the method comprising:

[0062] Step S100, capturing user operation behaviors in real time, extracting operation parameters in the operation behaviors, and generating target tags based on preset tag rules and operation parameters;

[0063] Step S200, forming an initial operation log based on associating the target tag with the timestamp of the operation behavior;

[0064] Step S300, identifying sensitive information in the initial operation log, analyzing the sensitive information, encrypting the initial operation log based on the analysis result, and obtaining an encrypted log;

[0065] Step S400, determining the total number of storage nodes based on the physical architecture and storage capacity of the lake-warehouse integrated platform, storing the encrypted log based on the analysis results of the total number and sensitive information, and obtaining the initial storage log;

[0066] Step S500: monitor the access records of the initial storage log in real time, adjust the initial storage log according to the monitoring result of the access records, update the storage of the initial storage log according to the adjustment result, and obtain the updated storage log.

[0067] Specifically, the embodiment of the present invention can ensure the real-time and accuracy of the operation log by capturing user operation behavior and extracting operation parameters in real time, which is helpful to timely discover potential security issues and quickly take corresponding processing measures. The target label is generated based on the preset label rules and operation parameters, and the label is associated with the timestamp of the operation behavior to form an initial operation log, so that the log is easier to manage and query, and the availability and readability of the log are improved. By identifying sensitive information in the initial operation log, analyzing and encrypting it, user privacy and system security can be effectively protected. The encryption process ensures the security of sensitive information during storage and transmission, and prevents the risks of data leakage and unauthorized access. The encrypted log is stored based on the number of storage nodes and the analysis results of sensitive information, and intelligent storage management can be realized. According to the load of the storage node and the level of sensitive information, storage resources are reasonably allocated, and storage efficiency and security are improved. By real-time monitoring of the access records of the initial storage log and adjusting the storage strategy according to the monitoring results, dynamic adjustment and optimization of the storage log is realized, potential security issues or performance bottlenecks are discovered and resolved in time, the stability and reliability of the system are improved, and the integrity and consistency of the storage log are guaranteed.

[0068] Specifically, the step of generating a target tag based on preset tag rules and operation parameters includes:

[0069] Determine operation parameters based on the operation behavior, wherein the operation parameters include operation type, operation time and operation object;

[0070] Setting an initial label based on the operation type, the operation time, the operation object and a preset label rule;

[0071] The uniqueness and accuracy of the initial label are verified, and the initial label is adjusted based on the verification result to obtain the target label.

[0072] Specifically, the operation types described in the embodiment of the present invention include read, write, update, and delete, the operation time is the operation timestamp, and the operation object is a data table, a data field, or a file;

[0073] The preset labeling rule is that the operation type, operation time and operation object can all be encoded as specific numbers or digital strings. For example, the operation type can be represented by a unique digital code (such as read = 1, write = 2), the operation time can be converted into a timestamp or a specific date code, and the operation object can be encoded according to its type, for example, operation type: read (encoded as 1), operation time: July 6, 2023 10:30 (encoded as a timestamp in a specific format, such as 1678013400), operation object: user data table (encoded as 1001), and these codes are combined into a digital label according to the preset labeling rule, such as "1_1678013400_1001".

[0074] Specifically, the embodiment of the present invention, through preset label rules, uniformly encodes the operation type, operation time and operation object into specific numbers or digital strings, thereby achieving standardization and consistency of operation logs, improving the readability and comprehensibility of the logs, facilitating subsequent log analysis and processing, and verifying the uniqueness and accuracy of the target label to provide an accurate data basis for subsequent target label-based queries, thereby improving the efficiency of system maintenance and management.

[0075] Specifically, the step of verifying the uniqueness and accuracy of the initial tag includes:

[0076] Initialize an empty tag set, add a number of the initial tags to the tag set according to the generation order of the initial tags, compare the initial tags with the initial tags in the tag set, and obtain the tag comparison result. If there is no identical tag in the tag comparison result, the initial tag is unique.

[0077] The actual lengths corresponding to the initial labels are compared with the preset lengths, the actual formats corresponding to the initial labels are compared with the preset formats, and the accuracy of the initial labels is determined based on the length comparison results and the format comparison results.

[0078] Specifically, the preset length in the embodiment of the present invention is the standard length of a label generated based on a preset label rule, and the preset format is the standard format based on the preset label rule;

[0079] Specifically, the step of determining the accuracy of the initial label based on the length comparison result and the format comparison result in the embodiment of the present invention includes:

[0080] If the actual length is consistent with the preset length, and the actual format is consistent with the preset format, then it is determined that the initial label is accurate;

[0081] If the actual length is inconsistent with the preset length, or the actual format is inconsistent with the preset format, it is determined that the initial label is inaccurate.

[0082] Specifically, the step of comparing the actual format with the preset format in the embodiment of the present invention includes:

[0083] Extracting special characters from the actual label;

[0084] Determine the arrangement order and the position of the special characters in the initial label, compare the arrangement order and the position with the corresponding preset arrangement order and preset position, and obtain a comparison result.

[0085] Specifically, the embodiment of the present invention initializes an empty tag set, adds the generated initial tags to the set one by one, and compares them with the existing tags in the set to ensure the uniqueness of each initial tag, avoids confusion and errors caused by duplication of tags, and improves the accuracy and reliability of log management. By comparing the actual length and format of the initial tag with the preset length and format, the accuracy and standardization of the tag are ensured, which helps to identify and correct potential errors or inconsistencies, thereby improving the quality and availability of log data. When it is found that the actual length or format does not match the preset, it is determined that the initial tag is inaccurate, so that timely measures are taken to correct it, which helps to maintain the integrity and consistency of the data and ensure that the log data can accurately reflect the behavior and operations of the system or user. Through a strict tag verification process, it can be ensured that only tags that meet the specifications and requirements can be added to the system, which helps to prevent the injection and attack of malicious tags and improve the security and stability of the system.

[0086] Specifically, the step of adjusting the initial label based on the verification result includes:

[0087] Determine the number of repetitions of the repeated initial labels based on the label comparison result, and obtain the number of repetitions;

[0088] Obtaining the number of inaccurate initial labels based on the determination result of the accuracy of the initial labels, obtaining an inaccurate number;

[0089] Analyze the initial labels corresponding to the repeated quantity and the initial labels corresponding to the inaccurate quantity, and determine the final quantity according to the analysis results;

[0090] The final number is compared with the preset number. If the final number is less than the preset number, the single initial label is readjusted. If the final number is greater than or equal to the preset number, the preset label rule is adjusted.

[0091] Specifically, the preset number in the embodiment of the present invention is 1 / 3 of the total number of the initial tags.

[0092] Specifically, the step of analyzing the initial labels corresponding to the repeated number and the inaccurate number to determine the final number in the embodiment of the present invention includes:

[0093] Compare the initial labels corresponding to the repeated number with the initial labels corresponding to the inaccurate number, and determine whether the repeated initial labels are included according to the comparison result;

[0094] The repeated initial labels are merged and the merged quantities are added up to obtain the final quantity.

[0095] Specifically, the embodiments of the present invention verify the uniqueness and accuracy of the initial labels, identify duplicate labels and inaccurate labels, and make corresponding adjustments, which helps to reduce data confusion and misleading caused by duplicate and erroneous labels, improve the quality and accuracy of log data, analyze the initial labels corresponding to the duplicate and inaccurate numbers, and determine the final number based on the analysis results, which helps to understand the problems and bottlenecks in the label generation process. According to the comparison result between the final number and the preset number, it is decided whether to readjust the single initial label or adjust the preset label rules, thereby optimizing the label generation and management process. When the final number is less than the preset number, only the single initial label is readjusted, avoiding large-scale label modification work and improving the operating efficiency of the system. When the final number is greater than or equal to the preset number, the preset label rules are adjusted to fundamentally solve the problems in the label generation process and improve the accuracy and efficiency of subsequent label generation.

[0096] See also Figure 2 As shown, the step of associating the target tag with the timestamp of the operation behavior includes:

[0097] Step S210, encoding the target tag according to a preset encoding rule to form a target code;

[0098] Step S220, obtaining a timestamp of the operation behavior, and converting the timestamp into a time code;

[0099] Step S230, combining the target code with the time code to form an operation log code;

[0100] Step S240: marking the initial operation log based on the operation log code to form a marked initial operation log.

[0101] Specifically, the preset encoding rule in the embodiment of the present invention may be a simple hash algorithm or Base64 encoding, and the time encoding is a time digital string generated based on the timestamp;

[0102] Specifically, the step of combining the target code with the time code to form the operation log code in the embodiment of the present invention includes:

[0103] Concatenating the target code with the time code to form a concatenated string;

[0104] A hash operation is performed on the concatenated character string to obtain a hash value, and the hash value is used as the operation log code.

[0105] Specifically, the embodiment of the present invention combines the target tag code with the timestamp code to generate a unique operation log code, ensuring that each operation log has a unique identifier, facilitating subsequent query, tracking and analysis, and further processing the code using preset coding rules to hide part of the original data while retaining its uniqueness, thereby increasing data security and reducing the risk of data leakage. Since each operation log has a time code, it helps to quickly locate the relevant operation log when a problem occurs, thereby improving the efficiency of troubleshooting. By combining the tag and timestamp to form an operation log code, the storage and retrieval process of the log is simplified, which helps to reduce the query burden of the database and improve the overall performance of the system.

[0106] See also Figure 3 As shown, the step of identifying sensitive information in the initial operation log includes

[0107] Step S310, scanning the initial operation log based on a preset sensitive word library to identify whether the initial operation log contains preset sensitive information in the preset sensitive word library;

[0108] Step S320: if the initial operation log contains the preset sensitive information, mark the information words corresponding to the preset sensitive information in the operation log to obtain first marked sensitive information;

[0109] Step S330: If the initial operation log does not contain the preset sensitive information, semantic analysis is performed on the initial operation log to obtain a semantic analysis result, and based on the semantic analysis result, it is determined whether the initial operation log contains sensitive information;

[0110] Step S340: If the semantic analysis result indicates that the initial operation log contains sensitive information, the sensitive information in the initial operation log is marked to obtain second marked sensitive information.

[0111] Specifically, the preset sensitive word library described in the embodiment of the present invention is established based on a number of historical sensitive information in a number of historical operation logs.

[0112] Specifically, the step of performing semantic analysis on the initial operation log in the embodiment of the present invention includes:

[0113] Using a natural language processing model to perform word segmentation processing on the initial operation log to form a word segmentation result;

[0114] Constructing a semantic vector according to the word segmentation result, and comparing the semantic vector with a preset sensitive semantic vector to obtain a comparison result;

[0115] If the comparison result shows that the similarity between the semantic vector and the preset sensitive semantic vector exceeds a preset threshold, it is determined that the initial operation log contains sensitive information;

[0116] If the comparison result shows that the similarity between the semantic vector and the preset sensitive semantic vector does not exceed a preset threshold, it is determined that the initial operation log does not contain sensitive information.

[0117] Specifically, the preset sensitive semantic vector in the embodiment of the present invention is a semantic vector pre-set based on historical data and used to describe sensitive information;

[0118] The preset threshold is 90%.

[0119] Specifically, the embodiment of the present invention scans the initial operation log through a preset sensitive word library, quickly identifies the log containing preset sensitive information, improves the accuracy of identification, marks the identified sensitive information, makes it easier to identify and manage this information in subsequent processing, and helps to quickly locate sensitive data in the log management and auditing process. For those logs that do not directly contain preset sensitive words but may contain sensitive information, the sensitive information is further identified through semantic analysis, thereby reducing the underreporting of potential sensitive information and improving the accuracy of sensitive information identification results.

[0120] Specifically, the steps of analyzing sensitive information include:

[0121] Determining a first sensitivity level based on a comparison result of an actual proportion of the first-marked sensitive information in the initial operation log and a preset proportion;

[0122] Determining a second sensitivity level based on a comparison result of an actual proportion of the second-marked sensitive information in the operation log and a preset proportion;

[0123] A final sensitivity level is determined based on the first sensitivity level and the second sensitivity level.

[0124] Specifically, the preset ratio in the embodiment of the present invention is 1 / 3 of the total character length of the initial operation log;

[0125] The step of determining the first sensitivity level based on a comparison result between an actual proportion of the first-marked sensitive information in the initial operation log and a preset proportion comprises:

[0126] If the actual proportion of the first-marked sensitive information in the initial operation log exceeds the preset proportion, determining that the first sensitivity level is a high sensitivity level;

[0127] If the actual proportion of the first-marked sensitive information in the initial operation log does not exceed the preset proportion, determining that the first sensitivity level is a low sensitivity level;

[0128] The step of determining the second sensitivity level based on a comparison result between an actual proportion of the second-marked sensitive information in the operation log and a preset proportion comprises:

[0129] If the actual proportion of the second-marked sensitive information in the initial operation log exceeds the preset proportion, determining that the second sensitivity level is a high sensitivity level;

[0130] If the actual proportion of the second-marked sensitive information in the initial operation log does not exceed the preset proportion, determining that the second sensitivity level is a low sensitivity level;

[0131] The step of determining a final sensitivity level based on the first sensitivity level and the second sensitivity level comprises:

[0132] If at least one of the first sensitivity level and the second sensitivity level is a high sensitivity level, determining the final sensitivity level to be a high sensitivity level;

[0133] If the first sensitivity level and the second sensitivity level are both low sensitivity levels, the final sensitivity level is determined to be the low sensitivity level.

[0134] Specifically, the embodiments of the present invention compare the actual proportion of sensitive information in the log with the preset proportion to quantitatively evaluate the sensitivity of the log, so that the determination of the sensitivity level is more objective and accurate. By analyzing the sensitive information and determining the sensitivity level, those logs containing highly sensitive information can be more accurately identified, which helps to perform different storage modes for operation logs, thereby improving data security.

[0135] Specifically, the step of encrypting the initial operation log based on the analysis result includes:

[0136] When the final sensitivity level is a high sensitivity level, performing data desensitization processing on the first sensitive information or the second sensitive information, encrypting the initial operation log after the data desensitization processing by using a first encryption algorithm, and obtaining an encrypted log;

[0137] When the final sensitivity level is a low sensitivity level, the initial operation log is encrypted using a second encryption algorithm to obtain an encrypted log.

[0138] Specifically, in the embodiment of the present invention, the first encryption algorithm is AES-256 bit encryption, and the second encryption algorithm is AES-128 bit encryption;

[0139] The data desensitization process is to replace sensitive information with fictitious or masked data.

[0140] Specifically, in the embodiment of the present invention, when the final sensitivity level is high, AES-256-bit encryption is used to encrypt the desensitized log, ensuring the high security of highly sensitive data during transmission and storage. AES-256-bit encryption, as a strong encryption algorithm currently widely used, provides powerful security protection capabilities. When the final sensitivity level is low, AES-128-bit encryption is used to protect low-sensitivity data from being easily leaked. At a high sensitivity level, sensitive information is desensitized, that is, sensitive information is replaced with fictitious or masked data, which effectively prevents direct leakage of sensitive information and protects the security of stored logs. According to different sensitivity levels, encryption algorithms of different strengths are used to ensure the security of highly sensitive data and avoid the use of overly complex encryption methods for low-sensitivity data, thereby improving processing efficiency.

[0141] See also Figure 4 As shown, the step of storing the encrypted log based on the analysis result of the total quantity and sensitive information includes:

[0142] Step S410, determining the total number of storage nodes;

[0143] Step S420, determining a storage location of the encrypted log based on the final sensitivity level and the total number;

[0144] Step S430: store the encrypted log to a corresponding storage node according to the storage location, and obtain an initial storage log.

[0145] Specifically, the total number of storage nodes described in the embodiment of the present invention is determined according to the physical architecture and storage capacity of the lake-warehouse integrated platform;

[0146] The step of determining the storage location of the encrypted log based on the sensitivity level of the encrypted log and the total number of the storage nodes includes:

[0147] Sorting the encrypted logs according to the sensitivity level from high to low;

[0148] According to the sorting result, the encrypted logs are stored in the storage nodes in sequence until all the encrypted logs are stored;

[0149] If the sensitivity levels of the encrypted logs are the same, the encrypted logs with an earlier generation time are stored preferentially according to the generation time of the encrypted logs.

[0150] Specifically, according to the embodiment of the present invention, the step of storing the encrypted log to the corresponding storage node according to the storage location and obtaining the initial storage log includes:

[0151] According to the storage location, acquiring address information of the corresponding storage node;

[0152] The encrypted log is transmitted to the corresponding storage node through the address information to complete the storage operation.

[0153] Specifically, the embodiments of the present invention further ensure the security of these important data by preferentially storing encrypted logs with high sensitivity levels on storage nodes with higher performance and stronger security. By sorting and allocating encrypted logs according to sensitivity levels and generation times, load balancing of storage nodes is achieved to avoid overloading of some nodes while other nodes are idle. Sorting and storing encrypted logs according to generation time helps to quickly locate log data of a specific time period when needed, thereby improving data traceability.

[0154] Specifically, the step of adjusting the initial storage log according to the monitoring result of the access record includes:

[0155] Identifying an access timestamp in the access record;

[0156] determining a frequency of access based on a number of access timestamps;

[0157] Comparing the access frequency with a preset frequency, and adjusting the final sensitivity level corresponding to the initial storage log based on the comparison result to obtain an adjusted sensitivity level;

[0158] The initial storage log is adjusted based on the adjustment sensitivity level, and the updated storage log is obtained.

[0159] Specifically, the embodiments of the present invention dynamically identify the actual access mode and sensitivity changes of data by monitoring access records and calculating access frequencies, and adjust the sensitivity level of data in a timely manner to ensure that the security of data matches its current value. If the access frequency of a certain initial storage log increases abnormally, it may mean that the log contains sensitive information and is receiving more attention. By adjusting its final sensitivity level, the system can strengthen the protection measures of the log accordingly, such as increasing encryption strength or restricting access rights, thereby improving data security. Adjusting the sensitivity level according to the access frequency helps the system allocate storage resources more reasonably. For logs with lower access frequencies, their storage requirements are reduced, freeing up more storage space for data that requires a higher level of protection, thereby reducing maintenance costs.

[0160] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically generating operation logs for a lake-warehouse integrated platform, characterized in that: include: Capture user operation behaviors in real time, extract operation parameters in the operation behaviors, and generate target tags based on preset tag rules and operation parameters; Associating the target tag with the timestamp of the operation behavior to form an initial operation log; Identify sensitive information in the initial operation log, analyze the sensitive information, encrypt the initial operation log based on the analysis result, and obtain an encrypted log; Determine the total number of storage nodes based on the physical architecture and storage capacity of the lake-warehouse integrated platform, store the encrypted logs based on the analysis results of the total number and sensitive information, and obtain the initial storage logs; Performing real-time monitoring on the access records of the initial storage log, adjusting the initial storage log according to the monitoring result of the access records, updating the storage of the initial storage log according to the adjustment result, and obtaining an updated storage log; The step of storing the encrypted log based on the analysis result of the total quantity and sensitive information includes: Determine the total number of storage nodes; Determining a storage location for the encrypted log based on the final sensitivity level and the total number; According to the storage location, the encrypted log is stored in a corresponding storage node to obtain an initial storage log; The step of adjusting the initial storage log according to the monitoring result of the access record includes: Identifying an access timestamp in the access record; determining a frequency of access based on a number of access timestamps; Comparing the access frequency with a preset frequency, and adjusting the final sensitivity level corresponding to the initial storage log based on the comparison result to obtain an adjusted sensitivity level; The initial storage log is adjusted based on the adjustment sensitivity level, and the updated storage log is obtained.

2. The method for automatically generating operation logs by the lake-warehouse integrated platform according to claim 1 is characterized in that: The step of generating a target tag based on preset tag rules and operation parameters includes: Determine operation parameters based on the operation behavior, wherein the operation parameters include operation type, operation time and operation object; Setting an initial label based on the operation type, the operation time, the operation object and a preset label rule; The uniqueness and accuracy of the initial label are verified, and the initial label is adjusted based on the verification result to obtain the target label.

3. The method for automatically generating operation logs by the lake-warehouse integrated platform according to claim 2 is characterized in that: The step of verifying the uniqueness and accuracy of the initial tag comprises: Initialize an empty tag set, add a number of the initial tags to the tag set according to the generation order of the initial tags, compare the initial tags with the initial tags in the tag set, and obtain the tag comparison result. If there is no identical tag in the tag comparison result, the initial tag is unique. The actual lengths corresponding to the initial labels are compared with the preset lengths, and the actual formats corresponding to the initial labels are compared with the preset formats, and the accuracy of the initial labels is determined based on the length comparison results and the format comparison results.

4. The method for automatically generating operation logs for the lake-warehouse integrated platform according to claim 3 is characterized in that: The step of adjusting the initial label based on the verification result includes: Determine the number of repetitions of the repeated initial labels based on the label comparison result, and obtain the number of repetitions; Obtaining the number of inaccurate initial labels based on the determination result of the accuracy of the initial labels, obtaining an inaccurate number; Analyze the initial labels corresponding to the repeated quantity and the initial labels corresponding to the inaccurate quantity, and determine the final quantity according to the analysis results; The final number is compared with the preset number. If the final number is less than the preset number, the single initial label is readjusted. If the final number is greater than or equal to the preset number, the preset label rule is adjusted.

5. The method for automatically generating operation logs for the lake-warehouse integrated platform according to claim 4 is characterized in that: The step of associating the target tag with the timestamp of the operation behavior comprises: Encoding the target tag according to a preset encoding rule to form a target code; Obtaining a timestamp of the operation behavior, and converting the timestamp into a time code; Combining the target code with the time code to form an operation log code; The initial operation log is marked based on the operation log code to form a marked initial operation log.

6. The method for automatically generating operation logs for the lake-warehouse integrated platform according to claim 5 is characterized in that: The step of identifying sensitive information in the initial operation log includes: Scanning the initial operation log based on a preset sensitive word library to identify whether the initial operation log contains preset sensitive information in the preset sensitive word library; If the initial operation log contains the preset sensitive information, marking the information words corresponding to the preset sensitive information in the operation log to obtain first marked sensitive information; If the initial operation log does not contain the preset sensitive information, performing semantic analysis on the initial operation log, obtaining a semantic analysis result, and determining whether the initial operation log contains sensitive information based on the semantic analysis result; If the semantic analysis result indicates that the initial operation log contains sensitive information, the sensitive information in the initial operation log is marked to obtain second marked sensitive information.

7. The method for automatically generating operation logs for the lake-warehouse integrated platform according to claim 6 is characterized in that: The steps of analyzing sensitive information include: Determining a first sensitivity level based on a comparison result of an actual proportion of the first-marked sensitive information in the initial operation log and a preset proportion; Determining a second sensitivity level based on a comparison result of an actual proportion of the second-marked sensitive information in the operation log and a preset proportion; A final sensitivity level is determined based on the first sensitivity level and the second sensitivity level.

8. The method for automatically generating operation logs for the lake-warehouse integrated platform according to claim 7 is characterized in that: The step of encrypting the initial operation log based on the analysis result includes: When the final sensitivity level is a high sensitivity level, performing data desensitization processing on the first marked sensitive information or the second marked sensitive information, and encrypting the initial operation log after the data desensitization processing by using a first encryption algorithm to obtain an encrypted log; When the final sensitivity level is a low sensitivity level, the initial operation log is encrypted using a second encryption algorithm to obtain an encrypted log.

Images