Server log data storage method, storage medium and system

Abstract

The present invention relates to the field of data processing technology, and in particular to a server log data storage method, storage medium and system. The method finds log data with strong importance and irreplaceability by determining the access frequency of log data; determines the time sensitivity of log data and grasps the time sensitivity of log data; determines the content sensitivity of log data so as to find content-sensitive log data to ensure the security of sensitive content in log data; determines the storage priority of log data according to the order of log data levels in log levels, time sensitivity and content sensitivity, reduces storage costs and also reduces the response time of the system; stores log data based on the storage priority of log data. The method can quickly respond to log data containing sensitive information or requiring timely processing, reasonably allocate storage resources, reduce storage costs, and ensure the security and availability of important log data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a server log data storage method, storage medium and system. Background Art

[0002] In the context of the rapid development of modern information technology, servers, as core components of data processing and storage, are responsible for the generation and management of massive amounts of data. Driven by emerging technologies such as cloud computing, big data, and the Internet of Things, the volume of server access logs, operation logs, system event logs, and other log data continues to grow. Server log data is not only used for system monitoring and troubleshooting, but also plays an important role in security auditing, performance analysis, and compliance management. Therefore, it is particularly important to efficiently and reliably store and manage these log data.

[0003] At present, the storage methods of server log data mainly include traditional file system storage, relational database storage and NoSQL database storage. Although the existing methods have solved the storage problem of log data to a certain extent, there are still some shortcomings: the capacity of local storage devices is limited, which makes it difficult to cope with the storage needs of large-scale log data, and in the case of huge data volume, the retrieval and analysis speed is slow, affecting real-time performance, and some log data containing sensitive information or requiring timely processing cannot get faster processing response. Therefore, there is an urgent need for a server log data storage method, storage medium and system to meet the diverse needs in the modern information technology environment. Summary of the invention

[0004] In order to solve the technical problem that log data containing sensitive information or requiring timely processing cannot be quickly processed and responded to, the purpose of the present invention is to provide a server log data storage method, storage medium and system, and the technical solutions adopted are as follows:

[0005] In a first aspect, the present invention provides a server log data storage method, the method comprising:

[0006] Get the server's log data;

[0007] Determine the access frequency of the log data based on the correlation coefficient between the number of log data accesses within a certain period and the time points within the period and the access credibility of the log data;

[0008] Determining a time sensitivity coefficient according to characteristic information of the log data, wherein the characteristic information includes the type of the log data, the sensitivity of the log data, and the security factor of the log data;

[0009] Determining the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency;

[0010] Determine the content sensitivity of the log data based on the sensitive words and the number of sensitive words in the log data, wherein the sensitive words include personal information, financial information, and authentication information;

[0011] Determine the storage priority of the log data according to the order of the level of the log data in the log level, the time sensitivity and the content sensitivity;

[0012] Log data is stored based on the storage priority of the log data.

[0013] In some embodiments, determining the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data within a period and the time points within the period and the access credibility of the log data includes:

[0014] Get the number of accesses to log data within a certain period of time and the network delay data of the server within that period of time;

[0015] Determine access credibility of the log data based on the number of accesses to the log data during the period and the network delay data of the server during the period;

[0016] Fitting the number of accesses to the log data within the period and the time sequence points within the period to obtain a trend curve of the number of accesses to the log data and a correlation coefficient between the number of accesses to the log data within the period and the time sequence points within the period;

[0017] Based on the correlation coefficient and the access credibility, the access frequency of the log data is determined.

[0018] In some embodiments, the access credibility of the log data is determined according to the following formula:

[0019] ;

[0020] In the formula, represents the access credibility of the vth log data, G represents the number of accesses to the vth log data within the period, It represents the network delay data of the vth log data at the mth moment in the time period, M represents the number of moments in the time period, m represents the sequence number of the moment in the time period, and v represents the sequence number of the log data.

[0021] In some embodiments, the access frequency of log data is determined according to the following formula:

[0022] ;

[0023] In the formula, Indicates the access frequency of the vth log data, It represents the correlation coefficient between the number of accesses to the vth log data in the period and the time points in the period. It indicates the access credibility of the vth log data in the nth sub-period in the period. The period includes multiple equally divided sub-periods. N indicates the number of sub-periods in the period, and v indicates the serial number of the log data.

[0024] In some embodiments, determining the time sensitivity coefficient according to the characteristic information of the log data includes:

[0025] Acquire feature information of the annotated log data, time-sensitive labels corresponding to the annotated log data, and a classification network to be trained, wherein the classification network to be trained is a five-layer fully connected neural network model, and the loss function of the classification network to be trained is a cross entropy function;

[0026] Based on the feature information of the annotated log data and the time-sensitivity label, the classification network to be trained is trained to obtain a target classification network;

[0027] The characteristic information of the log data is input into the target classification network to obtain a time-sensitive coefficient.

[0028] In some embodiments, the time sensitivity of the log data is determined according to the following formula:

[0029] ;

[0030] In the formula, Indicates the time sensitivity of the vth log data, Indicates the access frequency of the vth log data, Indicates the time sensitivity coefficient of the vth log data, where v is the serial number of the log data.

[0031] In some embodiments, the content sensitivity of the log data is determined according to the following formula:

[0032] ;

[0033] In the formula, Indicates the content sensitivity of the vth log data. Indicates the number of sensitive words contained in the vth log data. represents the hyperbolic tangent function, and v represents the number of log data.

[0034] In some embodiments, the storage priority of log data is determined according to the following formula:

[0035] ;

[0036] In the formula, Indicates the storage priority of the vth log data. Indicates the content sensitivity of the vth log data. Indicates the time sensitivity of the vth log data, Indicates the level of the vth log data. Indicates the serial number of the level of the vth log data in the log levels sorted from high to low, where the log levels are sorted from high to low as error level, warning level, normal level and debug level, and the serial numbers corresponding to the error level, warning level, normal level and debug level of the log data are 1, 2, 3 and 4 respectively.

[0037] In a second aspect, the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a server log data storage method as described in the first aspect.

[0038] In a third aspect, the present invention provides a server log data storage system, the system comprising:

[0039] The acquisition module is used to obtain the log data of the server;

[0040] A first determination module is used to determine the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data in a certain period and the time sequence points in the period and the access credibility of the log data;

[0041] A second determination module, configured to determine a time sensitivity coefficient according to characteristic information of the log data, wherein the characteristic information includes a type of the log data, a sensitivity of the log data, and a safety factor of the log data;

[0042] A third determination module, configured to determine the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency;

[0043] A fourth determination module, configured to determine the content sensitivity of the log data according to the sensitive words and the number of sensitive words in the log data, wherein the sensitive words include personal information, financial information, and authentication information;

[0044] a fifth determination module, configured to determine the storage priority of the log data according to the order of the level of the log data in the log level, the time sensitivity and the content sensitivity;

[0045] The storage module is used to store the log data based on the storage priority of the log data.

[0046] The present invention has the following beneficial effects:

[0047] The present invention provides a server log data storage method, storage medium and system. The method first obtains the log data of the server; then, according to the correlation coefficient between the number of log data accesses in a certain period and the time sequence point in the period and the access credibility of the log data, the access frequency of the log data is determined, so as to find the log data with strong importance and irreplaceability; according to the characteristic information of the log data, the time sensitivity coefficient is determined, so as to find the log data that needs to be processed or analyzed in time; further, based on the time sensitivity coefficient and the access frequency, the time sensitivity of the log data is determined, and the time sensitivity of the log data is comprehensively mastered; according to the sensitive words and the number of sensitive words in the log data, the content sensitivity of the log data is determined, so as to find the content-sensitive log data to ensure the security of the sensitive content in the log data; then, according to the order of the level of the log data in the log level, the time sensitivity and the content sensitivity, the storage priority of the log data is determined, so as to more reasonably allocate the storage resources, reduce the storage cost and also reduce the response time of the system; finally, based on the storage priority of the log data, the log data is stored, and by storing all the log data in priority, the log data of the server can be effectively managed and stored, the security and availability of important log data can be ensured, and the storage cost can also be optimized. This method can quickly respond to log data that contains sensitive information or needs to be processed in a timely manner, and process it in a timely manner, give priority to storage, reasonably allocate storage resources, reduce storage costs, and ensure the security and availability of important log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A method flow chart of a server log data storage method provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the structure of a server log data storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0053] The following is a detailed description of a server log data storage method provided by the present invention in conjunction with the accompanying drawings. Figure 1 , the method comprises the following steps:

[0054] Step 101, obtaining log data of the server.

[0055] The log data of the server is a stored object. In some embodiments, the log data of the server is obtained by the following steps:

[0056] Step 1011, deploy a lightweight log collection tool, such as Filebeat and Fluentd, on the server, and configure the log collection tool to monitor the specified log file or directory.

[0057] Step 1012, monitor the changes of log files in real time through the log collection tool, and capture the log data when a new log is generated or an existing log is updated.

[0058] Step 1013, using a log collection tool to perform preliminary filtering and formatting on the locally captured log data, and then using a local buffer to temporarily store the log data to balance the load.

[0059] Step 1014: Pack the log data in the local buffer and transmit it through a secure transmission protocol.

[0060] In the log data of the server, different log data correspond to respective log texts, and these information data may be frequently accessed, and the frequently accessed log data are more important and irreplaceable. In order to find the frequently accessed log data, step 102 is introduced.

[0061] Step 102 : determining the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data in a certain period and the time points in the period and the access credibility of the log data.

[0062] The access frequency of log data can reflect the importance and irreplaceability of log data. Log data with high access frequency often needs to be stored first.

[0063] In some embodiments, step 102 may include the following steps:

[0064] Step 1021, obtaining the number of accesses to the log data within a certain period of time and the network delay data of the server within the period of time.

[0065] The number of times the log data is accessed during this period can reflect the access situation of the log data during this period. The network delay data of the server during this period includes the specific values ​​of the network delay of the log data at each moment during this period, thereby quantifying the network delay situation of the log data during this period.

[0066] In some embodiments, the network delay data of the server during the period can be obtained from the network status data of the server.

[0067] Step 1022: Determine the access credibility of the log data based on the number of accesses to the log data in the period and the network delay data of the server in the period.

[0068] The access credibility of log data can comprehensively evaluate the access situation and network delay of log data in that period. Only log data with high access times and low network delay is more important.

[0069] In some embodiments, the access credibility of the log data is determined according to the following formula to implement step 1022:

[0070] ;

[0071] In the formula, It indicates the access credibility of the vth log data, which comprehensively reflects the access situation and network delay of the vth log data in the period. G indicates the number of accesses to the vth log data in the period, which reflects the access situation of the vth log data in the period. The larger G is, the higher the number of accesses to the log data is. Indicates the network delay data of the vth log data at the mth moment in the period, reflecting the network delay of the vth log data in the period and the value is not zero. The smaller it is, the lower the network delay of the log data is. M represents the number of moments in the period and its value is not zero. m represents the sequence number of the moment in the period and M is not zero. v represents the sequence number of the log data. The larger G is, The smaller the value, the higher the number of visits and the lower the network latency. The larger the value, the more accesses there are to the log data when the network status is good, which means the access credibility of the log data is higher.

[0072] Step 1023, fitting the number of accesses to the log data within the period and the time sequence points within the period to obtain a trend curve of the number of accesses to the log data and a correlation coefficient between the number of accesses to the log data within the period and the time sequence points within the period.

[0073] The time sequence point in the time period is a time point in the time period that is consistent with the time sequence. Each time sequence point in the time period has a corresponding number of accesses to the log data. The number of accesses to the log data in the time period and the time sequence points in the time period have a certain relationship. The change trend curve of the number of accesses to the log data is obtained by fitting. The change trend curve of the number of accesses to the log data can show that there is a certain change relationship between the number of accesses to the log data in the time period and the time sequence points in the time period. The obtained correlation coefficient between the number of accesses to the log data in the time period and the time sequence points in the time period can reflect the change trend of the number of accesses to the log data in the time period. The correlation coefficient can be reasonably selected as needed. In some embodiments, the correlation coefficient can be determined based on the Pearson correlation coefficient of the time series of the number of accesses on the change trend curve of the number of accesses and the time series of the time sequence points, that is, the quotient obtained by dividing the sum of the value 1 and the Pearson correlation coefficient by the value 2 is used as the correlation coefficient.

[0074] Step 1024 , based on the correlation coefficient and the access credibility, determine the access frequency of the log data, thereby reflecting the importance and irreplaceability of the log data.

[0075] In some embodiments, the access frequency of the log data is determined according to the following formula to implement step 1024:

[0076] ;

[0077] In the formula, Indicates the access frequency of the vth log data, reflecting the importance and irreplaceability of the log data. It represents the correlation coefficient between the number of accesses to the vth log data in the period and the time points in the period. The larger the value is, the more the number of log data accesses in this period will continue to rise. It represents the access credibility of the vth log data in the nth sub-period in the period. The period contains multiple equally divided sub-periods. The access credibility of the sub-period can be calculated by implementing the formula of step 1022. N represents the number of sub-periods in the period and its value is not zero. v represents the serial number of the log data.

[0078] It represents the overall access credibility of the vth log data in the period, which is calculated by the access credibility of multiple sub-periods. This can make the calculation of access credibility more accurate and better reflect the actual situation. The bigger, The larger the The larger the value is, the more frequent the access times of log data in this period is, and the higher the access reliability of log data is. In other words, the more frequent the access of the vth log data in the server is, that is, The larger it is, the more important and irreplaceable the log data is. More frequently accessed log data usually has a higher storage priority.

[0079] However, the frequency of access alone is not enough to accurately explain the storage priority of log data. The time sensitivity and content sensitivity of log data also need to be paid attention to, so that some log data that is not frequently accessed but has a high degree of sensitivity and privacy is not stored first.

[0080] Time-sensitive log data is log data that needs to be processed or analyzed in a timely manner. It is usually related to system performance, user experience, or security incidents. This type of log data often requires higher processing requirements and faster response times, and needs to be stored first.

[0081] Step 103: determining a time sensitivity coefficient according to characteristic information of the log data, wherein the characteristic information includes the type of the log data, the sensitivity of the log data, and the security coefficient of the log data.

[0082] Determine the time sensitivity coefficient to facilitate finding log data that needs to be processed or analyzed in a timely manner.

[0083] It should be noted that the type of log data can reflect error events, warning events, security events, and critical operation events, so the type of log data itself can also reflect the degree of time sensitivity.

[0084] In some embodiments, step 103 is implemented in the following manner:

[0085] Step 1031, obtaining feature information of the annotated log data, time-sensitive labels corresponding to the annotated log data, and a classification network to be trained, wherein the classification network to be trained is a five-layer fully connected neural network model, and the loss function of the classification network to be trained is a cross entropy function.

[0086] The characteristic information of the annotated log data and the time sensitivity label corresponding to the annotated log data are learning materials for training the classification network to be trained. In some embodiments, the time sensitivity label corresponding to the annotated log data can be obtained by manual judgment. The larger the numerical value corresponding to the time sensitivity label, the higher the time sensitivity of the log data.

[0087] In some embodiments, the feature information of the annotated log data and the time sensitivity labels corresponding to the annotated log data as the data set are divided into training proportions, and the ratio of the training set to the test set may be 7:3.

[0088] Step 1032: Based on the feature information and time-sensitivity labels of the annotated log data, the classification network to be trained is trained to obtain a target classification network.

[0089] The target classification network is a trained classification network with accurate prediction, which can be used to determine the time sensitivity coefficient of log data based on the feature information of log data.

[0090] In some embodiments, the classification network to be trained is trained by a gradient descent method until the loss function converges, that is, the training is completed.

[0091] Step 1033, input the characteristic information of the log data into the target classification network to obtain the time sensitivity coefficient.

[0092] The time sensitivity coefficient can reflect the degree of time sensitivity.

[0093] Step 104: Determine the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency.

[0094] The time sensitivity of log data can be comprehensively understood through the time sensitivity coefficient and access frequency.

[0095] In some embodiments, the time sensitivity of the log data is determined according to the following formula to implement step 104:

[0096] ;

[0097] In the formula, Indicates the time sensitivity of the vth log data, which comprehensively reflects the time sensitivity of the log data. Indicates the access frequency of the vth log data, It represents the time sensitivity coefficient of the vth log data, reflecting the sensitivity of the log data. v represents the serial number of the log data. The bigger, The larger the value, the more frequently the log data is accessed and the more time-sensitive the log data may be. The larger the The larger the value is, that is, the more time-sensitive the log data is, the more likely it is that the log data is of high storage priority.

[0098] Log data is not only time-sensitive, but some log data may contain some sensitive content. Log data with sensitive content requires a higher storage priority to ensure the security of the sensitive content in the log data. Therefore, it is also necessary to pay attention to the content sensitivity of the log data, and step 105 is introduced.

[0099] Step 105, determining the content sensitivity of the log data according to the sensitive words and the number of sensitive words in the log data, where the sensitive words include personal information, financial information, and authentication information.

[0100] In some implementations, personal information includes name, ID number, phone number, and email address; financial information includes credit card number and bank account information; authentication information includes passwords, keys, and tokens. Sensitive words may also include password, secret, ssn, and credit_card.

[0101] In some embodiments, the content sensitivity of the log data is determined according to the following formula to implement step 105:

[0102] ;

[0103] In the formula, Indicates the content sensitivity of the vth log data, reflecting the sensitivity of the content of the log data. Indicates the number of sensitive words contained in the vth log data. The larger the value, the more sensitive words the log data contains, and the more sensitive the content of the log data is. represents the hyperbolic tangent function, and v represents the number of log data. The bigger, The larger the The larger the size, the more sensitive the content of the log data is, and a higher storage priority is required to ensure the security of sensitive content in the log data.

[0104] After confirming the time sensitivity and content sensitivity of the log data, the log data with higher time sensitivity or content sensitivity can be found. In order to determine the order in which these log data are stored, step 106 is introduced.

[0105] Step 106 : Determine the storage priority of the log data according to the order of the log data levels in the log levels, time sensitivity, and content sensitivity.

[0106] Since different types of log data have different levels of importance, a large amount of log data needs to be stored in a hierarchical manner during real-time storage, and the storage priority of the log data needs to be determined, so as to more reasonably allocate storage resources, reduce storage costs, and also reduce system response time.

[0107] In some embodiments, the storage priority of the log data is determined according to the following formula to implement step 106:

[0108] ;

[0109] In the formula, Indicates the storage priority of the vth log data, reflecting the priority level of the storage of the vth log data. Indicates the content sensitivity of the vth log data. Indicates the time sensitivity of the vth log data, Indicates the level of the vth log data. It indicates the serial number of the level of the vth log data in the log levels sorted from high to low and the value is not zero. The smaller the serial number, the higher the level of the log data. The log levels are sorted from high to low as error level, warning level, normal level and debug level. The serial numbers corresponding to the error level, warning level, normal level and debug level of the log data are 1, 2, 3 and 4 respectively. The smaller, The bigger, The larger the value, the greater the Euclidean norm of the time sensitivity and content sensitivity of the log data. The bigger, The larger the value, the faster the system response speed required for the log data and the higher the log level of the log data. The larger the The larger the value, the higher the priority of log data storage, and the log data needs to be stored quickly and preferentially.

[0110] It should be noted that different log data often have different log levels, and log levels are usually used to identify the severity or importance of events. For example, the error level indicates that a major problem has occurred in the system or application), the warning level indicates potential problems, which is less serious than the error log. The normal level indicates recording normal operations, and the debug level is used in the development and debugging stages.

[0111] Step 107: storing the log data based on the storage priority of the log data.

[0112] Log data is stored according to its storage priority and storage resources are allocated reasonably. By storing all log data in priority order, the log data of the server can be effectively managed and stored to ensure the security and availability of important log data, while also optimizing storage costs.

[0113] In some embodiments, in response to the storage priority of the log data being within a first priority range, the log data is stored in a high-speed storage medium; in response to the storage priority of the log data being within a second priority range, the log data is stored in a medium-speed storage medium; in response to the storage priority of the log data being within a third priority range, the log data is stored in a low-speed storage medium.

[0114] In some embodiments, the high-speed storage medium can be an SSD, which can back up log data to cloud storage in real time to increase disaster recovery capabilities; the medium-speed storage medium can be a standard HDD, which can regularly back up log data to cloud storage or a tape library; the low-speed storage medium can be cold storage, which can regularly archive log data to free up space.

[0115] In some embodiments, the first priority range may be (0, 0.5], the second priority range may be (0.5, 0.8], and the third priority range may be (0.8, 1.0].

[0116] In summary, the present invention provides a server log data storage method, which can quickly respond to log data containing sensitive information or that needs to be processed in a timely manner, and perform timely processing, priority storage, reasonable allocation of storage resources, reduce storage costs, and ensure the security and availability of important log data.

[0117] In a second aspect, the present invention provides a storage medium storing a computer program. When the computer program is executed by a processor, a server log data storage method according to the first aspect is implemented.

[0118] The storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM, RAM, EPROM (Erasable Programmable Read-Only Memory), EEPROM, flash memory, magnetic card or optical card. In other words, readable media include any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0119] The storage medium provided in the embodiment of the present application has the same inventive concept as the embodiments described above. The contents not shown in detail in the storage medium can be referred to the embodiments described above and will not be repeated here.

[0120] Thirdly, Figure 2 As shown, the present invention provides a server log data storage system, the system comprising:

[0121] The acquisition module 201 is used to acquire log data of the server.

[0122] The first determination module 202 is used to determine the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data in a certain period and the time sequence points in the period and the access credibility of the log data.

[0123] The second determination module 203 is used to determine the time sensitivity coefficient according to the characteristic information of the log data, where the characteristic information includes the type of the log data, the sensitivity of the log data and the safety factor of the log data.

[0124] The third determination module 204 is used to determine the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency.

[0125] The fourth determination module 205 is used to determine the content sensitivity of the log data according to the sensitive words and the number of sensitive words in the log data, where the sensitive words include personal information, financial information and authentication information.

[0126] A fifth determination module 206, for determining the storage priority of the log data according to the order of the log data levels in the log levels, time sensitivity and content sensitivity;

[0127] The storage module 207 is used to store the log data based on the storage priority of the log data.

[0128] In some embodiments, the first determining module 202 includes:

[0129] A first acquisition unit is used to acquire the number of accesses to the log data within a certain period of time and the network delay data of the server within the period of time;

[0130] A first determining unit, configured to determine access credibility of the log data based on the number of accesses to the log data within the period and network delay data of the server within the period;

[0131] A first obtaining unit is used to fit the number of accesses to the log data in the period and the time sequence points in the period to obtain a variation trend curve of the number of accesses to the log data and a correlation coefficient between the number of accesses to the log data in the period and the time sequence points in the period;

[0132] The second determining unit is used to determine the access frequency of the log data based on the correlation coefficient and the access credibility.

[0133] In some embodiments, the access credibility of the log data is determined according to the following formula:

[0134] ;

[0135] In the formula, represents the access credibility of the vth log data, G represents the number of accesses to the vth log data within the period, It represents the network delay data of the vth log data at the mth moment in the time period, M represents the number of moments in the time period, m represents the sequence number of the moment in the time period, and v represents the sequence number of the log data.

[0136] In some embodiments, the access frequency of log data is determined according to the following formula:

[0137] ;

[0138] In the formula, Indicates the access frequency of the vth log data, It represents the correlation coefficient between the number of accesses to the vth log data in the period and the time points in the period. It indicates the access credibility of the vth log data in the nth sub-period in the period. The period includes multiple equally divided sub-periods. N indicates the number of sub-periods in the period, and v indicates the serial number of the log data.

[0139] In some embodiments, the second determining module 203 includes:

[0140] The second acquisition unit is used to acquire feature information of the annotated log data, time-sensitive labels corresponding to the annotated log data, and a classification network to be trained, wherein the classification network to be trained is a five-layer fully connected neural network model, and the loss function of the classification network to be trained is a cross entropy function;

[0141] The second obtaining unit is used to train the classification network to be trained based on the feature information and time sensitivity labels of the annotated log data to obtain a target classification network;

[0142] The third obtaining unit is used to input the feature information of the log data into the target classification network to obtain the time-sensitive coefficient.

[0143] In some embodiments, the time sensitivity of the log data is determined according to the following formula:

[0144] ;

[0145] In the formula, Indicates the time sensitivity of the vth log data, Indicates the access frequency of the vth log data, Indicates the time sensitivity coefficient of the vth log data, where v is the serial number of the log data.

[0146] In some embodiments, the content sensitivity of the log data is determined according to the following formula:

[0147] ;

[0148] In the formula, Indicates the content sensitivity of the vth log data. Indicates the number of sensitive words contained in the vth log data. represents the hyperbolic tangent function, and v represents the number of log data.

[0149] In some embodiments, the storage priority of log data is determined according to the following formula:

[0150] ;

[0151] In the formula, Indicates the storage priority of the vth log data. Indicates the content sensitivity of the vth log data. Indicates the time sensitivity of the vth log data, Indicates the level of the vth log data. Indicates the serial number of the level of the vth log data in the log levels sorted from high to low. The log levels are sorted from high to low as error level, warning level, normal level and debug level. The serial numbers corresponding to the error level, warning level, normal level and debug level of the log data are 1, 2, 3 and 4 respectively.

[0152] In summary, the present invention provides a server log data storage system, which can quickly respond to log data containing sensitive information or requiring timely processing, and perform timely processing, priority storage, reasonable allocation of storage resources, reduce storage costs, and ensure the security and availability of important log data.

[0153] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A server log data storage method, characterized in that: The method comprises: Get the server's log data; Determine the access frequency of the log data based on the correlation coefficient between the number of log data accesses within a certain period and the time points within the period and the access credibility of the log data; Determining a time sensitivity coefficient according to characteristic information of the log data, wherein the characteristic information includes the type of the log data, the sensitivity of the log data, and the security factor of the log data; Determining the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency; Determine the content sensitivity of the log data based on the sensitive words and the number of sensitive words in the log data, wherein the sensitive words include personal information, financial information, and authentication information; Determining the storage priority of the log data according to the order of the level of the log data in the log level, the time sensitivity and the content sensitivity; Based on the storage priority of the log data, the log data is stored; Determining the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data within a certain period and the time points within the period and the access credibility of the log data includes: Get the number of log data accesses within a certain period of time and the network delay data of the server within that period of time; Determine access credibility of the log data based on the number of accesses to the log data during the period and the network delay data of the server during the period; Fitting the number of accesses to the log data within the period and the time sequence points within the period to obtain a trend curve of the number of accesses to the log data and a correlation coefficient between the number of accesses to the log data within the period and the time sequence points within the period; Determining access frequency of log data based on the correlation coefficient and the access credibility; Determine the access credibility of log data according to the following formula: In the formula, represents the access credibility of the vth log data, G represents the number of accesses to the vth log data within the period, Indicates the network delay data of the vth log data at the mth moment in the period, M indicates the number of moments in the period, m indicates the sequence number of the moment in the period, and v indicates the sequence number of the log data; Determine the access frequency of log data according to the following formula: In the formula, Indicates the access frequency of the vth log data, It represents the correlation coefficient between the number of accesses to the vth log data in the period and the time points in the period. Indicates the access credibility of the vth log data in the nth sub-period in the period. The period contains multiple equally divided sub-periods. N indicates the number of sub-periods in the period, and v indicates the number of log data. Determine the time sensitivity of log data according to the following formula: In the formula, Indicates the time sensitivity of the vth log data, Indicates the access frequency of the vth log data, Indicates the time sensitivity coefficient of the vth log data, where v is the number of log data items; Determine the content sensitivity of log data according to the following formula: In the formula, Indicates the content sensitivity of the vth log data. Indicates the number of sensitive words contained in the vth log data. represents the hyperbolic tangent function, and v represents the number of log data; Determine the storage priority of log data according to the following formula: In the formula, Indicates the storage priority of the vth log data. Indicates the content sensitivity of the vth log data. Indicates the time sensitivity of the vth log data, Indicates the level of the vth log data. Indicates the serial number of the level of the vth log data in the log levels sorted from high to low, where the log levels are sorted from high to low as error level, warning level, normal level and debug level, and the serial numbers corresponding to the error level, warning level, normal level and debug level of the log data are 1, 2, 3 and 4 respectively.

2. A server log data storage method according to claim 1, characterized in that: Determining the time sensitivity coefficient according to the characteristic information of the log data includes: Acquire feature information of the annotated log data, time-sensitive labels corresponding to the annotated log data, and a classification network to be trained, wherein the classification network to be trained is a five-layer fully connected neural network model, and the loss function of the classification network to be trained is a cross entropy function; Based on the feature information of the annotated log data and the time-sensitivity label, the classification network to be trained is trained to obtain a target classification network; The characteristic information of the log data is input into the target classification network to obtain a time-sensitive coefficient.

3. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, a server log data storage method according to any one of claims 1 to 2 is implemented.

4. A server log data storage system, characterized in that: The system comprises: The acquisition module is used to obtain the log data of the server; A first determination module is used to determine the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data in a certain period and the time sequence points in the period and the access credibility of the log data; A second determination module, configured to determine a time sensitivity coefficient according to characteristic information of the log data, wherein the characteristic information includes a type of the log data, a sensitivity of the log data, and a safety factor of the log data; A third determination module, configured to determine the time sensitivity of the log data based on the time sensitivity coefficient and the access frequency; A fourth determination module, configured to determine the content sensitivity of the log data according to the sensitive words and the number of sensitive words in the log data, wherein the sensitive words include personal information, financial information, and authentication information; a fifth determination module, configured to determine the storage priority of the log data according to the order of the level of the log data in the log level, the time sensitivity and the content sensitivity; A storage module, used for storing log data based on the storage priority of the log data; Determining the access frequency of the log data according to the correlation coefficient between the number of accesses to the log data within a certain period and the time points within the period and the access credibility of the log data includes: Get the number of log data accesses within a certain period of time and the network delay data of the server within that period of time; Determine access credibility of the log data based on the number of accesses to the log data during the period and the network delay data of the server during the period; Fitting the number of accesses to the log data within the period and the time sequence points within the period to obtain a trend curve of the number of accesses to the log data and a correlation coefficient between the number of accesses to the log data within the period and the time sequence points within the period; Determining access frequency of log data based on the correlation coefficient and the access credibility; Determine the access credibility of log data according to the following formula: In the formula, represents the access credibility of the vth log data, G represents the number of accesses to the vth log data within the period, Indicates the network delay data of the vth log data at the mth moment in the period, M indicates the number of moments in the period, m indicates the sequence number of the moment in the period, and v indicates the sequence number of the log data; Determine the access frequency of log data according to the following formula: In the formula, Indicates the access frequency of the vth log data, It represents the correlation coefficient between the number of accesses to the vth log data in the period and the time points in the period. Indicates the access credibility of the vth log data in the nth sub-period in the period. The period contains multiple equally divided sub-periods. N indicates the number of sub-periods in the period, and v indicates the number of log data. Determine the time sensitivity of log data according to the following formula: In the formula, Indicates the time sensitivity of the vth log data, Indicates the access frequency of the vth log data, Indicates the time sensitivity coefficient of the vth log data, where v is the number of log data items; Determine the content sensitivity of log data according to the following formula: In the formula, Indicates the content sensitivity of the vth log data. Indicates the number of sensitive words contained in the vth log data. represents the hyperbolic tangent function, and v represents the number of log data; Determine the storage priority of log data according to the following formula: In the formula, Indicates the storage priority of the vth log data. Indicates the content sensitivity of the vth log data. Indicates the time sensitivity of the vth log data, Indicates the level of the vth log data. Indicates the serial number of the level of the vth log data in the log levels sorted from high to low, where the log levels are sorted from high to low as error level, warning level, normal level and debug level, and the serial numbers corresponding to the error level, warning level, normal level and debug level of the log data are 1, 2, 3 and 4 respectively.

Images