Log storage method and device, electronic equipment and storage medium
By encoding and compressing logs, converting them into numerical numbers, and writing them into byte stream files, the problem of large log storage space consumption in existing technologies is solved, achieving smaller storage space requirements and avoiding dependence on third-party storage tools.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING TOPSEC NETWORK SECURITY TECH
- Filing Date
- 2022-10-14
- Publication Date
- 2026-06-02
AI Technical Summary
Existing log storage solutions rely on third-party data storage tools, resulting in large storage space consumption and failing to effectively reduce the storage requirements of log data.
By encoding the attribute fields of the logs to be stored into numerical numbers and writing them into the block buffer, then compressing and encrypting them to form a byte stream file, and finally writing it into the byte stream file and updating the file attribute information, the use of third-party data storage tools is avoided.
Without relying on third-party storage tools, the space requirements for log storage are significantly reduced, and the storage space usage is reduced through encoding and compression techniques.
Smart Images
Figure CN115952141B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and more specifically, to a log storage method, apparatus, electronic device, and storage medium. Background Technology
[0002] Currently, various business systems, network security devices, server devices, middleware, etc. generate a large number of log records. In order to facilitate data analysis, these logs need to be stored on designated devices. However, existing log storage requires the use of third-party data storage tools, and log storage solutions based on third-party data storage tools have the disadvantage of large storage space consumption. Summary of the Invention
[0003] The purpose of this application is to provide a log storage method, apparatus, electronic device, and storage medium for writing logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application achieves a smaller byte stream file by encoding and compressing the logs to be stored.
[0004] In a first aspect, the present invention provides a log storage method, the method comprising:
[0005] Retrieve logs to be stored;
[0006] Obtain the file instance used to process the log to be stored based on the attribute fields of the log to be stored;
[0007] The attribute fields in the log to be stored are encoded based on the file instance to obtain the first byte stream corresponding to the number and the first byte stream is stored in the block buffer, wherein the byte stream in the block buffer includes at least one first byte stream;
[0008] Based on the file instance, the byte stream in the block buffer is compressed to obtain a second byte stream, and the second byte stream is written into the compression buffer.
[0009] The third byte stream is written to a byte stream file based on the file instance, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block;
[0010] Update the attribute information of the byte stream file, which is stored in the file header.
[0011] In the first aspect of this application, by acquiring a log to be stored, a file instance for processing the log can be obtained based on the attribute fields of the log to be stored. Then, the attribute fields in the log to be stored can be encoded based on the file instance to obtain a first byte stream corresponding to a numerical number, and the first byte stream is stored in a block buffer. The byte stream in the block buffer includes at least one first byte stream. Then, the byte stream in the block buffer can be compressed based on the file instance to obtain a second byte stream, and the second byte stream is written to a compression buffer. Then, the second byte stream in the compression buffer can be encrypted based on the file instance to obtain a third byte stream. Then, the third byte stream can be written to a byte stream file and the attribute information of the byte stream file can be updated based on the file instance.
[0012] In the above process, the attribute fields in the log to be stored are represented as strings before processing. String representations require more storage space. By encoding the attribute fields in the log, these string representations can be converted into numeric identifiers, which then form the first byte stream based on these smaller identifiers. Furthermore, compressing the first byte stream further reduces the storage space required for storing the log. Moreover, this process does not use third-party data storage tools to store the log.
[0013] In an optional implementation, obtaining the file instance for processing the log to be stored based on the attribute fields of the log to be stored includes:
[0014] Extract the type of the log to be stored, the IP address of the log to be stored, and the generation date of the log based on the attribute fields of the log to be stored;
[0015] Based on the type of the log to be stored, the IP address of the log to be stored, and the date the log was generated, retrieve the file instance from the instance cache for processing the log to be stored.
[0016] In the above optional implementation, the type of the log to be stored, the IP address of the log to be stored, and the generation date of the log to be stored can be extracted based on the attribute fields of the log to be stored. Then, a file instance for processing the log to be stored can be obtained from the instance cache based on the type of the log to be stored, the IP address of the log to be stored, and the generation date of the log to be stored.
[0017] In an optional implementation, the method further includes:
[0018] If the instance cache does not store the file instance of the log to be stored, then the file instance of the log to be stored is initialized in the instance cache.
[0019] In the above optional implementation, if the instance cache does not store the file instance of the log to be stored, then the file instance of the log to be stored can be initialized in the instance cache.
[0020] In an optional implementation, before compressing the byte stream in the block buffer based on the file instance to obtain a second byte stream and writing the second byte stream into the compression buffer, the method further includes:
[0021] Determine whether the block buffer has a first closing condition. If so, trigger the execution of compressing the byte stream in the block buffer based on the file instance to obtain a second byte stream and write the second byte stream into the compression buffer.
[0022] If the block buffer does not meet the first closing condition, it waits for the next first byte stream to be written.
[0023] In the above optional implementation, when the block buffer has the first closing condition, it is possible to trigger the execution of compressing the byte stream in the block buffer based on the file instance to obtain a second byte stream and write the second byte stream into the compression buffer.
[0024] In an optional implementation, before updating the attribute information of the byte stream file, the method further includes:
[0025] Determine whether the byte stream file meets the second closing condition; if so, trigger the execution of updating the attribute information of the byte stream file.
[0026] When the byte stream file does not meet the second closing condition, wait for the next third byte stream to be written.
[0027] In the above optional implementation, when the byte stream file meets the second closing condition, the execution of updating the attribute information of the byte stream file can be triggered.
[0028] In an optional implementation, after updating the attribute information of the byte stream file, the method further includes:
[0029] Close the byte stream file;
[0030] Clear the data in the block cache area and the data in the compressed cache area;
[0031] Remove the file instance of the log to be stored.
[0032] In the above optional implementation, the byte stream file can be closed, the data in the block buffer and the data in the compressed buffer can be cleared, and the file instance of the log to be stored can be removed.
[0033] In an optional implementation, the attribute information of the byte stream file includes at least the number of log entries and the end time.
[0034] In a second aspect, the present invention provides a log storage device, the device comprising:
[0035] The first acquisition module is used to acquire logs to be stored.
[0036] The second acquisition module is used to acquire a file instance for processing the log to be stored based on the attribute fields of the log to be stored;
[0037] An encoding module is used to encode the attribute fields in the log to be stored based on the file instance, to obtain a first byte stream corresponding to the numeric number, and to store the first byte stream in a block buffer, wherein the byte stream in the block buffer includes at least one first byte stream;
[0038] The compression module is used to compress the byte stream in the block buffer based on the file instance to obtain a second byte stream and write the second byte stream into the compression buffer.
[0039] The encryption module is used to encrypt the second byte stream in the compressed buffer based on the file instance to obtain the third byte stream;
[0040] The data writing module is used to write the third byte stream to a byte stream file based on the file instance, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block;
[0041] An update module is used to update the attribute information of the byte stream file, which is stored in the file header.
[0042] The apparatus of the second aspect of this application can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with a smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. String representations require more storage space. By encoding the attribute fields in the logs to be stored, the string representations can be converted into numeric numbers, and a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required for storing the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs.
[0043] Thirdly, the present invention provides an electronic device, comprising:
[0044] Processor; and
[0045] The memory is configured to store machine-readable instructions that, when executed by the processor, perform the log storage method as described in any of the foregoing embodiments.
[0046] The electronic device of the third aspect of this application can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with a smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. Data represented as strings requires more storage space. By encoding the attribute fields in the logs to be stored, the string-represented attribute fields can be converted into numeric numbers, and then a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required to store the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs.
[0047] Fourthly, the present invention provides a storage medium storing a computer program, the computer program being executed by a processor using the log storage method described in any of the foregoing embodiments.
[0048] The storage medium of the fourth aspect of this application can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with a smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. String representations require more storage space. By encoding the attribute fields in the logs to be stored, the string representations can be converted into numeric numbers, and a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required for storing the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs. Attached Figure Description
[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0050] Figure 1 This is a flowchart illustrating a log storage method disclosed in an embodiment of this application;
[0051] Figure 2 This is a schematic diagram of a dictionary in a file example provided in an embodiment of this application;
[0052] Figure 3 This is a schematic diagram of a byte stream file format disclosed in an embodiment of this application;
[0053] Figure 4 This is a schematic diagram of the structure of a log storage device disclosed in an embodiment of this application.
[0054] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation
[0055] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0056] Example 1
[0057] Please see Figure 1 , Figure 1 This is a flowchart illustrating a log storage method disclosed in an embodiment of this application, as shown below. Figure 1 As shown, the method in this application embodiment includes the following steps:
[0058] 101. Obtain the logs to be stored;
[0059] 102. Obtain the file instance used to process the log to be stored based on the attribute fields of the log to be stored;
[0060] 103. Encode the attribute fields in the log to be stored based on the file instance to obtain the first byte stream corresponding to the number and store the first byte stream in the block buffer, wherein the byte stream in the block buffer includes at least one first byte stream;
[0061] 104. Based on the byte stream in the compressed block buffer of the file instance, obtain the second byte stream and write the second byte stream into the compressed buffer;
[0062] 105. Encrypt the second byte stream in the compressed buffer based on the file instance to obtain the third byte stream;
[0063] 106. Based on the file instance, the third byte stream is written to a byte stream file, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block;
[0064] 107. Update the attribute information of the byte stream file, which is stored in the file header.
[0065] In this embodiment, by obtaining the log to be stored, a file instance for processing the log can be obtained based on the attribute fields of the log to be stored. Then, the attribute fields in the log to be stored can be encoded based on the file instance to obtain a first byte stream corresponding to the numerical number and stored in a block buffer. The byte stream in the block buffer includes at least one first byte stream. Then, the byte stream in the block buffer can be compressed based on the file instance to obtain a second byte stream and written to the compression buffer. Then, the second byte stream in the compression buffer can be encrypted based on the file instance to obtain a third byte stream. Then, the third byte stream can be written to a byte stream file and the attribute information of the byte stream file can be updated based on the file instance.
[0066] In the above steps, the attribute fields in the log to be stored are represented as strings before processing. String representations require more storage space. By encoding the attribute fields in the log, these string representations can be converted into numeric identifiers, which then form the first byte stream based on these smaller identifiers. Furthermore, compressing the first byte stream further reduces the storage space required for storing the log. Moreover, the above process does not use any third-party data storage tools to store the log.
[0067] In this embodiment of the application, for step 101, the log to be stored refers to the log that needs to be stored. The log may be generated from different data sources. For example, the log to be stored may be generated from device A or device B.
[0068] In this embodiment of the application, for step 102, the attribute fields of the log to be stored can be fields that represent the attributes of the log to be stored, such as version field, type field, and IP field.
[0069] In this embodiment, for step 102, each type of log to be stored corresponds to a file instance. For example, logs of type A correspond to file instance A, while logs of type B correspond to file instance B. Furthermore, the file instance is obtained through object instantiation in the Java language. An object describes the attributes and behaviors of a type of transaction. Therefore, the file instance includes the attributes of the log to be stored and methods for manipulating the log. For example, the file instance includes an IP field, or methods corresponding to encoded byte streams, compressed byte streams, and encrypted byte streams.
[0070] In this embodiment of the application, for step 103, encoding the attribute fields in the log to be stored based on the file instance to obtain the first byte stream corresponding to the numeric number specifically includes:
[0071] Convert the attribute fields in the log to be stored into numeric numbers based on the dictionary in the file instance;
[0072] The first byte stream is obtained by encoding the numerical ID of the log to be stored based on the compression method in the file instance.
[0073] The numerical ID of the log to be stored refers to numbers such as "1", "2", "3", etc., and each numerical ID uniquely corresponds to an attribute field. For example, 1 corresponds to the IP field and 2 corresponds to the type field.
[0074] In this embodiment of the application, for step 103, please refer to... Figure 2 , Figure 2 This is a schematic diagram of a dictionary in a file example provided in an embodiment of this application. For example... Figure 2 The attribute fields in the log to be stored are used as keys in the dictionary, corresponding to a numerical identifier. Figure 2 (Numbers not shown).
[0075] In this embodiment of the application, for step 103, the block cache area refers to the cache space used to store the first byte stream.
[0076] In this embodiment of the application, for step 105, a specific way to encrypt the second byte stream in the compressed buffer based on the file instance is to call the specified encryptor to encrypt the second byte stream based on the encryptor identifier in the file instance.
[0077] In this embodiment of the application, for step 106, please refer to... Figure 3 , Figure 3 This is a schematic diagram of a byte stream file format disclosed in an embodiment of this application. For example... Figure 3 As shown, a byte stream file includes a file header and a file block for storing the third byte stream. Figure 3 (block1, block2, block3...blockn in the file). Further, please refer to Table 1. As shown in Table 1, the file header includes attribute information of the byte stream file, such as IP address; that is, the attribute information of the byte stream file is stored in the file header.
[0078]
[0079]
[0080] Table 1
[0081] In an optional implementation, step 102: obtaining a file instance for processing the log to be stored based on the attribute fields of the log to be stored, including:
[0082] Extract the type of the log to be stored, the IP address of the log to be stored, and the date the log was generated based on the attribute fields of the log to be stored;
[0083] The file instance to be processed is retrieved from the instance cache based on the type of log to be stored, the IP address of the log to be stored, and the date the log was generated.
[0084] In the above optional implementation, the type of the log to be stored, the IP address of the log to be stored, and the date of its creation can be extracted based on the attribute fields of the log to be stored. Then, the file instance for processing the log to be stored can be obtained from the instance cache based on the type of the log to be stored, the IP address of the log to be stored, and the date of its creation.
[0085] Furthermore, in the above optional implementation, extracting the type, IP address, and creation date of the log to be stored from its attribute fields allows for the differentiation of logs from different times, devices, and types, thus distinguishing multiple types of logs. For example, the same device may generate both type A and type B logs on the same day. In this case, extracting the type, IP address, and creation date of the log to be stored from its attribute fields allows for the differentiation between type A and type B logs. This enables the accurate acquisition of file instances from the instance cache based on the log type, IP address, and creation date, ensuring that the data in the byte stream file belongs to the same type of log.
[0086] In an optional implementation, the method of this application embodiment further includes the following steps:
[0087] If the instance cache does not store the file instance of the log to be stored, then initialize the file instance of the log to be stored in the instance cache.
[0088] In the above optional implementation, if the instance cache does not store the file instance of the log to be stored, then the file instance of the log to be stored can be initialized in the instance cache.
[0089] In the above optional implementation, since some types of logs to be stored are being processed for the first time, the file instance of the log to be stored does not exist in the instance cache. For this type of scenario, by initializing the file instance of the log to be stored in the instance cache, the corresponding operations on the log to be stored can be performed using the file instance of the log to be stored. It should be noted that initializing the file instance of the log to be stored also refers to the instantiation of the byte stream file object. The byte stream file object can be instantiated using the "new" instruction in JAVA, thereby bringing the relevant data of the byte stream file object into memory. Further, when the byte stream file object is instantiated, a .dat format file is created in the specified file directory. This .dat format file serves as the byte stream file, and the file instance stores a handle to the byte stream file. Through the handle, the file instance can write a third byte stream to the byte stream file.
[0090] In an optional implementation, before the step of obtaining a second byte stream based on the byte stream in the compressed block buffer of the file instance and writing the second byte stream into the compressed buffer, the method of this application embodiment further includes the following steps:
[0091] Determine whether the block buffer meets the first closing condition. If so, trigger the execution of compressing the byte stream in the block buffer based on the file instance to obtain the second byte stream and write the second byte stream into the compressed buffer.
[0092] If the block buffer does not meet the first close condition, it waits for the next first byte stream to be written.
[0093] In the above optional implementation, when the block buffer meets the first closing condition, it is possible to trigger the execution of compressing the byte stream in the block buffer based on the file instance, obtaining a second byte stream, and writing the second byte stream into the compression buffer.
[0094] In the above implementation, the first shutdown condition may be that the remaining storage space of the block buffer is less than a preset threshold. For example, assuming that the storage capacity of the block buffer is 500M and the preset threshold is 6M, after writing byte streams multiple times, the remaining storage capacity of the block buffer is 5M. At this time, the remaining storage capacity of the block buffer is less than the preset threshold, and it can be determined that the block buffer meets the first shutdown condition.
[0095] In the above implementation, the first closing condition can also be that the first byte stream timed out when it is written to the block buffer. Specifically, if the time taken for the first byte stream to be written to the block buffer is greater than or equal to 1 minute, it can be determined that the block buffer meets the first closing condition.
[0096] In the above implementation, the first closing condition can also include storing more than or equal to 1,000 log entries.
[0097] In an optional implementation, before updating the attribute information of the byte stream file, the method of this application embodiment further includes the following steps:
[0098] Determine if the byte stream file meets the second closing condition; if so, trigger the execution to update the attribute information of the byte stream file.
[0099] When the byte stream file does not meet the second closing condition, wait for the next third byte stream to be written.
[0100] In the above optional implementation, when the byte stream file meets the second closing condition, it can trigger the execution of updating the attribute information of the byte stream file.
[0101] In the above optional embodiments, the second closing condition may be that the number of log entries stored in the byte stream file exceeds 30 million, or that the data volume of the byte stream file exceeds a preset threshold, such as the data volume of the byte stream file exceeding 2000MB. Further, the second closing condition may also be that the byte stream file...
[0102] In an optional implementation, after updating the attribute information of the byte stream file, the method of this application embodiment further includes:
[0103] Close the byte stream file;
[0104] Clear the data in the block buffer and compress the data in the buffer;
[0105] Remove the file instance to be stored logs.
[0106] In the above optional implementations, byte stream files can be closed, data in the block buffer can be cleared and data in the buffer can be compressed, and file instances of logs to be stored can be removed.
[0107] In an optional implementation, the attribute information of the byte stream file includes at least the number of log entries and the end time.
[0108] Example 2
[0109] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a log storage device disclosed in an embodiment of this application, as shown below. Figure 4 As shown, the apparatus in this embodiment includes the following functional modules:
[0110] The first acquisition module 201 is used to acquire the log to be stored;
[0111] The second acquisition module 202 is used to acquire the file instance for processing the log to be stored based on the attribute fields of the log to be stored;
[0112] Encoding module 203 is used to encode the attribute fields in the log to be stored based on the file instance, to obtain the first byte stream corresponding to the number and store the first byte stream in the block buffer, wherein the byte stream in the block buffer includes at least one first byte stream;
[0113] Compression module 204 is used to compress the byte stream in the block buffer based on the file instance, obtain the second byte stream, and write the second byte stream into the compression buffer;
[0114] Encryption module 205 is used to encrypt the second byte stream in the compressed buffer based on the file instance to obtain the third byte stream;
[0115] The data writing module 206 is used to write a third byte stream to a byte stream file based on a file instance, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block;
[0116] The update module 207 is used to update the attribute information of the byte stream file, which is stored in the file header.
[0117] The apparatus of this application embodiment can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. Data represented as strings requires more storage space. By encoding the attribute fields in the logs to be stored, the string-represented attribute fields can be converted into numeric numbers, and then a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required to store the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs.
[0118] It should be noted that for other detailed descriptions of the apparatus in the embodiments of this application, please refer to the relevant description in Embodiment 1 of this application, which will not be repeated in the embodiments of this application.
[0119] Example 3
[0120] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application, such as... Figure 5 As shown, the electronic device in this application embodiment includes:
[0121] Processor 301; and
[0122] The memory 302 is configured to store machine-readable instructions that, when executed by the processor 301, perform a log storage method as described in any of the foregoing embodiments.
[0123] The electronic device of this application embodiment can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. Data represented as strings requires more storage space. By encoding the attribute fields in the logs to be stored, the string-represented attribute fields can be converted into numeric numbers, and then a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required to store the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs.
[0124] It should be noted that for other detailed descriptions of the electronic devices in the embodiments of this application, please refer to the relevant descriptions in Embodiment 1 of this application, which will not be repeated in this embodiment.
[0125] Example 4
[0126] This application discloses a storage medium storing a computer program, which is executed by a processor using a log storage method as described in any of the foregoing embodiments.
[0127] The storage medium of this application embodiment can write logs to be stored into a byte stream file without relying on third-party data storage tools. Furthermore, this application can obtain a byte stream file with smaller storage space by encoding and compressing the logs to be stored. Specifically, the attribute fields in the logs to be stored are represented as strings before processing. Data represented as strings requires more storage space. By encoding the attribute fields in the logs to be stored, the string-represented attribute fields can be converted into numeric numbers, and then a first byte stream can be formed based on these smaller numeric numbers. Furthermore, by compressing the first byte stream, the storage space required for storing the logs can be further reduced. Moreover, the above process does not use third-party data storage tools to store the logs.
[0128] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0129] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0131] It should be noted that if a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0132] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.
[0133] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A log storage method, characterized in that, The method includes: Retrieve logs to be stored; Obtain the file instance used to process the log to be stored based on the attribute fields of the log to be stored; The attribute fields in the log to be stored are encoded based on the file instance to obtain the first byte stream corresponding to the number and the first byte stream is stored in the block buffer, wherein the byte stream in the block buffer includes at least one first byte stream; Based on the file instance, the byte stream in the block buffer is compressed to obtain a second byte stream, and the second byte stream is written into the compression buffer. Based on the file instance, the second byte stream in the compressed buffer is encrypted to obtain the third byte stream; The third byte stream is written to a byte stream file based on the file instance, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block; Update the attribute information of the byte stream file, which is stored in the file header.
2. The method as described in claim 1, characterized in that, The step of obtaining the file instance for processing the log to be stored based on the attribute fields of the log to be stored includes: Extract the type of the log to be stored, the IP address of the log to be stored, and the generation date of the log based on the attribute fields of the log to be stored; Based on the type of the log to be stored, the IP address of the log to be stored, and the date the log was generated, retrieve the file instance from the instance cache for processing the log to be stored.
3. The method as described in claim 2, characterized in that, The method further includes: If the instance cache does not store the file instance of the log to be stored, then the file instance of the log to be stored is initialized in the instance cache.
4. The method as described in claim 1, characterized in that, Before compressing the byte stream in the block buffer based on the file instance to obtain a second byte stream and writing the second byte stream into the compression buffer, the method further includes: Determine whether the block buffer has a first closing condition. If so, trigger the execution of compressing the byte stream in the block buffer based on the file instance to obtain a second byte stream and write the second byte stream into the compression buffer. If the block buffer does not meet the first closing condition, it waits for the next first byte stream to be written.
5. The method as described in claim 1, characterized in that, Before updating the attribute information of the byte stream file, the method further includes: Determine whether the byte stream file meets the second closing condition; if so, trigger the execution of updating the attribute information of the byte stream file. When the byte stream file does not meet the second closing condition, wait for the next third byte stream to be written.
6. The method as described in claim 5, characterized in that, After updating the attribute information of the byte stream file, the method further includes: Close the byte stream file; Clear the data in the block cache area and the data in the compressed cache area; Remove the file instance of the log to be stored.
7. The method as described in claim 5, characterized in that, The attribute information of the byte stream file includes at least the number of log entries and the end time.
8. A log storage device, characterized in that, The device includes: The first acquisition module is used to acquire logs to be stored. The second acquisition module is used to acquire a file instance for processing the log to be stored based on the attribute fields of the log to be stored; An encoding module is used to encode the attribute fields in the log to be stored based on the file instance, to obtain a first byte stream corresponding to the numeric number, and to store the first byte stream in a block buffer, wherein the byte stream in the block buffer includes at least one first byte stream; The compression module is used to compress the byte stream in the block buffer based on the file instance to obtain a second byte stream and write the second byte stream into the compression buffer. The encryption module is used to encrypt the second byte stream in the compressed buffer based on the file instance to obtain the third byte stream; The data writing module is used to write the third byte stream to a byte stream file based on the file instance, wherein the byte stream file includes a file block and a file header, and the third byte stream is stored in the file block; An update module is used to update the attribute information of the byte stream file, which is stored in the file header.
9. An electronic device, characterized in that, include: processor; as well as A memory configured to store machine-readable instructions that, when executed by the processor, perform the log storage method as described in any one of claims 1-7.
10. A storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor using the log storage method as described in any one of claims 1-7.