A log storage method, device, system, electronic equipment and storage medium

Abstract

The application discloses a log storage method, device and system, electronic equipment and a storage medium. The method comprises the following steps: obtaining log information, setting dimension labels for the log information to obtain labeled log information; performing dimension splitting on the labeled log information to obtain key-value log information of each dimension; determining a hash value of the key-value log information, performing repeated judgment on each key-value log information of the labeled log information based on the hash value to obtain a judgment result; and storing each key-value log information in the labeled log information and the hash value of each key-value log information based on the judgment result. The application obtains key-value log information of different dimensions by performing dimension splitting on the labeled log information, and stores the key-value log information based on the hash value of the key-value log information, so that the key-value log information with the same hash value is not stored repeatedly, and reference is used instead of copying, thereby solving the problem of redundant storage of the same log information in the prior art and reducing the storage space occupied by the log information.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a log storage method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] With the widespread application of big data and cloud computing on the Internet, the pace of integration between finance and technology is accelerating, generating more and more data in the process.

[0003] Currently, log information is stored using data collection tools such as Sqoop and Flume, which import relevant data into a Hadoop cluster to achieve data collection, processing, storage, and loading, and use HDFS to store files and data.

[0004] However, all log information is stored on the Hadoop cluster, and some meaningless log information is stored repeatedly, resulting in wasted resources. In addition, when log information is imported into the Hadoop cluster, the dimensional data in each log is the same, which results in a lot of noisy data during processing, leading to low processing efficiency. Summary of the Invention

[0005] This invention provides a log storage method, apparatus, system, electronic device, and storage medium to solve the problem of redundant log information storage in the prior art.

[0006] According to one aspect of the present invention, a log storage method is provided, characterized in that it includes:

[0007] Obtain log information, set dimension annotations for the log information, and obtain annotated log information;

[0008] The labeled log information is split into dimensions to obtain key-value log information for each dimension;

[0009] Determine the hash value of the key-value log information, and perform duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain the determination result;

[0010] Based on the determination result, the key-value log information and the hash value of each key-value log information in the labeled log information are stored.

[0011] According to another aspect of the present invention, a log storage device is provided, characterized in that it comprises:

[0012] The log information annotation module is used to obtain log information, set dimension annotations on the log information, and obtain annotated log information;

[0013] The log information splitting module is used to split the labeled log information into dimensions to obtain key-value log information for each dimension;

[0014] The hash value determination module is used to determine the hash value of the key-value log information, and to perform duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain a determination result;

[0015] The log storage module is used to store each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result.

[0016] According to another aspect of the present invention, a log storage system is provided, characterized in that it comprises: a log collector, a key-value sorter, and a storage cluster, wherein:

[0017] The log collector is used to acquire log information, set dimension labels on the log information to obtain labeled log information, and send the labeled log information to the key-value sorter;

[0018] The key-value sorter is used to receive labeled log information, split the labeled log information into dimensions to obtain key-value log information for each dimension, determine the hash value of the key-value log information, and send the key-value log information and the hash value of the key-value log information to the storage cluster.

[0019] The storage cluster is used to receive the key-value log information and the hash value of the key-value log information, perform duplicate determination on each key-value log information in the labeled log information based on the hash value, and obtain a determination result; and store each key-value log information in the labeled log information and the hash value of each key-value log information based on the determination result.

[0020] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log storage method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log storage method according to any embodiment of the present invention.

[0025] The technical solution of this invention obtains key-value log information by dimensionally splitting labeled log information, and stores the key-value log information based on the hash value of the key-value log information. This ensures that key-value log information with the same hash value is not stored repeatedly, and references are used instead of copies, which solves the problem of redundant storage of the same log information in the prior art and reduces the storage space occupied by log information. In addition, storing log information separately by different splitting dimensions facilitates the processing of log information and improves the processing efficiency of log information.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram illustrating the data storage structure implemented in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a log storage method provided in Embodiment 1 of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a log storage device provided in Embodiment 2 of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a log storage system provided in Embodiment 3 of the present invention;

[0032] Figure 5 This is a schematic diagram of a storage architecture provided in Embodiment 3 of the present invention;

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Figure 1 This is a schematic diagram illustrating the data storage structure (Merkle DAG structure) implemented in an embodiment of the present invention. For example... Figure 1 As shown, the Merkle DAG structure includes a root data block, a low-frequency key-value data layer, and a high-frequency key-value data layer.

[0037] The root data block stores a storage log information table, which maintains information such as the hash value and reference count of the stored log information.

[0038] The low-frequency key-value data layer is used to store low-frequency key-value log information, including data nodes for each low-frequency key-value log information. Low-frequency key-value log information is a combination of key dimension log fields in the log information, which can fully express each log information and reflect the value of the log information, and is noise-free data.

[0039] The high-frequency key-value data layer stores high-frequency key-value log information, including data nodes for each log entry. High-frequency key-value log information is determined by system configuration or is a partial key-value log information composed of multiple keys from non-critical log fields. These key-value combinations often represent business events occurring during system operation, such as: occurrence system, operating environment, event number, event text prompt, and translation of detailed error information. This information is generally quite long and largely identical. Large amounts of this type of information are generated during high-traffic periods, when upstream or downstream services fail, or when complex calculations time out. The layer focuses on recording and comparing high-frequency key-value log information. When a new high-frequency key-value data block is encountered, a record is dynamically added and the data block is copied. When a duplicate data block is encountered, a hash value is used to reference it instead of copying redundant data.

[0040] Example 1

[0041] Figure 2 This is a flowchart of a log storage method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where log information is stored according to different dimensions. The method can be executed by a log storage device and / or a log storage system, which can be implemented in hardware and / or software. The log storage device and / or log storage system can be configured in the electronic device provided in this embodiment of the present invention. Figure 2 As shown, the method includes:

[0042] S210. Obtain log information, set dimension annotations on the log information, and obtain annotated log information.

[0043] Log information refers to information recording user operations and system operating status. In this embodiment, log information sent by the consumer system is collected, and dimension labels are set for the log fields in the collected log information according to different dimensions to obtain labeled log information; where dimension labels refer to the labels set according to the dimensions of the log fields, and correspondingly, the labeled log information is the log information after setting dimension labels.

[0044] Based on the above embodiments, optionally, setting dimension annotations on the log information to obtain annotated log information includes: parsing the log information to obtain each log field of the log information, setting dimension annotations on each log field, and forming the annotated log information based on the annotated log fields; wherein, the dimensions include low-frequency dimensions and high-frequency dimensions.

[0045] In this context, log fields refer to the data fields in the log information. Specifically, log fields include, but are not limited to, event numbers, event text prompts, and translations of detailed error information, etc., without further limitation. In this embodiment, the log information is parsed to extract each log field, and each log field is labeled according to the set dimensions of the log field. Labeled log information is formed based on the labeled log fields. The frequency dimension includes a low-frequency dimension and a high-frequency dimension. The low-frequency dimension is used to label log fields that occur infrequently in the log information, and correspondingly, the high-frequency dimension is used to label log fields that occur frequently in the log information.

[0046] It is understandable that log fields that appear frequently in log information are non-critical log fields, such as event numbers, event text prompts, and translations of error details. Conversely, log fields that appear infrequently in log information are generally critical log fields.

[0047] S220. The labeled log information is split into dimensions to obtain key-value log information for each dimension.

[0048] Dimension splitting refers to splitting the labeled log information according to the dimension labels. Specifically, the dimensions for dimension splitting include, but are not limited to, low-frequency dimensions and high-frequency dimensions, etc., which are not limited here. In this embodiment, the labeled log information is split according to the dimension labels to obtain key-value log information for each dimension; where key-value log information refers to the log information for each dimension obtained after splitting the labeled log information.

[0049] Based on the above embodiments, optionally, the step of splitting the labeled log information into dimensions to obtain key-value log information for each dimension includes: determining the business type of the log information, determining a key-value sorter based on the business type, and splitting the labeled log information into dimensions based on the key-value sorter to obtain key-value log information for each dimension.

[0050] Here, "business type" refers to the business type of the business system that generates log information. The business type is associated with a key-value sorter, which is used to split the labeled log information. In this embodiment, the business type of the business system that generates log information is determined, the corresponding key-value sorter is determined based on the business type, and the labeled log information is split into dimensions based on the key-value sorter to obtain key-value log information for each dimension.

[0051] For example, suppose the labeled log information is X(A, B, C1, D, E, F, G, H, I, J), where A, B, C, D are low-frequency dimensions and E, F, G, H, I, J are high-frequency dimensions. After splitting the labeled log information X, (A, B, C, D) are low-frequency dimension log information and (E, F, G, H, I, J) are high-frequency dimension log information.

[0052] It should be noted that the splitting dimension of the labeled log information in the configuration information of the key-value sorter is the same as the labeling dimension of the labeled log information; for example, it can be a high-frequency dimension and a low-frequency dimension.

[0053] S230. Determine the hash value of the key-value log information, and perform duplicate determination on each key-value log information of the labeled log information based on the hash value to obtain a determination result.

[0054] The duplicate detection refers to determining whether key-value log information has been stored in the storage cluster. Specifically, it is determined by checking whether the hash value of the key-value log information is the same as the hash value of already stored key-value log information. In this embodiment, the hash value of each key-value log information is calculated and compared with the hash value of already stored key-value log information. If the hash value of an already stored key-value log information is the same as the hash value of the key-value log information, then the determination result of the key-value log information is that it has been stored; if the hash value of any already stored key-value log information is not the same as the hash value of the key-value log information, then the determination result of the key-value log information is that it has not been stored.

[0055] Based on the above embodiments, optionally, the key-value log information includes high-frequency key-value log information and low-frequency key-value log information; correspondingly, determining the hash value of the key-value log information and performing duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain a determination result specifically includes: establishing a reference relationship between the high-frequency key-value log information and the low-frequency key-value log information, and determining the hash values ​​of the high-frequency key-value log information and the low-frequency key-value log information respectively, and performing duplicate determination based on the hash value and the stored log information table to obtain a determination result; wherein, the stored log information table is used to store the hash values ​​of the stored key-value log information.

[0056] The key-value log information includes high-frequency key-value log information and low-frequency key-value log information. High-frequency key-value log information is a combination of high-frequency dimension log fields, that is, a combination of non-key dimension log fields; correspondingly, low-frequency key-value log information is a combination of low-frequency dimension log fields, that is, a combination of key dimension log fields. In this embodiment, a reference relationship is established between high-frequency key-value log information and low-frequency key-value log information. The hash values ​​of high-frequency and low-frequency key-value log information are calculated respectively. The hash values ​​of high-frequency and low-frequency key-value log information are compared with the hash values ​​in the storage log information table. If any hash value in the storage log information table is the same as the hash value of high-frequency or low-frequency key-value log information, the determination result of high-frequency or low-frequency key-value log information is that it has been stored; otherwise, the determination result of high-frequency or low-frequency key-value log information is that it has not been stored. The storage log information table is used to store the hash values ​​of stored key-value log information. The storage log information table is stored in the root data block (i.e., the root node). The storage log information table is also used to store the number of references between high-frequency and low-frequency key-value log information.

[0057] S240. Based on the determination result, store each key-value log information and the hash value of each key-value log information in the labeled log information.

[0058] The determination result includes two results: key-value log information is stored and key-value log information is not stored. In this embodiment, according to the determination result, if the determination result of key-value log information is not stored, a new data node is generated, the key-value log information is stored in the new data node, and the hash value of the key-value log information is stored in the storage log information table; if the determination result of key-value log information is stored, the key-value log information is not stored.

[0059] Based on the above embodiments, optionally, storing each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result specifically includes: if the determination result of the low-frequency key-value log information is that it has been stored, then the labeled log information is not stored; if the determination result of the low-frequency key-value log information is that it has not been stored, then the low-frequency key-value log information and the hash value of the low-frequency key-value log information are stored, and the determination result of the high-frequency key-value log information is determined; if the determination result of the high-frequency key-value log information is that it has been stored, then the reference relationship between the data node used to store the high-frequency key-value log information and the data node used to store the low-frequency key-value log information in the stored log information table is updated; if the determination result of the high-frequency key-value log information is that it has not been stored, then the high-frequency key-value log information and the hash value of the high-frequency key-value log information are stored, and the reference relationship between the data node used to store the high-frequency key-value log information and the data node used to store the low-frequency key-value log information in the stored log information table is updated.

[0060] In this embodiment, the determination result includes the determination result of high-frequency key-value log information and the determination result of low-frequency key-value log information. If the determination result of low-frequency key-value log information is that it has been stored, then the labeled log information is not stored. If the determination result of low-frequency key-value log information is that it has not been stored, then a new data node is generated to store the low-frequency key-value log information, and the hash value of the low-frequency key-value log information is stored in the storage log information table, and the determination result of high-frequency key-value log information is determined in conjunction with this. If the determination result of high-frequency key-value log information is that it has been stored, then the high-frequency key-value log information is not stored, and the reference relationship between the data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table is updated. If the determination result of high-frequency key-value log information is that it has not been stored, then a new data node is generated to store the high-frequency key-value log information, and the hash value of the high-frequency key-value log information is stored in the storage log information table, and the reference relationship between the new data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table is updated.

[0061] Understandably, if low-frequency key-value log information has been stored, it means that the key dimensions of the log information have been stored, and discarding this log information will not result in information loss, so the labeled log information can be directly omitted from storage; if low-frequency key-value log information has not been stored, it means that the key dimensions of the log information have not been stored, and this log information must be stored.

[0062] Based on the above embodiments, optionally, after storing each key-value log information and the hash value of each key-value log information, the method further includes: updating the storage status version based on the storage log information table after the log information storage is completed.

[0063] In this embodiment, after the log information storage is completed, a root hash value is calculated based on the hash values ​​of each key-value log information in the stored log information table after the log information storage is completed. The root hash value is then used as the storage state version of the current storage node. For example, the hash values ​​of each key-value log information are added together, and the hash value of the sum is used as the root hash value. The root hash value is then used as the storage state version.

[0064] The technical solution of this embodiment obtains key-value log information by dimensionally splitting the labeled log information, and stores the key-value log information based on the hash value of the key-value log information. This ensures that key-value log information with the same hash value is not stored repeatedly, and references are used instead of copies, which solves the problem of redundant storage of the same log information in the prior art and reduces the storage space occupied by log information. In addition, storing log information separately by different splitting dimensions facilitates the processing of log information and improves the processing efficiency of log information.

[0065] Example 2

[0066] Figure 3 This is a schematic diagram of the structure of a log storage device provided in Embodiment 2 of the present invention. Figure 3 As shown, the device includes:

[0067] Log information annotation module 310 is used to obtain log information, set dimension annotations on the log information, and obtain annotated log information;

[0068] The log information splitting module 320 is used to split the labeled log information into dimensions to obtain key-value log information for each dimension;

[0069] The determination module 330 is used to determine the hash value of the key-value log information, and to perform duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain a determination result;

[0070] The log storage module 340 is used to store each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result.

[0071] Based on the above embodiments, optionally, the log information annotation module 310 is specifically used to parse the log information to obtain each log field of the log information, set dimension annotations for each log field, and form the annotated log information based on each annotated log field; wherein, the dimension includes low-frequency dimension and high-frequency dimension.

[0072] Based on the above embodiments, optionally, the log information splitting module 320 is specifically used to determine the business type of the log information, determine a key-value sorter based on the business type, and perform dimensional splitting on the labeled log information based on the key-value sorter to obtain key-value log information for each dimension.

[0073] Based on the above embodiments, optionally, the key-value log information includes high-frequency key-value log information and low-frequency key-value log information; correspondingly, the determination module 330 is specifically used to establish the reference relationship between the high-frequency key-value log information and the low-frequency key-value log information, and respectively determine the hash value of the high-frequency key-value log information and the low-frequency key-value log information, and perform duplicate determination based on the hash value and the stored log information table to obtain the determination result; wherein, the stored log information table is used to store the hash values ​​of the stored key-value log information.

[0074] Based on the above embodiments, optionally, the log storage module 340 is specifically configured to: if the determination result of the low-frequency key-value log information is that it has been stored, then not store the labeled log information; if the determination result of the low-frequency key-value log information is that it has not been stored, then store the low-frequency key-value log information and the hash value of the low-frequency key-value log information, and determine the determination result of the high-frequency key-value log information; if the determination result of the high-frequency key-value log information is that it has been stored, then update the reference relationship between the data node used to store the high-frequency key-value log information and the data node used to store the low-frequency key-value log information in the stored log information table; if the determination result of the high-frequency key-value log information is that it has not been stored, then store the high-frequency key-value log information and the hash value of the high-frequency key-value log information, and establish the reference relationship between the data node used to store the high-frequency key-value log information and the data node used to store the low-frequency key-value log information in the stored log information table.

[0075] Optionally, based on the above embodiments, the device may further include a storage status version update module, used to update the storage status version based on the storage log information table after the log information storage is completed.

[0076] The log storage device provided in the embodiments of the present invention can execute the log storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0077] Example 3

[0078] Figure 4 This is a schematic diagram of the structure of a log storage system provided in Embodiment 3 of the present invention. Figure 4 As shown, the system includes: a log collector 410, a key-value sorter 420, and a storage cluster 430, wherein:

[0079] Log collector 410 is used to acquire log information, set dimension labels on the log information to obtain labeled log information, and send the labeled log information to key-value sorter 420;

[0080] The key-value sorter 420 is used to receive labeled log information, split the labeled log information into dimensions to obtain key-value log information for each dimension, determine the hash value of the key-value log information, and send the key-value log information and the hash value of the key-value log information to the storage cluster 430.

[0081] Storage cluster 430 is used to receive the key-value log information and the hash value of the key-value log information, perform duplicate determination on each of the key-value log information in the labeled log information based on the hash value, and obtain a determination result; and store each of the key-value log information in the labeled log information and the hash value of each of the key-value log information based on the determination result.

[0082] Figure 5 This is a schematic diagram of a storage architecture provided in Embodiment 3 of the present invention. Figure 5 As shown, the log collector gathers log information from various systems and sends it to the key-value sorters corresponding to different clusters. The key-value sorters are responsible for preprocessing the log information before it is stored in the storage cluster. This primarily involves splitting the log information into high-frequency and low-frequency key-value pairs based on the sorter's configuration file or the log information's labeled dimensions (critical and non-critical dimensions), and calculating the hash values ​​of these pairs. Low-frequency key-value pairs correspond to critical dimension data, while high-frequency key-value pairs correspond to non-critical dimension data. After preprocessing, the key-value sorters send the high-frequency and low-frequency key-value pairs, along with their hash values, to the storage cluster. The storage cluster checks for duplicates of the high-frequency and low-frequency key-value pairs, stores any unstored pairs and their hash values, and updates the references in the log information table.

[0083] Example 4

[0084] Figure 6This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the log storage method.

[0088] In some embodiments, the log storage method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log storage method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the log storage method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs for implementing the log storage method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] Example 5

[0092] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a log storage method, the method comprising:

[0093] Obtain log information, set dimension labels for the log information to obtain labeled log information; split the labeled log information into dimensions to obtain key-value log information for each dimension; determine the hash value of the key-value log information, and perform duplicate judgment on each key-value log information of the labeled log information based on the hash value to obtain the judgment result; store each key-value log information in the labeled log information and the hash value of each key-value log information based on the judgment result.

[0094] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0097] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0098] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log storage method, characterized in that, include: Obtain log information, set dimension annotations for the log information, and obtain annotated log information; The labeled log information is dimensionally split to obtain key-value log information for each dimension, including high-frequency key-value log information and low-frequency key-value log information. Determine the hash value of the key-value log information, and perform duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain a determination result. The determination result includes the determination result of the high-frequency key-value log information and the determination result of the low-frequency key-value log information. Based on the determination result, the key-value log information and the hash value of each key-value log information in the labeled log information are stored; The step of storing each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result includes: If the determination result of the low-frequency key-value log information is that it has been stored, then the labeled log information will not be stored. If the determination result of the low-frequency key-value log information is that it is not stored, a new data node is generated to store the low-frequency key-value log information, and the hash value of the low-frequency key-value log information is stored in the storage log information table, and the determination result of the high-frequency key-value log information is determined in conjunction with this. If the determination result of the high-frequency key-value log information is that it has been stored, then the high-frequency key-value log information will not be stored, and the reference relationship between the data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table will be updated. If the determination result of the high-frequency key-value log information is that it is not stored, a new data node is generated to store the high-frequency key-value log information, and the hash value of the high-frequency key-value log information is stored in the storage log information table. The reference relationship between the new data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table is updated.

2. The method according to claim 1, characterized in that, The step of setting dimension annotations for the log information to obtain annotated log information includes: The log information is parsed to obtain each log field, and dimension labels are set for each log field. The labeled log information is formed based on the labeled log fields. The dimensions include low-frequency dimensions and high-frequency dimensions.

3. The method according to claim 1, characterized in that, The step of splitting the labeled log information into dimensions to obtain key-value log information for each dimension includes: The business type of the log information is determined, a key-value sorter is determined based on the business type, and the labeled log information is split into dimensions based on the key-value sorter to obtain key-value log information for each dimension.

4. The method according to claim 3, characterized in that, The process of determining the hash value of the key-value log information, and performing duplicate checks on each of the key-value log information in the labeled log information based on the hash value to obtain a determination result includes: Establish a reference relationship between the high-frequency key-value log information and the low-frequency key-value log information, and determine the hash values ​​of the high-frequency key-value log information and the low-frequency key-value log information respectively; perform a duplicate determination based on the hash values ​​and the stored log information table to obtain the determination result; wherein, the stored log information table is used to store the hash values ​​of the stored key-value log information.

5. The method according to claim 1, characterized in that, After storing each of the key-value log information and the hash value of each of the key-value log information, the method further includes: Update the storage status version based on the storage log information table after the log information storage is completed.

6. A log storage device, characterized in that, include: The log information annotation module is used to obtain log information, set dimension annotations on the log information, and obtain annotated log information; The log information splitting module is used to split the labeled log information into dimensions to obtain key-value log information for each dimension. The key-value log information includes high-frequency key-value log information and low-frequency key-value log information. The determination module is used to determine the hash value of the key-value log information, and to perform duplicate determination on each of the key-value log information of the labeled log information based on the hash value to obtain a determination result. The determination result includes the determination result of the high-frequency key-value log information and the determination result of the low-frequency key-value log information. The log storage module is used to store each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result; The step of storing each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result includes: If the determination result of the low-frequency key-value log information is that it has been stored, then the labeled log information will not be stored. If the determination result of the low-frequency key-value log information is that it is not stored, a new data node is generated to store the low-frequency key-value log information, and the hash value of the low-frequency key-value log information is stored in the storage log information table, and the determination result of the high-frequency key-value log information is determined in conjunction with this. If the determination result of the high-frequency key-value log information is that it has been stored, then the high-frequency key-value log information will not be stored, and the reference relationship between the data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table will be updated. If the determination result of the high-frequency key-value log information is that it is not stored, a new data node is generated to store the high-frequency key-value log information, and the hash value of the high-frequency key-value log information is stored in the storage log information table. The reference relationship between the new data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table is updated.

7. A log storage system, characterized in that, include: Log collector, key-value sorter, and storage cluster, among which: The log collector is used to acquire log information, set dimension labels on the log information to obtain labeled log information, and send the labeled log information to the key-value sorter. The key-value sorter is used to receive labeled log information, split the labeled log information into dimensions to obtain key-value log information for each dimension, determine the hash value of the key-value log information, and send the key-value log information and the hash value of the key-value log information to the storage cluster. The key-value log information includes high-frequency key-value log information and low-frequency key-value log information. The storage cluster is used to receive the key-value log information and the hash value of the key-value log information, and to perform duplicate determination on each of the key-value log information in the labeled log information based on the hash value to obtain a determination result. The determination result includes the determination result of the high-frequency key-value log information and the determination result of the low-frequency key-value log information. Based on the determination result, each of the key-value log information in the labeled log information and the hash value of each of the key-value log information are stored. The step of storing each key-value log information and the hash value of each key-value log information in the labeled log information based on the determination result includes: If the determination result of the low-frequency key-value log information is that it has been stored, then the labeled log information will not be stored. If the determination result of the low-frequency key-value log information is that it is not stored, a new data node is generated to store the low-frequency key-value log information, and the hash value of the low-frequency key-value log information is stored in the storage log information table, and the determination result of the high-frequency key-value log information is determined in conjunction with this. If the determination result of the high-frequency key-value log information is that it has been stored, then the high-frequency key-value log information will not be stored, and the reference relationship between the data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table will be updated. If the determination result of the high-frequency key-value log information is that it is not stored, a new data node is generated to store the high-frequency key-value log information, and the hash value of the high-frequency key-value log information is stored in the storage log information table. The reference relationship between the new data node used to store the high-frequency key-value log information and the new data node used to store the low-frequency key-value log information in the storage log information table is updated.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log storage method according to any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log storage method according to any one of claims 1-5.

Images