Log generation method, log mining method, device, equipment and storage medium

Abstract

The application discloses a log generation method, a log mining method, a log generation device, a log mining device, equipment and a storage medium. The method comprises the following steps: acquiring metadata information and inserted data corresponding to a batch insertion operation; compressing each column of data in the inserted data to obtain compressed data of each column; determining a column data set according to the compressed data of each column and a data type corresponding to the compressed data of each column; and generating a logical log corresponding to the batch insertion operation according to the metadata information corresponding to the batch insertion operation and the column data set. Through the technical scheme, the generated log data amount can be reduced, the log file size can be reduced, the system IO load can be reduced, and the data amount of the database archive log file can be reduced under the condition that the batch insertion is very frequent and the inserted data amount is very large.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of vehicle technology, and in particular to a log generation method, log mining method, apparatus, device and storage medium. Background Technology

[0002] Logical logs are crucial for database systems. They record insert, update, delete, or DDL operations within transactions. Database systems can use logical logs for data replication, data synchronization and migration, data backup and recovery, and data auditing.

[0003] In a database system, the logical log of an INSERT operation typically includes: a log header, version information, table object information, table column description information, and column data.

[0004] During log mining, parsing table object information and column descriptions can reveal the table structure, column types, and column lengths. Parsing column data can reveal the actual data inserted into the table, thus allowing the execution of the INSERT statement to be reconstructed.

[0005] Existing database systems offer various optimized execution methods for handling batch inserts, but the logical log generation for batch inserts still primarily involves generating numerous INSERT logical log entries. The advantage of this approach is that log generation and mining are consistent with single inserts, requiring no additional processing for batch inserts. However, when insert operations are very frequent and the number of table columns is large, the database system generates a large number of logical log entries, resulting in a large archive log file size and high system I / O load. Summary of the Invention

[0006] The embodiments of the present invention provide a log generation method, a log mining method, an apparatus, a device, and a storage medium, which can reduce the amount of log data generated, reduce the log file size, reduce system IO load, and reduce the amount of data in database archive log files when batch insertion is frequent and the amount of inserted data is large.

[0007] According to one aspect of the present invention, a log generation method is provided, comprising:

[0008] Retrieve metadata information and inserted data corresponding to batch insert operations;

[0009] Compress each column of data in the inserted data to obtain compressed data for each column.

[0010] The column data set is determined based on the compressed data in each column and the data type corresponding to each compressed column.

[0011] The logical log for the batch insert operation is generated based on the metadata information and column data set corresponding to the batch insert operation.

[0012] According to another aspect of the present invention, a log mining method is provided, comprising:

[0013] Retrieve the metadata information and column data set of the logical log corresponding to the batch insert operation;

[0014] Decompress each column of data in the column dataset to obtain the decompressed column data.

[0015] The decompressed data in each column is divided according to the length information of each row of data in each column and the metadata information of the logical log, so as to obtain each row of data in each column of data after decompression.

[0016] Based on the metadata information of the logical log, each row of data in each column of decompressed data is concatenated to obtain the insertion data corresponding to the batch insertion operation.

[0017] According to another aspect of the present invention, a log generation apparatus is provided, the log generation apparatus comprising:

[0018] The first acquisition module is used to acquire metadata information and inserted data corresponding to the batch insert operation;

[0019] The compression module is used to compress each column of data in the inserted data to obtain compressed data for each column.

[0020] The column data set determination module is used to determine the column data set based on each compressed column of data and the data type corresponding to each compressed column of data;

[0021] The logical log generation module is used to generate logical logs corresponding to the batch insertion operations based on the metadata information and column data set corresponding to the batch insertion operations.

[0022] According to another aspect of the present invention, a log mining apparatus is provided, the log mining apparatus comprising:

[0023] The second acquisition module is used to acquire the metadata information and column data set of the logical logs corresponding to the batch insert operation;

[0024] The decompression module is used to decompress each column of data in the column data set to obtain the decompressed data for each column.

[0025] The segmentation module is used to segment the decompressed data into each row based on the length information of each row in each column and the metadata information of the logical log, so as to obtain each row of data in each column after decompression.

[0026] The concatenation module is used to concatenate each row of data in each column of the decompressed data according to the metadata information of the logical log, so as to obtain the insertion data corresponding to the batch insertion operation.

[0027] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0028] At least one processor; and

[0029] A memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log generation method or the log mining method according to any embodiment of the present invention.

[0031] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the log generation method or the log mining method described in any embodiment of the present invention.

[0032] This invention, through obtaining metadata information and inserted data corresponding to batch insert operations, compresses each column of data in the inserted data to obtain compressed column data; determines the column data set based on the compressed column data and the corresponding data type; and generates a logical log corresponding to the batch insert operation based on the metadata information and column data set. This reduces the amount of log data generated, lowers the log file size, reduces system IO load, and reduces the amount of data in the database archive log file when batch inserts are frequent and the amount of inserted data is large.

[0033] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a log generation method according to an embodiment of the present invention;

[0036] Figure 2 This is a flowchart of a log mining method according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of a log generation device according to an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of a log mining device according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0043] Example 1

[0044] Figure 1 This is a flowchart of a log generation method provided by an embodiment of the present invention. This embodiment is applicable to log generation in situations where batch insertion is frequent and the amount of inserted data is large. The method can be executed by the log generation device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0045] S110, retrieve the metadata information and inserted data corresponding to the batch insert operation.

[0046] The metadata information corresponding to the batch insert operation may include: log header, version information, table object information, table column description information, and row count. The inserted data refers to the data inserted corresponding to the batch insert operation. For example, if the batch insert operation is for table A, then the data inserted into table A will be considered the inserted data.

[0047] Specifically, the method to obtain the metadata information and inserted data corresponding to the batch insert operation can be: parse the batch insert operation to obtain the metadata information and inserted data corresponding to the batch insert operation.

[0048] It should be noted that for batch inserts, the metadata information (log header, version information, table object information, column description) in the logical log is a description of the same table. Since the table structure is the same for each batch of inserts, the metadata information (log header, version information, table object information, and table column description information) is also the same. If each INSERT log is recorded once, a lot of information redundancy will be generated in the log file. Each batch of inserts only needs to be recorded once.

[0049] S120 compresses each column of data in the inserted data to obtain compressed data for each column.

[0050] Specifically, the method for compressing each column of data in the inserted data to obtain the compressed data can be as follows: determine the data type of each column of data based on the metadata information corresponding to the batch insertion operation; determine the compression algorithm based on the data type of each column of data; and compress each column of data based on the compression algorithm to obtain the compressed data.

[0051] It's important to note that for each batch of inserted data, the log no longer stores each record row-by-row, but instead stores them column-by-column, meaning the data from each column of the table is organized and stored together. The main advantage of this storage method for logical logs is that it facilitates data compression. Since each column contains data of the same data type and length (except for variable-length types), the data has a high repetition rate and similar data storage patterns, making data compression much more effective than row-by-row storage. Using appropriate data compression algorithms for different data types can achieve extremely high data compression rates.

[0052] S130, determine the column data set based on the compressed data of each column and the data type corresponding to each compressed column.

[0053] The data type for each column can be either fixed-length or variable-length. It should be noted that if the data type for a column is fixed-length, the metadata information corresponding to the batch insert operation will also include the data length for that fixed-length type.

[0054] Specifically, the method for determining the column data set based on the compressed data in each column and the corresponding data type can be as follows: if the data type corresponding to the column data is a fixed-length type, then the compressed column data is added to the column data set; if the data type corresponding to the column data is a variable-length type, then the length information corresponding to each row of data in the column data is obtained, and the length information corresponding to each row of data in the column data and the compressed column data are added to the column data set.

[0055] S140, Generate a logical log corresponding to the batch insertion operation based on the metadata information and column data set corresponding to the batch insertion operation.

[0056] Specifically, a logical log corresponding to the batch insert operation is generated based on the metadata information and column data set corresponding to the batch insert operation. For example, the generated logical log is shown in Table 1:

[0057] Table 1

[0058]

[0059] Optionally, the metadata information corresponding to the batch insert operation includes: the number of rows and table column description information.

[0060] Wherein, the number of rows refers to the number of rows of data inserted corresponding to the batch insert operation. For example, if the table object to be inserted is table A, then the number of rows is the number of rows inserted into table A in this operation.

[0061] Optionally, each column of data can be compressed to obtain compressed data for each column, including:

[0062] The data type of each column is determined based on the metadata information corresponding to the batch insert operation.

[0063] The compression algorithm is determined based on the data type of each column.

[0064] Each column of data is compressed using a compression algorithm to obtain the compressed data for each column.

[0065] The data types include at least one of the following: FLOAT type, TIMESTAMP type, fixed-length CHAR type, and variable-length VARCHAR type.

[0066] It should be noted that different data types require different data compression algorithms. For example, the FLOAT type can use the XOR compression algorithm, the TIMESTAMP type can use the Delta2 compression algorithm, the fixed-length CHAR type can use the Delta+RLE compression algorithm, and the variable-length VARCHAR type can use the dictionary compression algorithm, and so on.

[0067] It should be noted that, due to the column-based storage of data, each group of columns can achieve a very high data compression rate, thereby minimizing the size of log files.

[0068] Optionally, the column data set is determined based on the compressed data in each column and the corresponding data type, including:

[0069] If the data type corresponding to the column data is a fixed-length type, then the compressed column data is added to the column data set;

[0070] If the data type corresponding to the column data is a variable length type, then obtain the length information corresponding to each row of data in the column data, and add the length information corresponding to each row of data in the column data and the compressed column data to the column data set.

[0071] It should be noted that if the data type corresponding to the column data is a fixed-length type, the column description information stores the length information corresponding to the fixed-length type. Therefore, there is no need to add the length information corresponding to the fixed-length type to the column data set, and the length information can still be obtained when mining the log file.

[0072] By adding the length information of each row of data in column data with variable data type only to the column data set, and not adding the length information of each row of data in column data with fixed data type, the size of the log file can be reduced.

[0073] In a specific example, if the batch insert operation corresponds to table A, which has 4 columns, and the data type of the first column of table A is fixed-length, the data type of the second column is variable-length, the data type of the third column is fixed-length, and the data type of the fourth column is variable-length, then after compressing the data in the first column of table A, the compressed data in the first column is added to the column data set. After compressing the data in the second column of table A, the compressed data in the second column and the length information of each row in the second column of table A are added to the column data set. After compressing the data in the third column of table A, the compressed data in the third column is added to the column data set. After compressing the data in the fourth column of table A, the compressed data in the fourth column and the length information of each row in the fourth column of table A are added to the column data set.

[0074] In this embodiment of the invention, when generating a batch insertion logical log, only one logical log is generated for each batch insertion operation. The metadata of this log includes the log header, version information, table object information, table column description information, and the number of rows (n) inserted in this batch. In the column data section, data is compressed column-wise before storage. Specifically, instead of storing each record row-wise, the data for each column of this batch of insertions is first stored together. Then, for different column data types, an appropriate data compression algorithm is selected for data compression. Due to the column-wise storage characteristic, each group of column data can achieve a high data compression rate. The compressed data from the n columns is then sequentially filled into the logical log. If it is a fixed-length field, it is filled directly. If it is a variable-length field, the length information of the n rows needs to be saved first to facilitate correct data parsing.

[0075] The technical solution of this embodiment obtains the metadata information and inserted data corresponding to the batch insert operation; compresses each column of data in the inserted data to obtain compressed column data; determines the column data set according to the compressed column data and the data type corresponding to the compressed column data; and generates the logical log corresponding to the batch insert operation according to the metadata information and column data set. This solution can eliminate the redundancy of metadata information in the batch insert logical log when batch inserts are frequent and the amount of inserted data is large. It also adopts a column-based storage method to adaptively compress data, thereby reducing the amount of generated log data, reducing the log file size, reducing system IO load, and reducing the amount of data in the database archive log file.

[0076] Example 2

[0077] Figure 2 This is a flowchart illustrating a log mining method provided in an embodiment of the present invention. This embodiment is applicable to log mining scenarios. The method can be executed by the log mining device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 2 As shown, the method specifically includes the following steps:

[0078] S210, obtain the metadata information and column data set of the logical log corresponding to the batch insert operation.

[0079] The metadata information of the logical log corresponding to the batch insertion operation includes: log header, version information, table object information, table column description information, and row number.

[0080] The column data set includes: compressed data for each column, and length information for each row of data in the variable-length column data.

[0081] S220 decompresses each column of data in the column data set to obtain the decompressed column data.

[0082] Specifically, the method for decompressing each column of data in the column data set to obtain the decompressed data can be as follows: determine the data type of each column of data in the column data set based on the table column description information; decompress each column of data according to the compression algorithm corresponding to the data type of each column of data to obtain the decompressed data.

[0083] S230: Based on the length information of each row of data in each column and the metadata information of the logical log, the decompressed data in each column is divided to obtain each row of data in each column of data.

[0084] The length information of each row of data in each column can be obtained as follows: if the data type of the column data is fixed-length, then the length information of each row of data in the column data is determined according to the data type of the column data; if the data type of the column data is variable-length, then the length information of each row of data in the column data carried by the decompressed column data is obtained.

[0085] Specifically, the decompressed column data is divided according to the length information of each row of data in each column and the metadata information of the logical log to obtain each row of data in each column of data. The method is as follows: the decompressed column data is divided sequentially until each row of data in each column of data is obtained. The division operation includes: dividing the decompressed column data according to the length information and number of rows of each row of data in the column data to obtain each row of data in the decompressed column data.

[0086] S240: Based on the metadata information of the logical log, concatenate each row of data in each column of decompressed data to obtain the insertion data corresponding to the batch insertion operation.

[0087] Specifically, the method for concatenating each row of data in each column of the decompressed data according to the metadata information of the logical log to obtain the insertion data corresponding to the batch insertion operation can be as follows: concatenating each row of data in each column of the decompressed data according to the table column description information to obtain each row of data in the insertion data corresponding to the batch insertion operation.

[0088] Optionally, the metadata information corresponding to the batch insert operation includes: table column description information;

[0089] Decompress each column of data in the column dataset to obtain the decompressed data for each column, including:

[0090] Determine the data type of each column in the column data set based on the table column description information;

[0091] Each column of data is decompressed according to the compression algorithm corresponding to its data type, resulting in decompressed data for each column.

[0092] The data types include at least one of the following: FLOAT type, TIMESTAMP type, fixed-length CHAR type, and variable-length VARCHAR type.

[0093] Specifically, the decompression of each column of data is performed based on the compression algorithm corresponding to the data type of each column. The decompression method for each column of data can be as follows: determine the decompression algorithm corresponding to each column of data based on the compression algorithm corresponding to the data type of each column of data, and then decompress each column of data using the decompression algorithm corresponding to each column of data.

[0094] Optionally, the metadata information corresponding to the batch insert operation may also include: the number of rows;

[0095] Based on the length information of each row in each column and the metadata information of the logical log, the decompressed data in each column is divided to obtain each row of data in each column, including:

[0096] The decompressed data is sequentially split into columns until each row of data in each column is obtained. The splitting operation includes:

[0097] If the data type of the column data is a fixed-length type, then the length information of each row of data in the column data is determined according to the data type of the column data;

[0098] If the data type of the column data is variable length, then obtain the length information of each row of data in the column data carried by the decompressed column data;

[0099] The decompressed column data is divided according to the length information and number of rows of each row in the column data to obtain each row of data in the decompressed column data.

[0100] It should be noted that if the data type of the column data is a fixed-length type, the table column description information stores the length information corresponding to the fixed-length type. Therefore, the length information of each row of data in the column data with a fixed-length data type can be directly determined based on the table column description information.

[0101] Optionally, based on the metadata information of the logical log, each row of data in each column of the decompressed data is concatenated to obtain the insertion data corresponding to the batch insertion operation, including:

[0102] Based on the table column description information, each row of data in each column of decompressed data is concatenated to obtain each row of data in the batch insertion data corresponding to the batch insertion operation.

[0103] The table column description information includes: the number of columns, the data type of each column, and the length of each row of data in columns with a fixed-length data type.

[0104] Specifically, based on the table column description information, each row of data in each decompressed column is concatenated to obtain each row of data in the batch insertion operation. For example, if the table object to be inserted is Table A, which has 4 columns and 3 rows, the first row of the first decompressed column, the first row of the second decompressed column, the first row of the third decompressed column, and the first row of the fourth decompressed column are concatenated to obtain the first row of data in Table A. The second row of the first decompressed column, the second row of the second decompressed column, the second row of the third decompressed column, and the second row of the fourth decompressed column are concatenated to obtain the second row of data in Table A. The third row of the first decompressed column, the third row of the second decompressed column, the third row of the third decompressed column, and the third row of the fourth decompressed column are concatenated to obtain the third row of data in Table A.

[0105] In a specific example, when parsing column data in log mining, the process first reads the table column descriptions and the number of rows (n) for this batch of insertions. Then, for each column, the compression algorithm used is determined based on the data type, and the compressed data in the logical log is decompressed to obtain column data organized by column. Next, the column data is split according to its length. The data length for fixed-length fields is fixed, while the length of each row for variable-length fields has already been saved in the logical log. After splitting the column data for each column, the data for each row is obtained. Then, based on the table column descriptions, the column data is concatenated row by row to parse out n rows of records.

[0106] The technical solution of this embodiment obtains the metadata information and column data set of the logical log corresponding to the batch insertion operation; decompresses each column of data in the column data set to obtain decompressed column data; segments each decompressed column of data according to the length information of each row of data in each column and the metadata information of the logical log to obtain each row of data in each decompressed column; and concatenates each row of data in each decompressed column of data according to the metadata information of the logical log to obtain the insertion data corresponding to the batch insertion operation. This enables the mining of the logical log corresponding to the batch insertion operation to obtain each row of data of the insertion data corresponding to the batch insertion operation.

[0107] Example 3

[0108] Figure 3 This is a schematic diagram of a log generation device provided in an embodiment of the present invention. This embodiment is applicable to log generation applications. The device can be implemented using software and / or hardware, and can be integrated into any device that provides log generation functionality, such as… Figure 3 As shown, the log generation device specifically includes: a first acquisition module 310, a compression module 320, a column data set determination module 330, and a logical log generation module 340.

[0109] The first acquisition module is used to acquire metadata information and inserted data corresponding to the batch insertion operation;

[0110] The compression module is used to compress each column of data in the inserted data to obtain compressed data for each column.

[0111] The column data set determination module is used to determine the column data set based on each compressed column of data and the data type corresponding to each compressed column of data;

[0112] The logical log generation module is used to generate logical logs corresponding to the batch insertion operations based on the metadata information and column data set corresponding to the batch insertion operations.

[0113] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0114] Example 4

[0115] Figure 4 This is a schematic diagram of a log mining device provided in an embodiment of the present invention. This embodiment is applicable to log mining applications. The device can be implemented using software and / or hardware, and can be integrated into any device that provides log mining functionality, such as… Figure 4 As shown, the log mining device specifically includes: a second acquisition module 410, a decompression module 420, a segmentation module 430, and a splicing module 440.

[0116] The second acquisition module is used to acquire the metadata information and column data set of the logical log corresponding to the batch insertion operation;

[0117] The decompression module is used to decompress each column of data in the column data set to obtain the decompressed data for each column.

[0118] The segmentation module is used to segment the decompressed data into each row based on the length information of each row in each column and the metadata information of the logical log, so as to obtain each row of data in each column after decompression.

[0119] The concatenation module is used to concatenate each row of data in each column of the decompressed data according to the metadata information of the logical log, so as to obtain the insertion data corresponding to the batch insertion operation.

[0120] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0121] Example 5

[0122] Figure 5A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0123] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0125] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as log generation methods or log mining methods.

[0126] In some embodiments, the log generation method or log mining method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log generation method or log mining method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the log generation method or log mining method by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0132] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0133] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0134] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log generation method, characterized in that, include: Retrieve metadata information and inserted data corresponding to batch insert operations; Compress each column of data in the inserted data to obtain compressed data for each column. The column data set is determined based on each compressed column of data and the corresponding data type. Generate a logical log corresponding to the batch insertion operation based on the metadata information and column data set corresponding to the batch insertion operation; The metadata information corresponding to the batch insertion operation is recorded only once. Each column of data is compressed to obtain the compressed data for each column, including: The data type of each column is determined based on the metadata information corresponding to the batch insert operation. The compression algorithm is determined based on the data type of each column. Each column of data is compressed using a compression algorithm to obtain the compressed data for each column. Each column of data corresponds to a fixed-length or variable-length data type. The method further includes: If the data type corresponding to the column data is a fixed-length type, the metadata information corresponding to the batch insertion operation includes: the data length of the fixed-length type; The process of determining the column data set based on each compressed column of data and the corresponding data type includes: If the data type corresponding to the column data is a fixed-length type, then the compressed column data is added to the column data set; If the data type corresponding to the column data is a variable length type, then obtain the length information corresponding to each row of data in the column data, and add the length information corresponding to each row of data in the column data and the compressed column data to the column data set.

2. The method according to claim 1, characterized in that, The metadata information corresponding to the batch insert operation also includes: the number of rows and table column description information.

3. A log mining method, characterized in that, include: Retrieve the metadata information and column data set of the logical log corresponding to the batch insert operation; Decompress each column of data in the column dataset to obtain the decompressed column data. The decompressed data in each column is divided according to the length information of each row of data in each column and the metadata information of the logical log, so as to obtain each row of data in each column of data after decompression. Based on the metadata information of the logical log, each row of data in each column of decompressed data is concatenated to obtain the insertion data corresponding to the batch insertion operation; The metadata information of the logical log corresponding to the batch insertion operation is recorded only once; The metadata information corresponding to the batch insert operation includes: table column description information; Decompress each column of data in the column dataset to obtain the decompressed column data, including: Determine the data type of each column in the column data set based on the table column description information; Each column of data is decompressed according to the compression algorithm corresponding to its data type, resulting in decompressed data for each column. Each column of data corresponds to a fixed-length or variable-length data type. The method further includes: If the data type corresponding to the column data is a fixed-length type, the metadata information corresponding to the batch insertion operation includes: the data length of the fixed-length type; The process of segmenting the decompressed data column based on the length information of each row in each column and the metadata information of the logical log to obtain each row of data in each decompressed column includes: The decompressed data is sequentially split into columns until each row of data in each column is obtained. The splitting operation includes: If the data type of the column data is a fixed-length type, then the length information of each row of data in the column data is determined according to the data type of the column data; If the data type of the column data is variable length, then obtain the length information of each row of data in the column data carried by the decompressed column data; The decompressed column data is divided according to the length information and number of rows of each row in the column data to obtain each row of data in the decompressed column data.

4. The method according to claim 3, characterized in that, The metadata information corresponding to the batch insert operation also includes: the number of rows.

5. The method according to claim 4, characterized in that, Based on the metadata information of the logical log, each row of data in each column of the decompressed data is concatenated to obtain the insertion data corresponding to the batch insertion operation, including: Based on the table column description information, each row of data in each column of decompressed data is concatenated to obtain each row of data in the batch insertion data corresponding to the batch insertion operation.

6. A log generation device, characterized in that, include: The first acquisition module is used to acquire metadata information and inserted data corresponding to the batch insert operation; The compression module is used to compress each column of data in the inserted data to obtain compressed data for each column. The column data set determination module is used to determine the column data set based on each compressed column of data and the data type corresponding to each compressed column of data; The logical log generation module is used to generate logical logs corresponding to the batch insertion operations based on the metadata information and column data set corresponding to the batch insertion operations. The metadata information corresponding to the batch insertion operation is recorded only once. Each column of data is compressed to obtain the compressed data for each column, including: The data type of each column is determined based on the metadata information corresponding to the batch insert operation; the compression algorithm is determined based on the data type of each column; the data in each column is compressed based on the compression algorithm to obtain the compressed data in each column. Each column of data corresponds to a fixed-length or variable-length data type. The device is also used for: If the data type corresponding to the column data is a fixed-length type, the metadata information corresponding to the batch insertion operation includes: the data length of the fixed-length type; The process of determining the column data set based on each compressed column of data and the corresponding data type includes: If the data type corresponding to the column data is a fixed-length type, then the compressed column data is added to the column data set; if the data type corresponding to the column data is a variable-length type, then the length information corresponding to each row of data in the column data is obtained, and the length information corresponding to each row of data in the column data and the compressed column data are added to the column data set.

7. A log mining device, characterized in that, include: The second acquisition module is used to acquire the metadata information and column data set of the logical logs corresponding to the batch insert operation; The decompression module is used to decompress each column of data in the column data set to obtain the decompressed data for each column. The segmentation module is used to segment the decompressed data into each row based on the length information of each row in each column and the metadata information of the logical log, so as to obtain each row of data in each column after decompression. The splicing module is used to splice each row of data in each column of the decompressed data according to the metadata information of the logical log to obtain the insertion data corresponding to the batch insertion operation. The metadata information of the logical log corresponding to the batch insertion operation is recorded only once; The metadata information corresponding to the batch insert operation includes: table column description information; Decompress each column of data in the column dataset to obtain the decompressed column data, including: Determine the data type of each column in the column data set based on the column description information; decompress each column data according to the compression algorithm corresponding to the data type of each column data to obtain the decompressed data of each column; Each column of data corresponds to a fixed-length or variable-length data type. The device is also used for: If the data type corresponding to the column data is a fixed-length type, the metadata information corresponding to the batch insertion operation includes: the data length of the fixed-length type; The process of segmenting the decompressed data column based on the length information of each row in each column and the metadata information of the logical log to obtain each row of data in each decompressed column includes: The decompressed data is sequentially split into rows until each row is obtained. The splitting operation includes: if the data type of the column data is fixed-length, determining the length information of each row in the column data based on the data type; if the data type of the column data is variable-length, obtaining the length information of each row in the column data carried by the decompressed column data; and splitting the decompressed column data according to the length information and number of rows to obtain each row in the decompressed column data.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log generation method of any one of claims 1-2, or the log mining method of any one of claims 3-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log generation method of any one of claims 1-2, or the log mining method of any one of claims 3-5.

Images