Log data backup method, device, equipment and storage medium

Abstract

The present application provides a log data backup method, device, equipment and storage medium. The log data backup method includes: collecting multiple log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, each regional service is pre-allocated with at least one pre-allocated storage area, determining the target storage area corresponding to each log record from the at least one pre-allocated storage area, using the target parsing thread corresponding to the target storage area, parsing each log record, obtaining the log data in each log record, and backing up the log data in multiple log records. The target parsing thread is used to parse the log records, and multi-threaded sharding and parallel processing of log records are realized. The target storage areas will not interfere with each other, so that each target storage area can also ensure high throughput while serial parsing, and no additional adjustment is required to the configuration of the HBase cluster.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log data backup method, device, equipment and storage medium. Background Art

[0002] HBase (Hadoop Database) is a highly reliable, high-performance, column-oriented, scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a personal computer server.

[0003] In the related technology, HBase's Write Ahead Log (WAL) mechanism provides a high-concurrency, persistent log storage and playback function. Each time a business data write operation is performed, the log corresponding to the data will be recorded in the log file. Therefore, based on HBase's replication function, the synchronization of business data between HBase clusters can be achieved by obtaining and replaying log files.

[0004] However, when backing up log files, the HBase cluster configuration needs to be adjusted, and the compressed log files cannot be parsed based on the Replication function. Summary of the invention

[0005] The purpose of the present application is to provide a log data backup method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to realize multi-threaded sharding and parallel processing of log records, without mutual interference between the target storage areas, so that each target storage area can ensure high throughput while serially parsing, and there is no need to make additional adjustments to the configuration of the HBase cluster.

[0006] In a first aspect, an embodiment of the present application provides a question-answering model training method, the method comprising:

[0007] Collecting a plurality of log records corresponding to each regional service from a pre-written log file of each regional service in the first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the first cluster is an HBase cluster;

[0008] Determine a target storage area corresponding to each log record from at least one pre-allocated storage area;

[0009] Using the target parsing thread corresponding to the target storage area, parse each log record to obtain the log data in each log record;

[0010] Back up log data in multiple log records.

[0011] Optionally, a target parsing thread corresponding to the target storage area is used to parse each log record to obtain log data in each log record, including:

[0012] Use the target parsing thread to deserialize each log record into a unit object;

[0013] Read the byte array from the unit object according to the preset starting character and reading length;

[0014] Write the read byte array into the preset data object to obtain the log data in each log record.

[0015] Optionally, read a byte array from the unit object according to a preset start character and read length, including:

[0016] Each attribute feature is read from the unit object according to the start character and the read length corresponding to each attribute feature; the byte array includes: at least one attribute feature read.

[0017] Optionally, each regional service also has a migration storage area, which is a storage area migrated from other regional services;

[0018] Use the target parsing thread corresponding to the target storage area to parse each log record and obtain the log data in each log record, including:

[0019] If the target storage area corresponding to the log record to be parsed is the migration storage area, obtain the deviation between the first sequence identifier of the most recently parsed log record in the service of other areas and the second sequence identifier of the log record to be parsed;

[0020] If the deviation is greater than the preset identification unit deviation, the parsing thread corresponding to the migration storage area is used in other regional services to parse the log records before the second sequence identifier;

[0021] After parsing of the log records before the second sequence identifier is completed, a parsing thread corresponding to the migration storage area is used in each regional service to parse the log records to be parsed to obtain log data in the log records to be parsed.

[0022] Optionally, multiple log records corresponding to each regional service are collected from the pre-written log file of each regional service in the first cluster, including:

[0023] Use the collection thread corresponding to each regional service to collect multiple log records from the pre-written log file.

[0024] Optionally, a collection thread corresponding to each regional service is used to collect multiple log records from the pre-written log file, including:

[0025] A collection thread is used to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file.

[0026] Optionally, a collection thread is used to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file, including:

[0027] Use the collection thread to determine whether there is a log record in the preset reading range of the pre-written log file;

[0028] If there is no log record, after waiting for a preset time, multiple log records within a preset reading range are re-read starting from the last reading position.

[0029] Optionally, before collecting a plurality of log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, the method further includes:

[0030] According to the preset log storage range corresponding to each regional service, the write-ahead log file of each regional service is determined from the write-ahead log files of the first cluster.

[0031] Optionally, determining a target storage area corresponding to each log record from at least one pre-allocated storage area includes:

[0032] Get the storage location of the business data for each log record in the HBase database;

[0033] A target storage area corresponding to each log record is determined according to the storage location and the storage location range corresponding to each pre-allocated storage area.

[0034] Optionally, before using the target parsing thread corresponding to the target storage area to parse each log record and obtain the log data in each log record, the method further includes:

[0035] The target parsing thread is determined according to the identifier of the target storage area and the number of preset multiple parsing threads.

[0036] Optionally, determining the target parsing thread according to the identifier of the target storage area and the number of preset multiple parsing threads includes:

[0037] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0038] A modulo operation is performed on the identification hash value and the number of multiple parsing threads to obtain a serial number of a target parsing thread among the multiple parsing threads.

[0039] Optionally, log data in multiple log records is backed up, including:

[0040] Sending log data in the plurality of log records to a second cluster, where the second cluster and the first cluster are a cluster of isomorphic libraries or a cluster of heterogeneous libraries;

[0041] or

[0042] Send log data from multiple log records to the message middleware.

[0043] Optionally, log data in multiple log records is sent to the message middleware, including:

[0044] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0045] Perform a modulo operation on the identification hash value and the number of topic sequences of the message middleware to obtain the sequence number of the target topic sequence corresponding to the target storage area;

[0046] Sends log data in multiple log records to a sequence of target topics.

[0047] In a third aspect, another embodiment of the present application provides a log data backup device, including:

[0048] A collection module, used to collect multiple log records corresponding to each regional service from a pre-written log file of each regional service in the first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the first cluster is an HBase cluster;

[0049] A determination module, used to determine a target storage area corresponding to each log record from at least one pre-allocated storage area;

[0050] The parsing module is used to parse each log record using a target parsing thread corresponding to the target storage area to obtain log data in each log record;

[0051] The backup module is used to back up the log data in multiple log records.

[0052] Optionally, a parsing module is specifically used to:

[0053] Use the target parsing thread to deserialize each log record into a unit object;

[0054] Read the byte array from the unit object according to the preset starting character and reading length;

[0055] Write the read byte array into the preset data object to obtain the log data in each log record.

[0056] Optionally, a parsing module is specifically used to:

[0057] Each attribute feature is read from the unit object according to the start character and the read length corresponding to each attribute feature; the byte array includes: at least one attribute feature read.

[0058] Optionally, each regional service also has a migration storage area, which is a storage area migrated from other regional services;

[0059] Parsing module, specifically used for:

[0060] If the target storage area corresponding to the log record to be parsed is the migration storage area, obtain the deviation between the first sequence identifier of the most recently parsed log record in the service of other areas and the second sequence identifier of the log record to be parsed;

[0061] If the deviation is greater than the preset identification unit deviation, the parsing thread corresponding to the migration storage area is used in other regional services to parse the log records before the second sequence identifier;

[0062] After parsing of the log records before the second sequence identifier is completed, a parsing thread corresponding to the migration storage area is used in each regional service to parse the log records to be parsed to obtain log data in the log records to be parsed.

[0063] Optionally, the acquisition module is specifically used to:

[0064] Use the collection thread corresponding to each regional service to collect multiple log records from the pre-written log file.

[0065] Optionally, the acquisition module is specifically used to:

[0066] A collection thread is used to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file.

[0067] Optionally, the acquisition module is specifically used to:

[0068] Use the collection thread to determine whether there is a log record in the preset reading range of the pre-written log file;

[0069] If there is no log record, after waiting for a preset time, multiple log records within a preset reading range are re-read starting from the last reading position.

[0070] Optionally, the determination module is further configured to:

[0071] According to the preset log storage range corresponding to each regional service, the write-ahead log file of each regional service is determined from the write-ahead log files of the first cluster.

[0072] Optionally, a module is determined, specifically for:

[0073] Get the storage location of the business data for each log record in the HBase database;

[0074] A target storage area corresponding to each log record is determined according to the storage location and the storage location range corresponding to each pre-allocated storage area.

[0075] Optionally, the determination module is further configured to:

[0076] The target parsing thread is determined according to the identifier of the target storage area and the number of preset multiple parsing threads.

[0077] Optionally, a module is determined, specifically for:

[0078] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0079] A modulo operation is performed on the identification hash value and the number of multiple parsing threads to obtain a serial number of a target parsing thread among the multiple parsing threads.

[0080] Optionally, the backup module is specifically used to:

[0081] Sending log data in the plurality of log records to a second cluster, where the second cluster and the first cluster are a cluster of isomorphic libraries or a cluster of heterogeneous libraries;

[0082] or

[0083] Send log data from multiple log records to the message middleware.

[0084] Optionally, the backup module is specifically used to:

[0085] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0086] Perform a modulo operation on the identification hash value and the number of topic sequences of the message middleware to obtain the sequence number of the target topic sequence corresponding to the target storage area;

[0087] Sends log data in multiple log records to a sequence of target topics.

[0088] In the third aspect, another embodiment of the present application provides a log data backup device, comprising: a processor, a memory and a bus, the memory storing a computer program executable by the processor, when the log data backup device is running, the processor and the memory communicate through the bus, and the processor executes the computer program to execute any one of the methods of the first aspect.

[0089] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method of any one of the first aspects is executed.

[0090] The present application provides a log data backup method, device, equipment and storage medium. The log data backup method includes: collecting multiple log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, determining the target storage area corresponding to each log record from the at least one pre-allocated storage area, using the target parsing thread corresponding to the target storage area, parsing each log record, obtaining the log data in each log record, and backing up the log data in multiple log records. The target parsing thread is used to parse the log records, and multi-threaded sharding and parallel processing of log records are realized. The target storage areas will not interfere with each other, so that each target storage area can also ensure high throughput while serial parsing, and no additional adjustment is required to the configuration of the HBase cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0092] Figure 1 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 1 ;

[0093] Figure 2 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 2 ;

[0094] Figure 3 A schematic diagram of region migration provided by an embodiment of the present application is shown;

[0095] Figure 4 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 3 ;

[0096] Figure 5 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 4 ;

[0097] Figure 6 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 5 ;

[0098] Figure 7 A schematic diagram of the structure of a log data backup device provided in an embodiment of the present application is shown;

[0099] Figure 8 A schematic diagram of the structure of a log data backup device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0100] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0101] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0102] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0103] In the prior art, in order to establish secondary indexes for HBase, synchronize HBase data to heterogeneous libraries in real time, or implement disaster recovery for HBase clusters, three common solutions are usually used: the first is to collect incremental data in batches by running batch programs and then synchronize them in batches to heterogeneous libraries; the second is to use HBase's coprocessor to monitor incremental and changed data and synchronize them to heterogeneous libraries; the third is to use HBase's replication function to achieve replication between clusters.

[0104] However, the first solution uses incremental batch collection, which can only collect incremental data based on HBase's Rowkey, and cannot collect updated or deleted data. This part of the data requires additional processing. In addition, the real-time performance of batch scheduled collection is not high enough. If the scheduled interval is too short, the incremental data of HBase will be read frequently, which may be in an empty read state and cause meaningless performance pressure on HBase. If the scheduled time interval is too long, the real-time performance is too low.

[0105] The second solution uses coprocessor synchronization, which requires writing related processing logic and packaging it into the coprocessor of HBase. All logic needs to be written into code and configured, which is relatively cumbersome and similar to the stored procedures and triggers of relational databases. In addition, the coprocessor monitors HBase data changes in real time to trigger events, which seriously affects the write performance of HBase.

[0106] The third solution is to achieve data synchronization between HBase clusters by acquiring and replaying log files based on the Replication function of HBase. This method has the following problems: it is necessary to make certain adjustments to the HBase cluster configuration and can only synchronize data between HBase clusters in homogeneous libraries; in extreme cases, data inconsistency may occur during Region migration. Although this problem can be solved through the serial Replication mechanism, it will still seriously affect the throughput; the compressed log files cannot be parsed based on the Replication function.

[0107] Based on the third solution, the present application provides a log data backup method, which uses the target parsing thread corresponding to the target storage area to parse and process the log records, realizes multi-threaded sharding and parallel processing of log records, and the target storage areas will not interfere with each other, so that each target storage area can ensure high throughput while serial parsing, and there is no need to make additional adjustments to the configuration of the HBase cluster; log backup can be used between homogeneous libraries, heterogeneous libraries and message middleware to achieve data synchronization; when migrating regions, serial processing can be used to ensure the orderliness and consistency of logs.

[0108] The log data backup method provided by the present application is described in detail below in conjunction with several specific embodiments.

[0109] Figure 1 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 1 The execution subject of this embodiment may be a log data backup device, such as a terminal device, a server, etc. Figure 1 As shown, the method may include:

[0110] S101. Collect multiple log records corresponding to each regional service from a pre-written log file of each regional service in a first cluster.

[0111] Among them, the first cluster can be a structured storage cluster built on a personal computer server using HBase technology. The first cluster is an HBase cluster. The first cluster includes multiple regional services. Taking three regional services as an example, they can be recorded as: RegionService1, RegionService2, RegionService3, each regional service is pre-allocated with at least one pre-allocated storage area. Taking three pre-allocated storage areas as an example, they can be recorded as: region1, region2, region3, region4, region5, region6, region7, region8, region9.

[0112] Normally, when HBase writes data, in order to ensure data consistency and downtime recovery, the write-ahead log function is enabled by default, that is, the data needs to be written to the log file while being written to the HMemStore. In other words, each regional service in the first HBase corresponds to a write-ahead log file, and each write-ahead log file includes multiple log records corresponding to each regional service. In this embodiment, multiple log records corresponding to each regional service can be collected from the write-ahead log file of each regional service in the first cluster.

[0113] Optionally, before step S101, the method may further include:

[0114] According to the preset log storage range corresponding to each regional service, the write-ahead log file of each regional service is determined from the write-ahead log files of the first cluster.

[0115] The preset log storage range corresponding to each regional service can be determined according to the log storage range corresponding to at least one pre-allocated storage area in each regional service. For example, the regional service RegionService1 includes three pre-allocated storage areas, which can be recorded as: Region1, Region2, and Region3. The log storage range corresponding to Region1 is 000-100, indicating that the primary key (rowkey) of the business data corresponding to the log records stored in the log storage range is 000-100 in the HBase database, the log storage range corresponding to Region2 is 100-200, and the log storage range corresponding to Region3 is 200-300. Then the preset log storage range of the regional service RegionService1 can be 000-300. It should be noted that the log storage range corresponding to the pre-allocated storage area can be the write range of the write operation in the HBase database, and the write range can be represented by rowkey.

[0116] Specifically, when a user performs a write operation on business data, the log record corresponding to the write operation will be stored in the write-ahead log file of the first cluster. Then, the write-ahead log file of each regional service can be determined from the write-ahead log file of the first cluster according to the preset log storage range corresponding to each regional service.

[0117] S102: Determine a target storage area corresponding to each log record from at least one pre-allocated storage area.

[0118] Each regional service is pre-allocated with at least one pre-allocated storage area. After collecting multiple log records corresponding to each regional service, the target storage area corresponding to each log record can also be determined from the at least one pre-allocated storage area. That is to say, there can be a corresponding relationship between the pre-allocated storage area and the log record. Then, according to the corresponding relationship, the target storage area corresponding to each log record is determined from the at least one pre-allocated storage area.

[0119] S103: using the target parsing thread corresponding to the target storage area, parse each log record to obtain log data in each log record.

[0120] Among them, there can be a corresponding relationship between the storage area and the parsing thread corresponding to each log record, so the target parsing thread corresponding to the target storage area can be used to parse each log record to obtain the log data in each log record. It should be noted that the parsing of the log record can be used to extract the operation type, operation value, operation timestamp and other attribute characteristics of the business data targeted by the log record.

[0121] Optionally, before step S103, the method may further include:

[0122] The target parsing thread is determined according to the identifier of the target storage area and the number of preset multiple parsing threads.

[0123] The identifier of the target storage area can be the identifier name of the target storage area, denoted as: RegionName. The preset number of multiple parsing threads can be the number of parsing threads corresponding to each region service. In a possible implementation, the identifier of the target storage area can be hashed to obtain the identifier hash value of the target storage area, and then the identifier hash value and the number of multiple parsing threads are modulo-operated to obtain the serial number of the target parsing thread among the multiple parsing threads.

[0124] For example, the preset multiple parsing threads can be respectively recorded as: Thread1, Thread2, Thread3, and the identifier RegionName of the target storage area is hashed to obtain an identifier hash value, which can be integer type (int) data, for example, it can be 1234567, and then the identifier hash value and the number of multiple parsing threads are modulo-operated, that is, remainder-operated, for example, 1234567%3=1, that is, 1 is the serial number of the target parsing thread among the multiple parsing threads, that is, the target parsing thread is Thread1.

[0125] S104: Back up the log data in the multiple log records.

[0126] After parsing each log record to obtain the log data in each log record, the log data in multiple log records can also be backed up so that the backup recipient can replay the business data targeted by multiple log records based on the log data, thereby achieving synchronization of business data.

[0127] Optionally, step S104 may include:

[0128] Sending log data in the plurality of log records to a second cluster;

[0129] or

[0130] Send log data from multiple log records to the message middleware.

[0131] Among them, the second cluster and the first cluster are clusters of isomorphic libraries or clusters of heterogeneous libraries, that is, the second cluster can be an HBase cluster, or other clusters, and multiple log records are parsed to obtain log data, which can be directly sent to the second cluster, that is, streaming replication of log data is realized. The second cluster can synchronize business data according to the log data in the multiple log records received.

[0132] Of course, the log data in the multiple log records may also be sent to a message middleware, which may be, for example, a message queue (MQ), so that the message middleware synchronizes the business data according to the received log data.

[0133] The log data backup method of this embodiment collects multiple log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the target storage area corresponding to each log record is determined from the at least one pre-allocated storage area, and the target parsing thread corresponding to the target storage area is used to parse each log record to obtain the log data in each log record, and the log data in multiple log records is backed up. In this embodiment, the target parsing thread is used to parse the log records, and multi-threaded sharding parallel processing of log records is realized, and the target storage areas will not interfere with each other, so that each target storage area can also ensure high throughput while serial parsing, and no additional adjustment is required to the configuration of the HBase cluster.

[0134] Combine the following Figure 2 The embodiment describes a possible implementation method for parsing log records. Figure 2 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 2 ,like Figure 2 As shown, step S103 may include:

[0135] S201. Use the target parsing thread to deserialize each log record into a unit object.

[0136] S202. Read a byte array from the unit object according to a preset start character and read length.

[0137] Deserialization refers to the process of restoring a byte sequence to a Java object. Each log record is a binary byte sequence, and the unit object is a Java object, such as a Cell (KeyValue) object. The target parsing thread can be used to deserialize each log record and deserialize each log record into a unit object.

[0138] Among them, the preset start character can be recorded as: offset start position, the read length can be recorded as: length length, the start character can be the start character of the byte array, and the read length can be the read length of the byte array. That is to say, according to the specified start character and read length of the byte array, the byte array can be read from the unit object.

[0139] Optionally, step S202 may include:

[0140] Read each attribute feature from the unit object according to the start character and read length corresponding to each attribute feature.

[0141] The byte array includes: at least one attribute feature read, and the at least one attribute feature may include, for example: an operation type, an operation value, an operation timestamp, etc. of the business data targeted by the log record.

[0142] The starting character and the reading length corresponding to each attribute feature are fixed, so each attribute feature can be read from the unit object according to the starting character and the reading length corresponding to each attribute feature.

[0143] S203: Write the read byte array into a preset data object to obtain log data in each log record.

[0144] The preset data object can be a custom pojo object or a Protobuf object. According to the preset writing rules, the read byte array can be used as a write parameter and written into the preset data object. The written data object is the log data in each log record. The preset data object can be a custom pojo object or a Protobuf object.

[0145] Among them, the preset data object may include multiple parameter items, which are recorded as: space name prefix (namespace), table name (table name), column family (family), column (columnqualifier), primary key (rowkey), value (value). According to the preset writing rules, the read byte array is written into the corresponding parameter items respectively. For example, the operation value of the business data targeted by the log record is written into value, and the storage location of the business data targeted by the log record in the HBase database is written into rowkey.

[0146] Optionally, you can also set a blacklist and whitelist to filter the contents of the blacklist from the written data object. Specifically, you can set a blacklist and whitelist about namespace, table name, column family and value, filter the blacklist data from the written data object, and then get the log data in each log record according to the filtered data object. For example, if the blacklist data is the data of the cf:name field of table A, then the data with tablename=A, family=cf and column=name can be automatically filtered out.

[0147] The log data backup method of this embodiment adopts a target parsing thread to deserialize each log record into a unit object, reads a byte array from the unit object according to a preset start character and a read length, writes the read byte array into a preset data object, and obtains the log data in each log record. In this embodiment, the log record can be parsed in this way, which facilitates the backup recipient to synchronize business data based on the log data later.

[0148] For example, using Figure 1-Figure 2 The method provided in the embodiment can parse the log records to obtain log data, thereby satisfying the orderliness of log record processing within the partition. However, in some cases, such as when a region is migrated to another RegionServer, the parsing order of the log records may be disrupted, resulting in inconsistency in the subsequent synchronized business data.

[0149] refer to Figure 3 , Figure 3 A schematic diagram of region migration provided by an embodiment of the present application is shown, such as Figure 3 As shown in the figure, before the migration, the two regional services of the first cluster can be recorded as: Regional Service 1 (RegionService1) and Regional Service 2 (RegionService2), and each regional service is pre-allocated with three pre-allocated storage areas, which can be recorded as: Pre-allocated Storage Area 1 (region1), Pre-allocated Storage Area 2 (region2), Pre-allocated Storage Area 3 (region3), Pre-allocated Storage Area 4 (region4), Pre-allocated Storage Area 5 (region5), and Pre-allocated Storage Area 6 (region6); if put and delete operations are performed on a certain data in region1 in RegionService1, but region1 is moved from RegionService1 to RegionService2 before the log record corresponding to the delete operation is parsed, because the two RegionServer1 and RegionService2 are parsed by different threads, the log record corresponding to the delete operation may be parsed before the log record corresponding to the put operation, so a dirty data is generated, that is, the order of the log records for a certain data is disordered, resulting in the inability to accurately synchronize the data.

[0150] Based on this, this application provides a solution to the problem that the order of log records of a certain data is disordered after region migration. Figure 4 Example. Figure 4 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 3 ,like Figure 4As shown, step S103 may include:

[0151] S301: If the target storage area corresponding to the log record to be parsed is a migration storage area, obtain the deviation between the first sequence identifier of the latest parsed log record in other area services and the second sequence identifier of the log record to be parsed.

[0152] Each regional service may also have a migration storage area, which is a storage area migrated from other regional services. That is, each regional service includes at least one pre-allocated storage area and a migration storage area, and the target storage area of ​​the log record may be determined from the at least one pre-allocated storage area and the migration storage area.

[0153] When parsing the log records under each regional service, if the target storage area corresponding to the log record to be parsed is the migration storage area, then obtain the first sequence identifier of the most recently parsed log record in other regional services and the second sequence identifier of the log record to be parsed. Among them, the first sequence identifier can be the identifier of the most recently parsed log record in the storage area of ​​other regional services (i.e., the migration storage area), which can be recorded as: sequence Id. It should be noted that the log records in each pre-allocated storage area and migration storage area have a sequenceId, and the sequenceId attribute is strictly incremented in the order of writing in each pre-allocated storage area and migration storage area. Similarly, the second sequence identifier can also be the identifier of the log record to be parsed in the storage area of ​​other regional services (i.e., the migration storage area).

[0154] refer to Figure 3 The first sequence identifier may be the sequence ID of the latest parsed log record in the regional service RegionService1, and the second sequence identifier may be the sequence ID of the log record to be parsed in region1 of RegionService1.

[0155] S302: If the deviation is greater than a preset identification unit deviation, the parsing thread corresponding to the migration storage area is used in other area services to parse the log records before the second sequence identifier.

[0156] S303: After parsing the log records before the second sequence identifier is completed, in each regional service, a parsing thread corresponding to the migration storage area is used to parse the log records to be parsed to obtain log data in the log records to be parsed.

[0157] If the deviation between the first sequence identifier of the most recently parsed log record in other regional services and the second sequence identifier of the log record to be parsed is greater than the preset identifier unit deviation, it means that there are unparsed log records before the second sequence identifier of the log record to be parsed. In this case, the parsing thread corresponding to the migration storage area can be used in other regional services to parse the log records before the second sequence identifier. After the parsing is completed, the parsing thread corresponding to the migration storage area can be used in each regional service to parse the log record to be parsed to obtain the log data in the log record to be parsed.

[0158] That is to say, if the latest parsed log record in other regional services has not differed from the second sequence identifier by 1, the parsing thread corresponding to the migration storage area in each regional service is blocked and waits for the latest parsed log record in other regional services to differ from the second sequence identifier by 1, then the parsing thread corresponding to the migration storage area in each regional service continues to execute, that is, parses the log record to be parsed, and obtains the log data in the log record to be parsed, thereby using the serial parsing method to ensure the orderliness and consistency of the log records.

[0159] The specific process of parsing the logs to be parsed can be found in Figure 2 The relevant description of the embodiments will not be repeated here.

[0160] refer to Figure 3 , the identification unit deviation is 1, the first sequence identification 1 is 2, and the second sequence identification is 10. Since the deviation between the first sequence identification and the second sequence identification is 8, RegionService1 uses the parsing thread corresponding to region1 to parse the log records before the sequence identification 8, that is, the log records with sequence identifications 3 to 7. After the parsing of the log records with sequence identifications 3 to 7 is completed, RegionService2 uses the parsing thread corresponding to region1 to parse the log records to be parsed and obtain the log data in the log records to be parsed.

[0161] It should be noted that if each regional service also has a migration storage area, then the preset log storage range corresponding to each regional service can be determined according to the log storage range corresponding to at least one pre-allocated storage area in each regional service and the log storage range corresponding to the migration storage area. For example, the log storage range corresponding to the migration storage area is 300-400, and the log storage range corresponding to at least one pre-allocated storage area is 000-100, 100-200, and 200-300, respectively. Then the preset log storage range corresponding to each regional service can be 000-400.

[0162] In the log data backup method of this embodiment, each regional service also has a migration storage area, and the migration storage area is a storage area migrated from other regional services. If the target storage area corresponding to the log record to be parsed is the migration storage area, the first sequence identifier of the latest parsed log record in the other regional service is obtained, and the deviation from the second sequence identifier of the log record to be parsed is obtained. If the deviation is greater than the preset identifier unit deviation, the parsing thread corresponding to the migration storage area is used in the other regional service to parse the log record before the second sequence identifier. After the parsing of the log record before the second sequence identifier is completed, the parsing thread corresponding to the migration storage area is used in each regional service to parse the log record to be parsed, and the log data in the log record to be parsed is obtained. In this embodiment, when the region is migrated, the serial processing method is used to ensure the orderliness and consistency of the log record, and the accurate synchronization of subsequent business data is ensured.

[0163] Optionally, step S101 may include:

[0164] Use the collection thread corresponding to each regional service to collect multiple log records from the pre-written log file.

[0165] That is, a collection thread can be started for each regional service, and the collection thread is used to collect multiple log records from the pre-write log file of each regional server. It should be noted that the number of pre-write log files for each regional service is at least one, so for each pre-write log file, a collection thread can be started for collection, wherein the number of pre-write log files for each regional service can be determined according to the preset system settings, and this embodiment does not specifically limit this. If the number of pre-write log files is multiple, it means that each pre-write log file of each regional service is small, which can improve the writing performance of the file.

[0166] Optionally, a collection thread corresponding to each regional service is used to collect multiple log records from the pre-written log file, including:

[0167] A collection thread is used to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file.

[0168] The multiple log records corresponding to each regional service may have position numbers, such as line numbers, recorded as position numbers. The previous read position may be the position number of the previous read in the pre-write log file. Since the previous read position has been read when the log data is backed up last time, that is, the previous read position and the log records before it have been backed up, when the log data is backed up this time, it is only necessary to read the log records from the pre-write log file after the previous read position.

[0169] Among them, the range of each batch reading, that is, the preset reading range, can also be set. The preset reading range can be the position range of log records read in batches from the pre-write log file, denoted as: batch size. In this embodiment, an acquisition thread is used to read multiple log records within the preset range starting from the previous reading position in the pre-write log file, for example, reading multiple log records from lines 101 to 110, the preset reading range is 10 lines, and the previous reading position is line 100.

[0170] Optionally, using a collection thread to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file may include:

[0171] The collection thread is used to determine whether there is a log record within a preset read range in the pre-written log file.

[0172] If there is no log record, after waiting for a preset time, multiple log records within a preset reading range are re-read starting from the last reading position.

[0173] A collection thread is used to determine whether there are log records in the preset reading range of the pre-write log file. If there are no log records, it means that there is no corresponding write operation in the preset reading range. Then, a preset time can be waited, and multiple log records in the preset reading range can be re-read from the previous reading position. This step is repeated until multiple log records are read from the preset reading range.

[0174] It should be noted that the preset duration can be selected according to actual conditions, for example, it can be 50 ms, and this embodiment does not specifically limit this.

[0175] Combine the following Figure 5 The embodiment describes a possible implementation method for determining the target storage area. Figure 5 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 4 ,like Figure 5 As shown, step S102 may include:

[0176] S401. Obtain the storage location of the business data targeted by each log record in the HBase database.

[0177] S402: Determine a target storage area corresponding to each log record according to the storage position and the storage position range corresponding to each pre-allocated storage area.

[0178] Each pre-allocated storage area corresponds to a storage location range, which may be a log storage range. The log storage range is used to represent the row number of the business data corresponding to the log record stored in the log storage range in the HBase database, for example, 000-100.

[0179] In this embodiment, the storage location of the business data targeted by each log record in the HBase database is obtained, and then the target storage area corresponding to each log record can be determined based on the storage location and the storage location range corresponding to each pre-allocated storage area. For example, the storage location range of a pre-allocated storage area is 000-100, and the storage location of the log record is 50. It can be seen that the storage location of the log record falls within the storage location range of the storage area, and the pre-allocated storage area is the target storage area corresponding to the log record.

[0180] The log data backup method of this embodiment obtains the storage location of the business data for each log record in the HBase database, and determines the target storage area corresponding to each log record according to the storage location and the storage location range corresponding to each pre-allocated storage area. In this embodiment, the target storage area corresponding to each log record can be determined according to this method, so that each log record can be parsed by using the target parsing thread corresponding to the target storage area.

[0181] Combine the following Figure 6 The embodiment describes a possible implementation method of backing up log data in multiple log records to a message middleware. Figure 6 The following is a flow chart showing the log data backup method provided by the embodiment of the present application. Figure 5 ,like Figure 6 As shown, sending log data in multiple log records to the message middleware may include:

[0182] S501. Perform hash processing on the identifier of the target storage area to obtain a hash value of the identifier of the target storage area.

[0183] S502: Perform a modulo operation on the identification hash value and the number of topic sequences of the message middleware to obtain a sequence number of a target topic sequence corresponding to the target storage area.

[0184] S503: Send the log data in the multiple log records to the target subject sequence.

[0185] The message middleware may be provided with multiple topic sequences for storing message queues, and the topic sequences may be recorded as: Topic_0, Topic_1, Topic_2.

[0186] The identifier of the target storage area can be recorded as: RegionName. The identifier of the target storage area is hashed to obtain an identifier hash value. The identifier hash value can be integer type (int) data, for example, 1234567. A modulo operation is performed on the identifier hash value and the number of topic sequences of the message middleware, and then 1234567%3=1, that is, 1 is the sequence number of the target topic sequence in the topic sequence of the message middleware, that is, the target topic sequence is Topic_1.

[0187] It should be noted that RegionName is used as the data sharding condition instead of RegionServer because each Region will send migration information. If RegionServer is used as the basis for sharding, data in the same storage area may be distributed to two Topics, and the Topics are consumed in parallel, so the orderliness of data consumption within the partition cannot be guaranteed, resulting in inconsistent business data in subsequent synchronization.

[0188] The log data backup method of this embodiment performs hash processing on the identifier of the target storage area to obtain the identifier hash value of the target storage area, performs a modulo operation on the identifier hash value and the number of topic sequences of the message middleware to obtain the sequence number of the target topic sequence corresponding to the target storage area, and sends the log data in multiple log records to the target topic sequence. In this embodiment, the log data is streamed to the message middleware, and the consistency of the data source within the partition can be guaranteed, so that the processing of streaming data can be realized.

[0189] Figure 7 The structure diagram of the log data backup device provided in the embodiment of the present application is shown, and the log data backup device can be integrated in the log data backup device. Figure 7 As shown, the log data backup device 70 may include:

[0190] The collection module 701 is used to collect multiple log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the first cluster is an HBase cluster

[0191] A determination module 702, configured to determine a target storage area corresponding to each log record from at least one pre-allocated storage area;

[0192] The parsing module 703 is used to parse each log record using a target parsing thread corresponding to the target storage area to obtain log data in each log record;

[0193] The backup module 704 is used to back up the log data in the multiple log records.

[0194] Optionally, the parsing module 703 is specifically configured to:

[0195] Use the target parsing thread to deserialize each log record into a unit object;

[0196] Read the byte array from the unit object according to the preset starting character and reading length;

[0197] Write the read byte array into the preset data object to obtain the log data in each log record.

[0198] Optionally, the parsing module 703 is specifically configured to:

[0199] Each attribute feature is read from the unit object according to the start character and the read length corresponding to each attribute feature; the byte array includes: at least one attribute feature read.

[0200] Optionally, each regional service also has a migration storage area, which is a storage area migrated from other regional services;

[0201] The parsing module 703 is specifically used for:

[0202] If the target storage area corresponding to the log record to be parsed is the migration storage area, obtain the deviation between the first sequence identifier of the most recently parsed log record in the service of other areas and the second sequence identifier of the log record to be parsed;

[0203] If the deviation is greater than the preset identification unit deviation, the parsing thread corresponding to the migration storage area is used in other regional services to parse the log records before the second sequence identifier;

[0204] After parsing of the log records before the second sequence identifier is completed, a parsing thread corresponding to the migration storage area is used in each regional service to parse the log records to be parsed to obtain log data in the log records to be parsed.

[0205] Optionally, the acquisition module 701 is specifically used for:

[0206] Use the collection thread corresponding to each regional service to collect multiple log records from the pre-written log file.

[0207] Optionally, the acquisition module 701 is specifically used for:

[0208] A collection thread is used to read multiple log records within a preset reading range starting from the last read position in the write-ahead log file.

[0209] Optionally, the acquisition module 701 is specifically used for:

[0210] Use the collection thread to determine whether there is a log record in the preset reading range of the pre-written log file;

[0211] If there is no log record, after waiting for a preset time, multiple log records within a preset reading range are re-read starting from the last reading position.

[0212] Optionally, the determination module 702 is further configured to:

[0213] According to the preset log storage range corresponding to each regional service, the write-ahead log file of each regional service is determined from the write-ahead log files of the first cluster.

[0214] Optionally, the determination module 702 is specifically configured to:

[0215] Get the storage location of the business data for each log record in the HBase database;

[0216] A target storage area corresponding to each log record is determined according to the storage location and the storage location range corresponding to each pre-allocated storage area.

[0217] Optionally, the determination module 702 is further configured to:

[0218] The target parsing thread is determined according to the identifier of the target storage area and the number of preset multiple parsing threads.

[0219] Optionally, the determination module 702 is specifically configured to:

[0220] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0221] A modulo operation is performed on the identification hash value and the number of multiple parsing threads to obtain a serial number of a target parsing thread among the multiple parsing threads.

[0222] Optionally, the backup module 704 is specifically configured to:

[0223] Sending log data in the plurality of log records to a second cluster, where the second cluster and the first cluster are a cluster of isomorphic libraries or a cluster of heterogeneous libraries;

[0224] or

[0225] Send log data from multiple log records to the message middleware.

[0226] Optionally, the backup module 704 is specifically configured to:

[0227] Performing hash processing on the identifier of the target storage area to obtain an identifier hash value of the target storage area;

[0228] Perform a modulo operation on the identification hash value and the number of topic sequences of the message middleware to obtain the sequence number of the target topic sequence corresponding to the target storage area;

[0229] Sends log data in multiple log records to a sequence of target topics.

[0230] The implementation process and principle of the log data backup device of this embodiment can refer to the log data backup method provided in the above method embodiment, and will not be described in detail here.

[0231] Figure 8 The structure diagram of the log data backup device provided by the embodiment of the present application is shown as follows: Figure 8 As shown, the log data backup device 80 includes: a processor 801, a memory 802 and a bus 803. The memory 802 stores a computer program executable by the processor 801. When the log data backup device 80 is running, the processor 801 communicates with the memory 802 through the bus 803, and the processor 801 executes the computer program to execute the above method embodiment.

[0232] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method embodiment is executed.

[0233] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0234] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0235] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A log data backup method, characterized in that: include: Collecting a plurality of log records corresponding to each regional service from a pre-written log file of each regional service in a first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the first cluster is an HBase cluster; Determine a target storage area corresponding to each log record from the at least one pre-allocated storage area; Using the target parsing thread corresponding to the target storage area, parsing each log record to obtain log data in each log record; Backing up the log data in the multiple log records; The determining of the target storage area corresponding to each log record from the at least one pre-allocated storage area includes: Obtain the storage location of the business data targeted by each log record in the HBase database; Determine a target storage area corresponding to each log record according to the storage location and a storage location range corresponding to each pre-allocated storage area; Each regional service also has a migration storage area, and the migration storage area is a storage area migrated from other regional services; The adopting the target parsing thread corresponding to the target storage area to parse each log record to obtain the log data in each log record includes: If the target storage area corresponding to the log record to be parsed is the migration storage area, obtaining a first sequence identifier of the most recently parsed log record in the other regional service and a deviation between the first sequence identifier and the second sequence identifier of the log record to be parsed, wherein the first sequence identifier is an identifier of the most recently parsed log record in the migration storage area of ​​the other regional service, and the second sequence identifier is an identifier of the log record to be parsed in the migration storage area of ​​the other regional service; If the deviation is greater than a preset identification unit deviation, the log record before the second sequence identifier is parsed using a parsing thread corresponding to the migration storage area in the other area service; After parsing of the log record before the second sequence identifier is completed, the parsing thread corresponding to the migration storage area is used in each regional service to parse the log record to be parsed to obtain log data in the log record to be parsed.

2. The method according to claim 1, characterized in that The adopting the target parsing thread corresponding to the target storage area to parse each log record to obtain the log data in each log record includes: Deserialize each log record into a unit object by using the target parsing thread; Read the byte array from the unit object according to the preset start character and read length; The read byte array is written into a preset data object to obtain the log data in each log record.

3. The method according to claim 2, characterized in that The step of reading a byte array from the unit object according to a preset start character and a read length includes: According to the starting character and the read length corresponding to each attribute feature, each attribute feature is read from the unit object; the byte array includes: at least one attribute feature read.

4. The method according to claim 1, characterized in that: The collecting a plurality of log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster includes: The plurality of log records are collected from the write-ahead log file using a collection thread corresponding to each regional service.

5. The method according to claim 4, characterized in that The adopting the collection thread corresponding to each regional service to collect the plurality of log records from the pre-written log file includes: The acquisition thread is used to read the multiple log records within a preset reading range starting from the last read position in the write-ahead log file.

6. The method according to claim 5, characterized in that The adopting the acquisition thread to read the plurality of log records within a preset reading range starting from a previous reading position in the write-ahead log file includes: Using the acquisition thread, determining whether there is a log record within the preset reading range of the pre-written log file; If there is no log record, after waiting for a preset time, the plurality of log records within the preset reading range are re-read starting from after the previous reading position.

7. The method according to claim 1, characterized in that Before collecting the plurality of log records corresponding to each regional service from the pre-written log file of each regional service in the first cluster, the method further includes: According to the preset log storage range corresponding to each regional service, the write-ahead log file of each regional service is determined from the write-ahead log files of the first cluster.

8. The method according to claim 1, characterized in that Before the target parsing thread corresponding to the target storage area is used to parse each log record to obtain the log data in each log record, the method further includes: The target parsing thread is determined according to the identifier of the target storage area and the number of preset multiple parsing threads.

9. The method according to claim 8, characterized in that The step of determining the target parsing thread according to the identifier of the target storage area and the number of the preset parsing threads includes: Performing hash processing on the identifier of the target storage area to obtain a hash value of the identifier of the target storage area; A modulo operation is performed on the identification hash value and the number of the multiple parsing threads to obtain a serial number of the target parsing thread among the multiple parsing threads.

10. The method according to claim 1, characterized in that The backing up the log data in the plurality of log records includes: Sending the log data in the plurality of log records to a second cluster, where the second cluster and the first cluster are a cluster of homogeneous libraries or a cluster of heterogeneous libraries; or The log data in the plurality of log records are sent to the message middleware.

11. The method according to claim 10, characterized in that The sending the log data in the plurality of log records to the message middleware includes: Performing hash processing on the identifier of the target storage area to obtain a hash value of the identifier of the target storage area; Performing a modulo operation on the identification hash value and the number of topic sequences of the message middleware to obtain a sequence number of a target topic sequence corresponding to the target storage area; The log data in the plurality of log records is sent to the target topic sequence.

12. A log data backup device, characterized in that: include: A collection module, configured to collect a plurality of log records corresponding to each regional service from a pre-written log file of each regional service in a first cluster, wherein each regional service is pre-allocated with at least one pre-allocated storage area, and the first cluster is an HBase cluster; A determination module, configured to determine a target storage area corresponding to each log record from the at least one pre-allocated storage area; A parsing module, configured to parse each log record using a target parsing thread corresponding to the target storage area to obtain log data in each log record; A backup module, used for backing up the log data in the plurality of log records; The determination module is specifically used for: Obtain the storage location of the business data targeted by each log record in the HBase database; Determine a target storage area corresponding to each log record according to the storage location and a storage location range corresponding to each pre-allocated storage area; Each regional service also has a migration storage area, and the migration storage area is a storage area migrated from other regional services; The analysis module is specifically used for: If the target storage area corresponding to the log record to be parsed is the migration storage area, obtaining a first sequence identifier of the most recently parsed log record in the other regional service and a deviation between the first sequence identifier and the second sequence identifier of the log record to be parsed, wherein the first sequence identifier is an identifier of the most recently parsed log record in the migration storage area of ​​the other regional service, and the second sequence identifier is an identifier of the log record to be parsed in the migration storage area of ​​the other regional service; If the deviation is greater than a preset identification unit deviation, the log record before the second sequence identifier is parsed using a parsing thread corresponding to the migration storage area in the other area service; After parsing of the log record before the second sequence identifier is completed, the parsing thread corresponding to the migration storage area is used in each regional service to parse the log record to be parsed to obtain log data in the log record to be parsed.

13. A log data backup device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores a computer program executable by the processor, and when the log data backup device is running, the processor communicates with the memory via the bus, and the processor executes the computer program to execute the method described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the method according to any one of claims 1 to 11.

Images