Data processing method, system and device

Abstract

The embodiments of this specification provide data processing methods, systems and devices, wherein the first data processing method includes a first data storage device receiving first data to be written and obtaining a preset data management file; when the first data is determined to be target data according to the data management file, the first data is written into a log file according to a first preset requirement; when a synchronization device monitors that the first data is written into the log file and determines that the first data is the target data, the synchronization device writes the first data into the second data storage device.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular to a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] Solr (search engine) is a distributed full-text search engine widely used in the industry. Solr supports various complex conditional queries and full-text indexes. By combining with HBase (database), the advantages of HBase and Solr can be maximized, allowing us to build complex big data storage and retrieval services.

[0003] In the prior art, since the search engine needs to update the full amount of data, once it is determined that there is write data in HBase, it is necessary to read back the full amount of data corresponding to the write data before writing it into Solr, resulting in low synchronization efficiency.

[0004] Therefore, it is necessary to provide a data processing method that can synchronize the written data in HBase to the Solr search engine according to user needs without reading it back. Summary of the invention

[0005] In view of this, the embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing device, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a data processing method is provided, including:

[0007] The first data storage device receives the written first data and obtains a preset data management file;

[0008] When it is determined according to the data management file that the first data is target data, writing the first data into a log file according to a first preset requirement;

[0009] The synchronization device writes the first data into the second data storage device when monitoring that the first data is written into the log file and determining that the first data is the target data.

[0010] According to a second aspect of an embodiment of this specification, there is provided a data processing system, including:

[0011] A first data storage device, a synchronization device and a second data storage device, wherein:

[0012] The first data storage device is configured to receive the written first data and obtain a preset data management file;

[0013] When it is determined according to the data management file that the first data is target data, writing the first data into a log file according to a first preset requirement;

[0014] The synchronization device is configured to write the first data into the second data storage device when monitoring that the first data is written into the log file and determining that the first data is the target data.

[0015] According to a third aspect of an embodiment of this specification, a data processing method is provided, including:

[0016] Monitor log files;

[0017] In a case where it is monitored that first data received by the first data storage device is written into the log file, acquiring the first data;

[0018] The first data is determined to be target data, and the first data is written into a second data storage device.

[0019] According to a fourth aspect of an embodiment of this specification, there is provided a data processing device, including:

[0020] A monitoring module, configured to monitor log files;

[0021] An acquisition module is configured to acquire the first data when monitoring that the first data received by the first data storage device is written into the log file;

[0022] The writing module is configured to determine that the first data is target data and write the first data into a second data storage device.

[0023] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0024] Memory and processor;

[0025] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the data processing method when executing the computer-executable instructions.

[0026] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0027] An embodiment of the present specification implements a data processing method, system and device, wherein the first data processing method includes a first data storage device receiving first data to be written and obtaining a preset data management file; when the first data is determined to be target data according to the data management file, the first data is written into a log file according to a first preset requirement; when a synchronization device monitors that the first data is written into the log file and determines that the first data is the target data, the first data is written into the second data storage device;

[0028] The first data storage device of the data processing method stores the write data (first data) to be written to the second data device in a log file according to a special format. After monitoring the write data in the log file, the synchronization module can directly obtain the write data from the log file, parse it and synchronize it to the second data storage device, without having to read back the data at the intersection of the index column to be written to the second data device and the row corresponding to the write data from the first data storage device, thereby improving the synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a data processing system implementing an HBase+Solr / ES architecture provided by an embodiment of this specification;

[0030] Figure 2 is a flow chart of a data processing method provided by an embodiment of this specification;

[0031] Figure 3 This is a schematic diagram of a specific scenario of writing first data into a log file in a data processing method provided by an embodiment of this specification;

[0032] Figure 4 This is a schematic diagram of a writing format of first data written into a log file according to a first preset requirement in a data processing method provided by an embodiment of this specification;

[0033] Figure 5 is a processing schematic diagram of a synchronization device in a data processing method provided by an embodiment of this specification;

[0034] Figure 6 is a flowchart of a data processing method provided by an embodiment of the present specification applied to a synchronization device;

[0035] Figure 7 is a structural diagram of a data processing system provided by an embodiment of this specification;

[0036] Figure 8 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0037] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0038] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0039] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] First, the terms involved in one or more embodiments of this specification are explained.

[0042] Zookeeper: An open source distributed coordination system. It is a software that provides consistency services for distributed applications. Its functions include: configuration maintenance, domain name services, distributed synchronization, group services, etc.

[0043] HBase: Database, HBase is an open source NoSQL (NoSQL = Not Only SQL, meaning "not only SQL") database.

[0044] RegionServer: It is the server that provides services in HBase. It is the most important component in HBase and is responsible for the actual reading and writing of table data.

[0045] Replication: Replication is used in HBase. It is a method of synchronization between HBase clusters, that is, the data in Hlog (i.e., HBase's log file) is asynchronously sent to the standby cluster.

[0046] Solr: Search engine, a full-text search engine based on Lucene.

[0047] ES: English full name: Elasticsearch, English abbreviation: ES, Lucene-based search server ElasticSearch: ES for short, like Solr, is another full-text search engine based on Lucene.

[0048] WAL: Write-Ahead-Log, that is, before writing data in HBase, a log will be written to the disk to prevent data loss. This log is called WAL.

[0049] Synchronization Schema: a description file used by users to specify which columns in HBase are synchronized to Solr / ES.

[0050] Indexing: specifically refers to the process of synchronizing HBase data to the search engine for indexing and querying.

[0051] In this specification, a data processing method is provided. One or more embodiments of this specification simultaneously relate to a data processing system, a data processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0052] Solr / ES is a distributed full-text search engine widely used in the industry. Solr / ES supports various complex conditional queries and full-text indexes. By combining HBase, the advantages of HBase and Solr / ES can be maximized, so that complex big data storage and retrieval services can be built. HBase+Solr / ES is suitable for: application scenarios where large amounts of data need to be stored, while the field data of the query condition only accounts for a small part of the original data, and various condition combination queries are required, such as:

[0053] Common logistics application scenarios: A large amount of track logistics information needs to be stored, and data needs to be queried based on any combination of query conditions of multiple fields;

[0054] Traffic monitoring application scenario: save a large number of vehicle passing records, and retrieve interesting records based on any combination of vehicle information conditions;

[0055] Various website member and product information retrieval scenarios: generally save a large amount of product / member information, and need to perform complex and arbitrary queries based on a small number of conditions to meet the arbitrary search needs of website users, etc.

[0056] Based on this, the embodiments of this specification provide a data processing method for synchronously writing the data written in Hbase to the Solr / ES search engine according to user needs, so as to realize an application scenario that can both save large amounts of data and support various condition combination queries.

[0057] See also Figure 1 , Figure 1 A schematic diagram of a data processing system implementing an HBase+Solr / ES architecture according to an embodiment of the present specification is shown, including: a database 102 (i.e., HBase), a synchronization module 104, and a search engine 106 (i.e., Solr / ES), wherein HBase includes a service module 1022 (i.e., RegionServer in Hbase) and an index module 1024.

[0058] First, the user writes data to the database 102. After receiving the data written by the user, the service module 1022 of the database 102 synchronously obtains a preset data management file (i.e., synchronization Schema) through the index module 1024. The data management file stores description information used by the user to specify which columns in the database 102 are synchronized to the search engine 106. Through the data management file, it can be determined whether the data written by the user is to be synchronously written to the search engine 106. If so, the data written by the user is converted into log data in a special format and stored in the write-ahead log; if not, the data written by the user is converted into log data in a common format and stored in the write-ahead log.

[0059] The synchronization module 104 monitors the write-ahead log in real time or at a preset time interval. When it detects that there is newly written log data in the write-ahead log, the newly written log data will be read out. At the same time, the index module 1024 of the database 102 will also synchronize the data management file to the synchronization module 104. At this time, the synchronization module 104 determines whether the newly written log data is to be synchronously written into the search engine 106 according to the data management file. If so, the synchronization module 104 parses the newly written log data and performs format conversion and writes it into the search engine 106. If not, the synchronization module 104 does not process the newly written log data.

[0060] In another feasible solution, when the synchronization module 104 detects the presence of newly written log data in the write-ahead log, the synchronization module 104 can directly determine whether the newly written log data is to be written into the search engine 106 by judging the storage format of the newly written log data. For example, the synchronization module 104 can synchronously write the newly written log data in a special format into the search engine 106, but does not perform any processing on the newly written log data in an ordinary format.

[0061] Specifically, in the embodiments of the present specification, the data written by the user is written into the write-ahead log in a special format, that is, the data written by the user is written into the write-ahead log in the format of: the original write-ahead log of the database + write timestamp + cross data of the index column to be written into the search engine and the row corresponding to the data written by the user; and the data written by the user is written into the write-ahead log in a normal format, that is, the data written by the user is directly written into the write-ahead log, without adding a write timestamp and the original write-ahead log, etc.

[0062] The data processing system provided in the embodiment of this specification supports multiple search engines such as Solr / ES. No matter how many tables are synchronized, it only needs to read WAL once, and the synchronization module 104 will automatically filter out the data of the table that the user needs to synchronize, and synchronize it to the search engine 106, and no longer pass through the Replication link of HBase102, but directly read WAL, which greatly improves the synchronization efficiency. At the same time, the synchronization module 104 can be infinitely expanded and freely scalable independently of HBase102 to cope with the problem of insufficient synchronization capability when the write volume of HBase102 is large; in addition, since the index module 1024 has written the full amount of columns that need to be indexed into WAL, the synchronization module 106 does not need to read back, and will not affect the HBase102 cluster. The index module 1024 handles many complex issues such as HBase multiple versions, column timestamps, deletions, etc. inside HBase102, and there will be no data inconsistency problems.

[0063] See also Figure 2 , Figure 2 A flowchart of a data processing method provided according to an embodiment of the present specification is shown, comprising the following steps:

[0064] Step 202: The first data storage device receives the written first data and obtains a preset data management file.

[0065] Among them, the first data storage device can be understood as the above-mentioned database HBase, that is, a distributed database, and the first data can be either updated data or newly written data; for example, the first data pre-existing in the table of the distributed database is: Xiao a’s mobile phone number 131***, and the first data received and written by the first data storage device is: Xiao a’s mobile phone number 132***. At this time, it can be determined that the first data received by the first data storage device is updated data, that is, Xiao a’s mobile phone number is updated from 131*** to 132***; if the first data did not exist in the table of the distributed database before, the first data is the newly written data.

[0066] In practical applications, when the first data storage device is a distributed database, correspondingly, the first data storage device receives the written first data, and obtaining the preset data management file includes:

[0067] The distributed database receives the written first data and obtains a preset data management file.

[0068] Specifically, when the first data storage device is a distributed database, in the embodiment of this specification, the corresponding second data storage device is a search engine. Specifically, writing the first data into the second data storage device includes:

[0069] The first data is written into the search engine.

[0070] Among them, search engines include Solr and ES.

[0071] In the embodiment of this specification, the first data storage device is a distributed database, and the second data storage device is a search engine. This HBase+Solr / ES structure is used to support the embodiment of this specification to provide a data processing method for synchronously writing data written in Hbase to the Solr / ES search engine according to user needs, thereby achieving both the ability to save large amounts of data and support various condition combination queries.

[0072] Specifically, the first data storage device receives the first written data, which can be understood as the database receiving the first data updated or newly written by the user. For example, if the database stores the company's employee personal information file table, then what the database receives may be the employee's updated mobile phone number or the personal information of a new employee.

[0073] Among them, the preset data management file can refer to the description of the data management file (i.e., synchronization Schema) in the above embodiment. During the specific implementation, the data management file can store: description information used by the user to specify which columns in the database are synchronized to the search engine (i.e., the second data storage device) or description information used by the user to specify which types of data in the database are synchronized to the search engine, etc.

[0074] Still taking the company employee personal information file table stored in the database as an example, the data management file may include: synchronizing the mobile phone number column and name column in the database to the search engine's description information, or synchronizing the numerical data in the database (such as mobile phone number, age, height, weight, etc.) to the search engine's description information, etc.

[0075] Step 204: When it is determined according to the data management file that the first data is target data, the first data is written into a log file according to a first preset requirement.

[0076] Specifically, after the first data storage device receives the written first data, the first data may be parsed to determine a column or a type corresponding to the first data.

[0077] After determining the column or type corresponding to the first data, it can be determined whether the first data is the target data based on the description information in the data management file. That is, when the column corresponding to the first data matches the column to be written to the second data storage device in the data management file, it can be determined that the first data is the target data, or when the type corresponding to the first data matches the type of target data to be written to the second data storage device in the data management file, it can also be determined that the first data is the target data, wherein the target data is the data to be written to the second data storage device.

[0078] In a specific implementation, the data management file includes an index column;

[0079] Determining the first data as target data according to the data management file includes:

[0080] According to the data management file, a column corresponding to the first data is determined as the index column, and the first data is determined as the target data.

[0081] The data management file includes index columns, which can be understood as the data management file including which index columns the user specifies need to be synchronized to the second data storage device.

[0082] Specifically, the first data is first parsed to determine which column the first data corresponds to. If the column corresponds to an index column in the data management file, it can be determined that the first data is data to be written into the second data storage device, that is, target data.

[0083] In the embodiments of the present specification, by matching the index column in the data management file with the column corresponding to the first data, it is possible to quickly determine whether the first data is the target data, so that the first data as the target data can be subsequently written into the log file according to the first preset requirements, thereby realizing the first screening of the target data written into the second data storage device.

[0084] In another possible implementation of this specification, the data management file includes a target data type;

[0085] Determining the first data as target data according to the data management file includes:

[0086] Determine, according to the data management file, that the type corresponding to the first data is the target data type, and determine the first data as the target data.

[0087] The data management file includes the target data type, which can be understood as including in the data management file which types of data specified by the user need to be synchronized to the second data storage device.

[0088] Specifically, the first data is first parsed to determine the type corresponding to the first data, such as a numeric type, a character type, etc. If the target data type included in the data management file is a numeric type, and the type corresponding to the first data is also a numeric type, then it can be determined that the first data is data to be written to the second data storage device, that is, the target data; and if the type corresponding to the first data is a character type, it can be determined that the first data is not data to be written to the second data storage device, and is not the target data.

[0089] In the embodiments of the present specification, by matching the target data type in the data management file with the type corresponding to the first data, it is possible to quickly determine whether the first data is the target data, so that the first data as the target data can be subsequently written into the log file according to the first preset requirements, thereby realizing the first screening of the target data written into the second data storage device.

[0090] In practical applications, when the first data is determined to be target data according to any of the above methods, the first data can be written into the log file according to the first preset requirement.

[0091] In another embodiment of the present specification, writing the first data into a log file according to a first preset requirement includes:

[0092] Determine second data according to the data management file and the first data;

[0093] Acquire a log file in which the first data is not written, a writing timestamp of the first data, and the second data;

[0094] The log file in which the first data is not written, the writing timestamp of the first data, and the second data are combined and written into the log file.

[0095] The log file into which the first data is not written can be understood as the original log file before the first data is written.

[0096] Specifically, determining the second data according to the data management file and the first data includes:

[0097] Determine an index column according to the data management file, and determine a row corresponding to the first data;

[0098] The data at the intersection of the row corresponding to the first data and the index column is determined as the second data.

[0099] Continuing with the above example, if the index columns in the data management file are the mobile phone number column and the name column, the column corresponding to the first data is the mobile phone number column, and the corresponding row is the first row of the mobile phone number column, then the second data is the row data at the intersection of the first row and the mobile phone number column and the name column, that is, the data of the first row, the mobile phone number column and the data of the first row of the name column.

[0100] Specifically, after determining that the first data is the target data to be written to the second data storage device, the original log file in which the first data is not written, the write timestamp of the first data, and the second data are obtained, and then these data are combined to form new log data in a special format and written into the log file to ensure that the subsequent synchronization device can directly synchronize and write to the search engine based on the log data in the special format, thereby improving synchronization efficiency.

[0101] During specific implementation, the first data storage device will receive the first data written by the user in real time. If the first data storage device has written three versions of the first data in the write-ahead log before the synchronization device writes the first data to the second data storage device, then when the synchronization device writes the first data to the second data storage device, it is possible that the first data first written to the write-ahead log will be synchronized to the second data storage device. In fact, it is necessary to synchronize the first data last written to the write-ahead log to the second data storage device to ensure the accuracy of the data. Therefore, based on this situation, the first data storage device will add a write timestamp to each first data, and the subsequent synchronization device can select the last written first data according to the write timestamp and synchronize it to the second data storage device.

[0102] See also Figure 3 , Figure 3 A schematic diagram of a specific scenario of writing first data into a log file in a data processing method provided in one or more embodiments of this specification.

[0103] Step 1: The user writes the first data as row1, and the column name is f:q1;

[0104] Specifically, the first data written by the user is row1, and the column name is f:q1, which can be understood as the index module of the first data storage device determines that the position of the first data written by the user received by the first data storage device is the first row, and the column name is f:q1.

[0105] Step 2: Get the data management file and find out that the two columns f:q1 and f:q2 need to be synchronized;

[0106] Specifically, the data management file is obtained to learn that the two columns f:q1 and f:q2 need to be synchronized, which can be understood as the index module of the first data storage device obtaining the synchronization Schema information, and through the synchronization Schema information, it is learned that the two columns f:q1 and f:q2 need to be synchronously written to the second data storage device.

[0107] Step 3: Get the data in the column f:q2 and complete the entire row;

[0108] Specifically, obtaining the data of the column f:q2 and completing the entire row can be understood as the index module of the first data storage device obtaining the data of the column f:q2 in row1 from the service module and completing row1.

[0109] Step 4: Write the entire line of data into the log file;

[0110] Specifically, writing the entire row of data into the log file can be understood as the index module writing the data of row1 in the two columns f:q1 and f:q2 into the log file WAL.

[0111] In specific implementation, when writing a whole line of data into a log file, the index module of the first data storage device will change the format of the written WAL. The new WAL format is shown in Figure 4 In addition to the original HBase WAL information, the index module will add a current timestamp (i.e., the write timestamp of the first data) to the WAL, and attach the cross position data of the index column to be written to the search engine and the row corresponding to the first data to the WAL. Among them, the current timestamp is used to determine whether it is the latest data when writing to Solr. This is the key to ensuring that Solr data will not be wrong when dealing with HBaseWAL disorder, and writing the entire row of data to the WAL is the key to preventing the synchronization module from reading back.

[0112] See also Figure 4 , Figure 4 A schematic diagram of a writing format of first data written into a log file according to a first preset requirement in a data processing method provided in one or more embodiments of this specification.

[0113] Still taking the above example, Figure 4 It includes three parts. The first part is the original WAL information, that is, the log file before the first data is written, the HBase original WAL, the second part is the timestamp, the current timestamp, that is, the write timestamp when the first data is written, and the third part is<row1,f:q1,f:q2> , that is, the row data that needs to be written into the search engine's index column and the row where the data is written intersects.

[0114] In an embodiment of the present specification, when the first data is the target data to be written to the search engine, the database will combine the original log file, the write timestamp of the first data, and the data at the intersection of the index column and the row of the write data that needs to be written to the search engine, and write them into the log file to form new log data. When the subsequent synchronization device synchronously writes the new log data to the search engine, it can determine the latest data based on the current timestamp in the new log data to avoid disorder of the data written to the search engine, and based on the original log file in the new log data, it can determine the row data corresponding to the index column in the data management file, without the need to read back the data, thereby improving synchronization efficiency.

[0115] In another embodiment of the present specification, after the first data storage device receives the first data to be written and obtains the preset data management file, it also includes:

[0116] When it is determined according to the data management file that the first data is non-target data, the first data is written into a log file according to a second preset requirement.

[0117] Specifically, the second preset requirement is different from the first preset requirement. Therefore, the format of the first data written to the log file according to the first preset requirement is different from the format of the first data written to the log file according to the second preset requirement. The first data of non-target data is written to the log file according to the second preset requirement, so that the target data and the non-target data can be distinguished only by the data format of the first data in the log file.

[0118] In actual applications, writing the first data into the log file according to the second preset requirement can be understood as writing only the first data or only the first data and the write timestamp of the first data into the log file. Since the first data of non-target data does not need to be written to the search engine synchronously, writing only the first data into the log file can reduce the amount of written data and reduce the storage space utilization rate of the log file. In addition, in another implementable scheme, writing the first data and the write timestamp of the first data into the log file together can facilitate subsequent query of the latest written data of the log file based on the write timestamp, thereby improving user experience.

[0119] Step 206: When the synchronization device monitors that the first data is written into the log file and determines that the first data is the target data, the synchronization device writes the first data into the second data storage device.

[0120] Among them, the synchronization device in the embodiments of this specification can refer to the introduction of the synchronization device in the above embodiments, and will not be repeated here.

[0121] The synchronization device will monitor the log file in real time or at a preset time interval (for example, every 2 seconds, etc.). When it is monitored that the first data is written into the log file, the synchronization device will judge the first data. When the first data is determined to be the target data, the first data will be written into the second data storage device.

[0122] In the embodiment of this specification, there are at least two solutions for determining whether the first data written into the log file is the target data, and the specific implementation methods are as follows:

[0123] In a first aspect, determining the first data as the target data includes:

[0124] Acquire the first data written into the log file, and determine a storage format of the first data in the log file;

[0125] In a case where the storage format is the same as a writing format in which the first data is written into the log file according to the first preset requirement, the first data is determined to be target data.

[0126] Specifically, when the synchronization device monitors that the first data is written into the log file, the synchronization device obtains the first data from the log file and parses the first data to determine the storage format of the first data in the log file; then the storage format of the first data in the log file is compared with the writing format of the first data written into the log file according to the first preset requirement. If the formats are the same, the first data is determined to be the target data; if the formats are different, it can be determined that the first data is not written into the log file according to the first preset requirement, that is, it is non-target data.

[0127] The data processing method provided in the embodiment of this specification can determine whether the first data is the target data only by the storage format of the first data in the log file, and implement a second screening of the target data to determine the accuracy of the target data written into the search engine.

[0128] Second, the determining that the first data is the target data includes:

[0129] acquiring the first data written to the log file, and acquiring the data management file;

[0130] The first data written into the log file is determined as the target data according to the data management file.

[0131] In a specific implementation, determining, according to the data management file, that the first data written into the log file is the target data includes:

[0132] When it is determined according to the data management file that a column corresponding to the first data written into the log file is an index column, determining the first data as the target data; or

[0133] When it is determined according to the data management file that the type corresponding to the first data written into the log file is a target data type, the first data is determined to be the target data.

[0134] Specifically, when the synchronization device monitors that the first data is written into the log file, it obtains the first data from the log file and simultaneously obtains the data management file, and determines that the first data is the target data when the column corresponding to the first data written into the log file is the index column or the type corresponding to the first data is the target data type according to the index column or the target data type in the data management file.

[0135] The data processing method provided in the embodiment of this specification determines whether the first data is the target data through the data management file, and implements a second screening of the target data to determine the accuracy of the target data written into the search engine.

[0136] See also Figure 5 , Figure 5 It is a processing schematic diagram showing a synchronization device in a data processing method provided by an embodiment of this specification.

[0137] In practical applications, the synchronization device is an open source distributed coordination system, such as Zookeeper, wherein the distributed coordination system Zookeeper includes a master node (Master node) and a worker node (Worker node). In specific implementation, the Master node is responsible for polling the WAL (Write-Ahead Log) file generated by HBase (first data storage device). If there is new log data in the WAL, the Master node will assign the synchronization task of the new log data of the WAL to the Worker node when determining that the new log data is to be written to the search engine (second data storage device). The Worker node then converts the new log data received by HBase and written to the WAL into data that can be recognized by Solr / ES (search engine) and writes it to the search engine. That is, in the embodiment of this specification, the synchronization device uses Zookeeper for distributed coordination and task persistence storage, so that the synchronization device can be infinitely expanded and freely scalable independent of HBase to cope with the problem of insufficient synchronization capability when the amount of HBase writes is large.

[0138] In the data processing method provided in the embodiments of the present specification, after receiving the first data which is the target data, the first data storage device stores the first data in a log file in a special format. After the synchronization module monitors the first data written into the log file and needs to be synchronously written into the search engine, the first data stored in the log file in a special format can be directly obtained from the log file, parsed and synchronously written into the search engine without reading back the first data in the database, thereby improving the synchronization efficiency. In addition, the first data stored in the log file in a special format also carries a write timestamp. When it is synchronously written into the search engine, the write timestamp can be used to ensure that the first data synchronously written into the search engine is the latest data, thereby avoiding data confusion.

[0139] See also Figure 6 , Figure 6 A flowchart of a data processing method provided by an embodiment of the present specification is applied to a synchronization device, which specifically includes the following steps:

[0140] Step 602: Monitor log files.

[0141] Specifically, the log file can be understood as the log file in which the database writes the first data in the above embodiment.

[0142] In actual applications, in order to ensure that the data written in the log file is synchronized to the search engine in a timely manner, the log file can be monitored in real time; and in general, real-time monitoring of the log file may occupy a large amount of network resources. Therefore, in order to ensure that the data in the log file is synchronized to the search engine and save network resources, the data in the log file is obtained once every period of time and synchronized to the search engine. The specific implementation method is as follows:

[0143] The monitoring log file includes:

[0144] Monitor log files based on preset time intervals.

[0145] Among them, the time interval can be set according to actual needs, for example, set to 2 seconds, 3 seconds, etc., and this application does not impose any limitation on this.

[0146] Step 604: When it is monitored that the first data received by the first data storage device is written into the log file, the first data is acquired.

[0147] The first data storage device may be understood as the distributed database of the above embodiment, namely HBase, and the first data is the first data written into the first data storage device.

[0148] Specifically, the synchronization device monitors the log file in real time or at a preset time interval, and if it is detected that the first data received by the first data storage device is written into the log file, the first data is obtained from the log file.

[0149] Step 606: Determine that the first data is target data, and write the first data into a second data storage device.

[0150] The target data is data to be written into the second data storage device, and the manner of determining whether the first data is the target data includes at least two methods:

[0151] The first type: determining that the first data is target data includes:

[0152] determining a storage format of first data written to the log file in the log file;

[0153] In a case where the storage format is the same as a writing format in which the first data is written into the log file according to a first preset requirement, the first data is determined to be target data.

[0154] Specifically, after obtaining the first data, the first data is parsed to determine the storage format of the first data written into the log file in the log file; when it is determined that the storage format is the same as the writing format of the first data written into the log file according to the first preset requirement, the first data is determined to be the target data.

[0155] The first preset requirement includes combining a log file in which the first data is not written, a writing timestamp of the first data, and second data determined according to a preset data management file and the first data.

[0156] Then the writing format of the first data written into the log file according to the first preset requirement is: the log file in which the first data is not written + the writing timestamp of the first data + the second data, wherein the second data determined according to the preset data management file and the first data includes:

[0157] Determine an index column according to the data management file, and determine a row corresponding to the first data;

[0158] The data at the intersection of the row corresponding to the first data and the index column is determined as the second data.

[0159] In specific implementation, after parsing the first data, it is determined that the storage format of the first data written to the log file in the log file is: the log file in which the first data is not written + the write timestamp of the first data + the second data; at this time, it can be determined that the storage format of the first data in the log file is the same as the write format written into the log file, and the first data can be determined to be the target data.

[0160] In the embodiments of the present specification, by comparing the storage format of the first data in the log file with the writing format of the first data written into the log file, it is possible to quickly and accurately determine whether the first data is the target data, so that the first data as the target data can be subsequently written into the second data storage device.

[0161] Second type: determining the first data as target data includes:

[0162] Acquire the first data written into the log file, and acquire a preset data management file, wherein the data management file includes an index column;

[0163] When it is determined according to the data management file that the column corresponding to the first data written into the log file is the index column, the first data is determined to be the target data.

[0164] The detailed description of the data management file can be found in the above embodiment and will not be elaborated here.

[0165] Specifically, the first data written into the log file is first parsed to determine which column the first data corresponds to. If the column corresponding to the first data is an index column, it can be determined that the first data is the target data, that is, the data to be written into the second data storage device.

[0166] In the embodiment of the present specification, by matching the index column in the data management file with the column corresponding to the first data, it can be quickly determined whether the first data is the target data, so that the first data as the target data can be subsequently written into the second data storage device.

[0167] The data processing method provided in the embodiment of this specification, after monitoring the existence of newly written log data in the WAL of HBase, first judges the newly written log data, and after determining that the newly written log data is data to be written to the search engine, directly reads the newly written log data in the WAL and synchronizes it to the search engine, thereby realizing a low-latency and high-performance solution for synchronizing HBase data with the search engine, and the data corresponding to the column to be written to the search engine is directly written into the WAL, and readback is avoided during the synchronization process, thereby improving the synchronization efficiency, and at the same time, the latest written data can be synchronized with the search engine based on the write timestamp in the newly written log data, thereby avoiding data confusion.

[0168] Corresponding to the above method embodiment, this specification also provides a data processing system embodiment. Figure 7 FIG. 1 shows a schematic diagram of a data processing system provided by an embodiment of the present specification. Figure 7 As shown, the system includes:

[0169] A first data storage device 702, a synchronization device 704 and a second data storage device 706, wherein:

[0170] The first data storage device 702 is configured to receive the written first data and obtain a preset data management file;

[0171] When it is determined according to the data management file that the first data is target data, writing the first data into a log file according to a first preset requirement;

[0172] The synchronization device 704 is configured to write the first data into the second data storage device 706 when monitoring that the first data is written into the log file and determining that the first data is the target data.

[0173] Optionally, the data management file includes an index column;

[0174] The first data storage device 702 is further configured to:

[0175] According to the data management file, a column corresponding to the first data is determined as the index column, and the first data is determined as the target data.

[0176] Optionally, the data management file includes a target data type;

[0177] The first data storage device 702 is further configured to:

[0178] Determine, according to the data management file, that the type corresponding to the first data is the target data type, and determine the first data as the target data.

[0179] Optionally, the synchronization device 704 is further configured to:

[0180] Acquire the first data written into the log file, and determine a storage format of the first data in the log file;

[0181] In a case where the storage format is the same as a writing format in which the first data is written into the log file according to the first preset requirement, the first data is determined to be target data.

[0182] Optionally, the synchronization device 704 is further configured to:

[0183] acquiring the first data written to the log file, and acquiring the data management file;

[0184] The first data written into the log file is determined as the target data according to the data management file.

[0185] Optionally, the device further includes:

[0186] The writing device is configured to write the first data into a log file according to a second preset requirement when it is determined according to the data management file that the first data is non-target data.

[0187] Optionally, the first data storage device 702 is further configured to:

[0188] Determine second data according to the data management file and the first data;

[0189] Acquire a log file in which the first data is not written, a writing timestamp of the first data, and the second data;

[0190] The log file in which the first data is not written, the writing timestamp of the first data, and the second data are combined and written into the log file.

[0191] Optionally, the first data storage device 702 is further configured to:

[0192] Determine an index column according to the data management file, and determine a row corresponding to the first data;

[0193] The data at the intersection of the row corresponding to the first data and the index column is determined as the second data.

[0194] Optionally, the first data storage device is a distributed database, and the second data storage device is a search engine;

[0195] Accordingly, the first data storage device 702 is further configured as follows:

[0196] The distributed database receives the written first data and obtains a preset data management file;

[0197] Accordingly, the synchronization device 704 is further configured as follows:

[0198] The first data is written into the search engine.

[0199] In the data processing system provided by the embodiment of the present specification, after receiving the first data which is the target data, the first data storage device stores the first data in a log file in a special format. After the synchronization module monitors the first data written into the log file and needs to be synchronously written into the search engine, the first data stored in the log file in a special format can be directly obtained from the log file, parsed and synchronously written into the search engine, without having to read back the data at the intersection of the index column of the second data device to be written and the row corresponding to the written data in the database, thereby improving the synchronization efficiency. The first data stored in the log file in a special format also carries a write timestamp. When it is synchronously written into the search engine, the write timestamp can be used to ensure that the first data synchronously written into the search engine is the latest data, thereby avoiding data confusion.

[0200] The above is a schematic scheme of a data processing system of this embodiment. It should be noted that the technical scheme of the data processing system and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the data processing system can be referred to the description of the technical scheme of the above data processing method.

[0201] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 8 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0202] A monitoring module 802 is configured to monitor log files;

[0203] The acquisition module 804 is configured to acquire the first data when monitoring that the first data received by the first data storage device is written into the log file;

[0204] The writing module 806 is configured to determine that the first data is target data and write the first data into a second data storage device.

[0205] Optionally, the monitoring module 802 is further configured to:

[0206] Monitor log files based on preset time intervals.

[0207] Optionally, the writing module 806 is further configured to:

[0208] determining a storage format of first data written to the log file in the log file;

[0209] In a case where the storage format is the same as a writing format in which the first data is written into the log file according to a first preset requirement, the first data is determined to be target data.

[0210] Optionally, the writing module 806 is further configured to:

[0211] Acquire the first data written into the log file, and acquire a preset data management file, wherein the data management file includes an index column;

[0212] When it is determined according to the data management file that the column corresponding to the first data written into the log file is the index column, the first data is determined to be the target data.

[0213] The data processing device provided in the embodiment of the present specification, after monitoring the existence of newly written log data in the WAL of HBase, first judges the newly written log data, and after determining that the newly written log data is data to be written to the search engine, directly reads the newly written log data in the WAL and synchronizes it to the search engine, thereby realizing a low-latency and high-performance solution for synchronizing HBase data with the search engine, and the data corresponding to the column to be written to the search engine is directly written into the WAL, and readback is avoided during the synchronization process, thereby improving the synchronization efficiency, and at the same time, the latest written data can be synchronized with the search engine based on the write timestamp in the newly written log data, thereby avoiding data confusion.

[0214] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0215] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0216] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0217] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0218] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 900 may also be a mobile or stationary server.

[0219] The processor 920 is used to execute the following computer executable instructions, wherein the processor implements the steps of the data processing method when executing the computer executable instructions.

[0220] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0221] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the data processing method when executed by a processor.

[0222] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0223] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0224] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0225] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0226] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0227] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: The first data storage device receives the written first data and obtains a preset data management file, wherein the data management file includes description information of target data that needs to be synchronized to the second data storage device, the first data storage device is a distributed database, and the second data storage device is a search engine; In the case where the first data is determined as target data according to the data management file, the first data is written into a log file according to a first preset requirement, wherein the writing of the first data into the log file according to the first preset requirement comprises: determining second data according to the data management file and the first data; obtaining a log file into which the first data is not written, a write timestamp of the first data, and the second data; combining the log file into which the first data is not written, the write timestamp of the first data, and the second data, and writing them into the log file, wherein the log file into which the first data is not written is an original log file before the first data is written, and the log data of the original log file is used to enable the second data storage device to obtain row data corresponding to the first data, and the second data is intersection data determined according to the row corresponding to the first data and the index column determined by the data management file; The synchronization device writes the first data into the second data storage device when monitoring that the first data is written into the log file and determining that the first data is the target data.

2. The data processing method according to claim 1, wherein the data management file comprises an index column; Determining the first data as target data according to the data management file includes: According to the data management file, a column corresponding to the first data is determined as the index column, and the first data is determined as the target data.

3. The data processing method according to claim 1, wherein the data management file includes a target data type; Determining the first data as target data according to the data management file includes: Determine, according to the data management file, that the type corresponding to the first data is the target data type, and determine the first data as the target data.

4. The data processing method according to claim 1, wherein determining the first data as the target data comprises: Acquire the first data written into the log file, and determine a storage format of the first data in the log file; In a case where the storage format is the same as a writing format in which the first data is written into the log file according to the first preset requirement, the first data is determined to be target data.

5. The data processing method according to claim 2 or 3, wherein determining the first data as the target data comprises: acquiring the first data written to the log file, and acquiring the data management file; The first data written into the log file is determined as the target data according to the data management file.

6. The data processing method according to claim 1, after the first data storage device receives the first data to be written and obtains the preset data management file, further comprising: When it is determined according to the data management file that the first data is non-target data, the first data is written into a log file according to a second preset requirement.

7. The data processing method according to claim 1, wherein determining the second data according to the data management file and the first data comprises: Determine an index column according to the data management file, and determine a row corresponding to the first data; The data at the intersection of the row corresponding to the first data and the index column is determined as the second data.

8. The data processing method according to claim 1, wherein the first data storage device receives the written first data, and obtaining the preset data management file comprises: The distributed database receives the written first data and obtains a preset data management file; Accordingly, writing the first data into the second data storage device includes: The first data is written into the search engine.

9. A data processing system comprising: A first data storage device, a synchronization device and a second data storage device, wherein: The first data storage device is configured to receive the written first data and obtain a preset data management file, wherein the data management file includes description information of the data to be synchronized to the second data storage device; In the case where the first data is determined as the target data according to the data management file, the first data is written into a log file according to a first preset requirement, wherein the first data storage device is a distributed database, and the second data storage device is a search engine, and the writing of the first data into the log file according to the first preset requirement comprises: determining the second data according to the data management file and the first data; obtaining a log file into which the first data is not written, a write timestamp of the first data, and the second data; combining the log file into which the first data is not written, the write timestamp of the first data, and the second data, and writing them into the log file, wherein the log file into which the first data is not written is the original log file before the first data is written, and the log data of the original log file is used to enable the second data storage device to obtain the row data corresponding to the first data, and the second data is the intersection data determined according to the row corresponding to the first data and the index column determined by the data management file; The synchronization device is configured to write the first data into the second data storage device when monitoring that the first data is written into the log file and determining that the first data is the target data.

10. A data processing method, comprising: Monitor log files; In the case where it is monitored that the first data received by the first data storage device is written into the log file, the first data is obtained, wherein the first data is written into the log file according to the first preset requirement based on the first data storage device determining that the first data is the target data according to the data management file, the data management file contains description information of the data that needs to be synchronized to the second data storage device, and the case where the first data is determined to be the target data according to the data management file, the writing of the log file according to the first preset requirement includes: determining the second data according to the data management file and the first data; obtaining the log file to which the first data is not written, the write timestamp of the first data, and the second data; combining the log file to which the first data is not written, the write timestamp of the first data, and the second data, and writing them into the log file, the log file to which the first data is not written is the original log file before the first data is written, and the log data of the original log file is used to enable the second data storage device to obtain the row data corresponding to the first data, and the second data is the intersection data determined according to the row corresponding to the first data and the index column determined by the data management file; The first data is determined to be target data, and the first data is written into a second data storage device, the first data storage device is a distributed database, and the second data storage device is a search engine.

11. The data processing method according to claim 10, wherein the monitoring log file comprises: Monitor log files based on preset time intervals.

12. The data processing method according to claim 10, wherein determining the first data as target data comprises: determining a storage format of first data written to the log file in the log file; In a case where the storage format is the same as a writing format in which the first data is written into the log file according to a first preset requirement, the first data is determined to be target data.

13. The data processing method according to claim 10, wherein determining the first data as target data comprises: Acquire the first data written into the log file, and acquire a preset data management file, wherein the data management file includes an index column; When it is determined according to the data management file that the column corresponding to the first data written into the log file is the index column, the first data is determined to be the target data.

14. A data processing device, comprising: A monitoring module, configured to monitor log files; an acquisition module, configured to acquire the first data upon monitoring that the first data received by the first data storage device is written into the log file, wherein the first data is written into the log file according to a first preset requirement based on the first data storage device determining that the first data is target data according to a data management file, the data management file contains description information of data that needs to be synchronized to the second data storage device, and the writing of the log file according to the first preset requirement upon determining that the first data is target data according to the data management file comprises: determining the second data according to the data management file and the first data; acquiring the log file into which the first data is not written, the write timestamp of the first data, and the second data; combining the log file into which the first data is not written, the write timestamp of the first data, and the second data, and writing the log file, wherein the log file into which the first data is not written is the original log file before the first data is written, and the log data of the original log file is used to enable the second data storage device to obtain the row data corresponding to the first data, and the second data is the intersection data determined according to the row corresponding to the first data and the index column determined by the data management file; The writing module is configured to determine that the first data is target data and write the first data into a second data storage device, wherein the first data storage device is a distributed database and the second data storage device is a search engine.

15. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the data processing method described in any one of claims 1-8 or 10-13 when executing the computer-executable instructions.

16. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the data processing method described in any one of claims 1-8 or 10-13.

17. A computer program product, characterized in that The method comprises computer instructions which, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 8 or 10 to 13.

Images