Method, device and system for updating data of hbase server

By using a combination of view map tables and multiple data tables in the Hbase server, the problem caused by data deletion dependency expiration time is solved, data consistency and accuracy are ensured, and data transmission speed in big data collection scenarios and Server-side performance.

CN119025531APending Publication Date: 2024-11-26CHINA CITIC BANK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411042111.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Data deletion in the existing Hbase server relies on expiration time control. When the settings are unreasonable, the data that should be deleted will not be deleted and the data that should not be deleted will be deleted. In the case of big data collection, the writing speed will be slow and the server will be deleted. The pressure is high and the data cannot be guaranteed.

Method used

Create a view mapping table and multiple data tables in the Hbase server. The data generated by the data warehouse is stored in a new data table. The view mapping table always points to the latest data table name. By monitoring and updating the view mapping table regularly, ensure that Read the latest data, avoid updating error accumulation, and reduce server pressure through batch operations.

Benefits of technology

It realizes the overall consistency and accuracy of data in the Hbase server, avoids the problem of data that should not be deleted and data that should be deleted are not deleted, improves data transmission speed, reduces the storage pressure on the server, and is applicable In big data collection scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025531A_ABST
    Figure CN119025531A_ABST
Patent Text Reader

Abstract

The invention relates to the field of databases, and provides an hbase server data updating method, device and system.The method applied to an hbase server comprises the steps that a data writing request sent by a data warehouse is received; establishing a new data table; writing the data sent by the data warehouse into the new data table; and updating the view pointing data table name in the view mapping table into a new data table name, so as to read data according to the view pointing data table name in the view mapping table when a reading request is responded. According to the method, the multiple data tables and the view mapping table are stored in the hbase server side, each data table is used for storing the data sent by the data warehouse at a time, and the view in the view mapping table points to the mode that the data table name is the latest data table name all the time, it can be guaranteed that the latest data is read, and the situation that the data cannot be read does not occur.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of databases, and more particularly to an HBase server data updating method, device and system. Background Art

[0002] Hbase server is a distributed, column-oriented open source database that provides insert / update functions. In the use scenario of batch and stream integration, it is usually necessary to regularly store the data that meets the conditions in the hbase server for related queries. For example, the customer group table is stored in the hbase server so that the customer group can be queried from the hbase server and information recommendations can be made for the customer group. At the same time, the relevant data in the hbase server needs to be updated when the information in the table changes.

[0003] Usually when writing data, the new data is written in batches using the HBase server API. To avoid querying data errors, an expiration time needs to be set for the new data. When updating data, it is updated directly through the HBase server API. When deleting data, no operation is performed, and the expiration time set when the data is first written to the database determines the deletion timing.

[0004] Usually, if the data is updated daily, the data expiration time is set to 25 hours, and the data is updated regularly every 24 hours, and the data to be deleted is automatically deleted.

[0005] The data operation method based on the HBase server API is not suitable for scenarios with large data sets. It has problems of slow speed and high server pressure. In addition, since data deletion is controlled by the expiration time, if the expiration time is set improperly, data that should be deleted will not be deleted and data that should not be deleted will be deleted.

[0006] For example, suppose the scenario is to update the customer data in the hbase server at 10 o'clock every day. Under normal circumstances, data1 generated by the data warehouse is imported into the hbase server one by one at 10 o'clock on the first day, and the expiration time is set to 24 hours; at 10 o'clock on the second day, data2 generated by the data warehouse is imported into the hbase server one by one. At this time, data1 is expired, so the data in the hbase server is all data2.

[0007] If a problem occurs in the data warehouse processing flow, data2 is not processed by 10 o'clock the next day. At this time, data1 has expired and the data in the hbase server is empty. However, the business requires that if no new customer group is processed, the old customer group should continue to be used. At this time, all data1 in the hbase server is expired and deleted, so there is a problem of data that should not be deleted being deleted.

[0008] To solve this problem, the data expiration time is set slightly larger than the update cycle, such as 25 hours. When data2 is updated the next day, data2 is allowed to be imported at 11 o'clock. Within the tolerance of 1 hour, the old data is still available in the hbase server. If d2 updates the data at 10:10, then within the time range of 10:10-11:00, the hbase server has data1+data2 at the same time, not just data2. Therefore, the deleted data is not deleted.

[0009] In addition, if the data on the hbase server is updated and iterated frequently, errors may accumulate and the correctness of the full data cannot be guaranteed. Summary of the invention

[0010] This article is used to solve the problem in the prior art that data deletion relies on expiration time control. If the expiration time is set unreasonably, the data that should be deleted will not be deleted and the data that should not be deleted will be deleted.

[0011] In order to solve the above technical problems, this paper provides an HBase server data update method, which is applied to the HBase server. The HBase server has a view mapping table and a data table for storing the view pointing to the data table name. The method includes:

[0012] Receive data writing requests sent by the data warehouse;

[0013] Create a new data table;

[0014] Writing the data sent by the data warehouse into the new data table;

[0015] The view-pointing data table name in the view mapping table is updated to the new data table name, so that when responding to a read request, data is read according to the view-pointing data table name in the view mapping table.

[0016] As a further embodiment of this invention, the hbase server data update method further includes:

[0017] Receive a data read request;

[0018] Determine the data table name pointed to by the view from the view mapping table;

[0019] According to the data table name pointed to by the view, read the data from the relevant data table on the hbase server;

[0020] Monitor in real time whether the view mapping table is updated. If updated, re-execute the steps of determining the data table name pointed to by the view from the view mapping table and the subsequent steps.

[0021] As a further embodiment of this invention, after writing the data sent by the data warehouse into the new data table, the method further includes:

[0022] Check whether the data in the new data table meets the preset requirements;

[0023] If satisfied, the data table name pointed to by the view in the view mapping table is updated to the new data table name.

[0024] As a further embodiment of this invention, the hbase server data update method further includes:

[0025] Count the number of times the detected data does not meet the preset requirements;

[0026] When the number reaches the preset number, a data abnormality reminder message is issued.

[0027] As a further embodiment of this invention, the view mapping table further stores the historical data table name, and the method further includes:

[0028] Count the number of historical data table names in the view mapping table;

[0029] When the number of historical data table names reaches a preset threshold, the top N data tables in the hbase server are deleted in the order of the data table creation time, where N is less than the preset threshold.

[0030] A second aspect of the present invention provides an HBase server data update method, using an HBase server update system, the method comprising:

[0031] The service end periodically reads the name of the data table pointed to by the view from the view mapping table of the hbase server; when the read data table name is different from the currently used data table name, a new read thread is started, and the data table name is loaded into the new read thread, so that the new read thread reads data from the data table pointed to by the view of the hbase server;

[0032] The data warehouse processes data periodically, converts the processed data into an HBase underlying binary data structure file, and sends the HBase underlying binary data structure file to the HBase server;

[0033] The hbase server creates a new data table, stores the files sent by the data warehouse in the new data table, and updates the data table name pointed to by the view in the view mapping table to the new data table name.

[0034] As a further embodiment of this invention, after the service end starts a new read thread, it also includes:

[0035] Determine whether there is an old read thread. If so, close the old read thread.

[0036] As a further embodiment of this invention, the method further comprises:

[0037] The business end monitors the data read by the reading program, and if the read data is abnormal, sends a read exception request to the hbase server;

[0038] After receiving the read exception request, the hbase server modifies the view pointing data table name in the view mapping table to the view pointing data table name set last time.

[0039] A third aspect of the present invention provides an HBase server data update device, which is applied to an HBase server. A view mapping table and a data table for storing a view pointing to a data table name are established in the HBase server. The device includes:

[0040] A write request receiving unit, used for receiving a data write request sent by the data warehouse;

[0041] A data table creation unit, used to create a new data table;

[0042] A data writing unit, used for writing the data sent by the data warehouse into the new data table;

[0043] The mapping update unit is used to update the view pointing data table name in the view mapping table to a new data table name, so as to read data according to the view pointing data table name in the view mapping table when responding to a read request.

[0044] The fourth aspect of this article provides an hbase server data update system, a business end, an hbase server and a data warehouse;

[0045] The service end is used to periodically read the name of the data table pointed to by the view from the view mapping table of the hbase server; when the read data table name is different from the currently used data table name, a new reading thread is started, and the data table name is loaded into the new reading thread, so that the new reading thread reads data from the data table pointed to by the view of the hbase server;

[0046] The data warehouse is used to periodically process data, convert the processed data into hbase underlying binary data structure files, and send the hbase underlying binary data structure files to the hbase server;

[0047] The hbase server is used to create a new data table, store the files sent by the data warehouse in the new data table, and update the view pointing to the data table name in the view mapping table to the new data table name.

[0048] The hbase server data update method, device and system provided in this article can ensure that the latest data is read and the data cannot be read by storing multiple data tables and view mapping tables in the hbase server. Each data table is used to store the data processed by the data warehouse once. The data table name pointed to by the view in the view mapping table is always the latest data table name. Because the data generated by the data warehouse each time is stored in a new data table, that is, the data is fully updated, the accumulation of update errors can be avoided, and the consistency and accuracy of the overall data can be guaranteed. In addition, since there is no traffic access to the data each time the data table is updated, it can prevent pressure on the performance of the hbase server.

[0049] The data warehouse converts the processed data into hbase underlying binary data structure files, sends the hbase underlying binary data structure files to the hbase server, and the hbase server stores the underlying binary data structure files in a new data table. This can be applied to scenarios with large data sets, improve data transmission speed, and reduce the pressure on hbase server to store data.

[0050] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 The structure diagram of the hbase server data update system in the embodiment of this article is shown;

[0053] Figure 2 The first interactive flow chart of the hbase server data update system in the embodiment of this article is shown;

[0054] Figure 3 The second interactive flow chart of the hbase server data update system in the embodiment of this article is shown;

[0055] Figure 4 The third interactive flow chart of the hbase server data update system in the embodiment of this article is shown;

[0056] Figure 5 The first flow chart of the hbase server data update method according to the embodiment of this invention is shown;

[0057] Figure 6 A second flow chart of the hbase server data update method according to the embodiment of this invention is shown;

[0058] Figure 7 The third flow chart of the hbase server data update method of the embodiment of this article is shown;

[0059] Figure 8 The fourth flow chart of the hbase server data update method of the embodiment of this article is shown;

[0060] Fig. 9 The structure diagram of the hbase server data update device in the embodiment of this article is shown;

[0061] Fig.10 A schematic diagram of the service end of the embodiment of this article is shown.

[0062] Description of the accompanying symbols:

[0063] 101. Business end;

[0064] 102. hbase server;

[0065] 103. Data warehouse;

[0066] 901, write request receiving unit;

[0067] 902. Data table creation unit;

[0068] 903, data writing unit;

[0069] 904, mapping update unit;

[0070] 911, read request receiving unit;

[0071] 912, query unit;

[0072] 913, reading unit;

[0073] 914, monitoring unit;

[0074] 1002, business end;

[0075] 1004. Processor;

[0076] 1006. Memory;

[0077] 1008. Driving mechanism;

[0078] 1010, input / output module;

[0079] 1012. Input device;

[0080] 1014. Output device;

[0081] 1016. Presentation equipment;

[0082] 1018. Graphical user interface;

[0083] 1020. Network interface;

[0084] 1022. Communication link;

[0085] 1024. Communication bus. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of this article to clearly and completely describe the technical solutions in the embodiments of this article. Obviously, the described embodiments are only part of the embodiments of this article, not all of the embodiments. Based on the embodiments of this article, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this article.

[0087] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0088] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel.

[0089] It should be noted that the hbase server data update method and device in this article can be used in the financial field, and can also be used in any field other than the financial field. The application field of the hbase server data update method and device in this article is not limited.

[0090] In the prior art, since the hbase server has only one data table for the business, the expiration time is automatically set when the data is stored in the data table, that is, the data stored in the hbase data table is deleted based on the expiration time. If the expiration time is set unreasonably, the data that should be deleted will not be deleted and the data that should not be deleted will be deleted. In addition, the processed data generated in the data warehouse is written into the hbase data table one by one or batch by batch by calling the API of the hbase server. The hbase data table may respond to read and write operations at the same time. Therefore, this data writing method is not suitable for scenarios with large data sets, and there are problems of slow speed and high pressure on the hbase server.

[0091] In order to solve the above technical problems, an HBase server data update system is provided in one embodiment of the present invention. Figure 1 As shown, it includes: a business end 101, an hbase server 102 and a data warehouse 103.

[0092] The business end 101 is used to execute business logic, such as obtaining customer data from the hbase server and pushing information for the obtained data. Specifically, the business end 101 can be a server, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, etc. Among them, the smart wearable device can include a smart bracelet, a smart watch, smart glasses, a smart helmet, etc. Of course, the business end is not limited to the above-mentioned electronic devices with a certain entity, and it can also be software running in the above-mentioned electronic devices.

[0093] The hbase server 102 stores a view mapping table and at least two data tables. The view mapping table stores the data table name pointed to by the view, that is, the currently valid data table name. The data table stores the processed data uploaded by the data warehouse.

[0094] The data warehouse 103 is provided with periodic tasks for batch processing of real-time generated data to obtain processed data that meets the conditions, such as real-time generated transaction data, and screening out customer group information (such as name, contact number, etc.) that meets the conditions by batch analysis of transaction data. In specific implementation, the data warehouse 103 is, for example, hive, and the batch data generated by the data warehouse 103 is stored in the hbase server for the business end to make associated queries.

[0095] Specifically, Figure 2 As shown, the interaction process between the business end 101, the hbase server 102 and the data warehouse 103 in the hbase server data update system includes:

[0096] Step 201: The business end periodically reads the name of the data table pointed to by the view from the view mapping table of the hbase server.

[0097] In detail, the business end can read the data table name pointed to by the view from the view mapping table of the hbase server at fixed time intervals (for example, 5 minutes). The specific time interval can be determined according to the frequency of generating processed data in the data warehouse.

[0098] Step 202, the business end compares the name of the data table pointed to by the read view with the name of the data table currently in use to see whether they are the same;

[0099] If the name of the data table to be read is different from the name of the currently used data table, a new read thread is started, the data table name is loaded into the new read thread, and the new read thread reads data from the data table pointed to by the view on the hbase server;

[0100] If the read data table name is the same as the currently used data table name, then continue to execute step 201.

[0101] Step 203, the data warehouse periodically processes the data, converts the processed data into an HBase underlying binary data structure file, and sends the HBase underlying binary data structure file to the HBase server.

[0102] Specifically, the data warehouse can process the unprocessed data according to preset rules, and this document does not limit the data processing and processing results. In one implementation, the data processing is to analyze the transaction data and determine the customer group information that meets the conditions.

[0103] Step 204, the hbase server creates a new data table, stores the file sent by the data warehouse in the new data table, and updates the data table name pointed to by the view in the view mapping table to the new data table name.

[0104] When this step is implemented, the new data table name can be derived based on the name of the last newly created data table, or can be named according to the file name sent by the data warehouse, or named according to the data warehouse update time. Each data table name is different, and the data table can be uniquely located by the data table name.

[0105] When this step is implemented, a new data table can be created after receiving a file storage request from the data warehouse, or a new data table can be created according to the data warehouse data update cycle. When it is created according to the data warehouse data update cycle, before the next data update time arrives, monitor whether the data in the new data table meets the preset requirements. If not, delete the new data table and keep the view pointing to the data table name in the view mapping table unchanged.

[0106] This embodiment stores multiple data tables and view mapping tables in the hbase server. Each data table is used to store the data processed by the data warehouse at one time. The data table name pointed to by the view in the view mapping table is always the latest data table name. This ensures that the business end reads the latest data and does not fail to read data. The underlying data table is shielded at the business end. Each time the data table is updated, the data generated by the data warehouse is stored in a new data table, that is, the data is fully updated, which can avoid the accumulation of update errors and ensure the overall consistency and accuracy of the data. In addition, since there is no traffic access to the data each time the data table is updated, it can prevent pressure on the performance of the hbase server.

[0107] The data warehouse converts the processed data into hbase underlying binary data structure files, sends the hbase underlying binary data structure files to the hbase server, and the hbase server stores the underlying binary data structure files in a new data table. The data is generated through batch operations and succeeds or fails as a whole. The data as a whole has transactional guarantees and can be applied to scenarios with large data sets, improve data transmission speed, and reduce the pressure on hbase server to store data.

[0108] In one embodiment of this invention, Figure 3 As shown, after the business end starts a new read thread in step 202, it also determines whether there is an old read thread. If so, the old read thread is closed.

[0109] This embodiment can transfer the read operation completely to the new data table in a timely manner, avoiding continuing to read data in the old data table and affecting the precise implementation of the business.

[0110] In one embodiment of this invention, Figure 4 As shown, the interaction process between the business end 101, the hbase server 102 and the data warehouse 103 in the hbase server data update system also includes:

[0111] Step 401, the business end monitors the data reading status of the reading program, and if the data reading is abnormal, a read exception request is sent to the hbase server.

[0112] Step 402, after receiving the read exception request, the hbase server modifies the view pointing to the data table name in the view mapping table to the view pointing to the data table name set last time.

[0113] This embodiment can timely discover the abnormal reading of the currently valid data table (i.e., the data table with the view pointing to the data table name), and timely modify the view pointing to the data table name in the view mapping table to the view pointing to the data table name set last time, thereby ensuring the effective reading of data. In specific implementation, if the reading abnormality is determined M times, an hbase abnormality alarm information is issued so that relevant personnel can detect the performance of hbase.

[0114] In one embodiment of the present invention, an HBase server data update method is provided, which is applied to an HBase server. A view mapping table and a data table for storing a view pointing to a data table name are established in the HBase server, such as Figure 5 As shown, including:

[0115] Step 501: Receive a data write request sent by a data warehouse.

[0116] Step 502: Create a new data table.

[0117] Step 503: Write the data sent by the data warehouse into the new data table.

[0118] Step 504: Update the data table name pointed to by the view in the view mapping table to the new data table name, so that when responding to a read request, data is read according to the data table name pointed to by the view in the view mapping table.

[0119] This step is performed after the new data table is written. For example, the data writing time can be detected, or if there is no input writing for a predetermined time, it is considered that there is no more data to be written to the new data table, that is, the writing is completed.

[0120] When writing data, the view in the view mapping table points to the data table before the current data table is written. The data table for writing data and the data table for reading data are independent of each other, so that there is no traffic access to the data each time the data table is updated, which can prevent pressure on the performance of the hbase server.

[0121] In one embodiment of this invention, Figure 6 As shown, the hbase server data update method also includes:

[0122] Step 601: receiving a data read request.

[0123] In this step, the data reading request is sent by the business end at a scheduled time or manually triggered.

[0124] Step 602: Determine the name of the data table pointed to by the view from the view mapping table. This step can determine the data table currently in use.

[0125] Step 603, read data from the relevant data table on the HBase server according to the data table name pointed to by the view.

[0126] When this step is implemented, the data table address can be uniquely determined according to the data table name pointed to by the view, and data can be read according to the data table address.

[0127] Step 604, monitor in real time whether the view mapping table is updated. If updated, re-execute the steps of determining the data table name pointed to by the view from the view mapping table and the subsequent steps. If not updated, wait until the next monitoring time arrives to continue to determine whether the view mapping table is updated.

[0128] In specific implementation, this step can also be performed by the business end, which periodically accesses the view mapping table and determines whether the view mapping table is updated by comparing the data table name pointed to by the currently used view with the data table name pointed to by the most recently read view.

[0129] This embodiment can timely discover and connect to the latest data table by real-time monitoring of the view mapping table, further ensuring stable and effective business development.

[0130] In one embodiment of this invention, Figure 7 As shown, after writing the data sent by the data warehouse into the new data table, it also includes:

[0131] Step 701: Check whether the data in the new data table meets the preset requirements.

[0132] The preset requirements in this step include but are not limited to whether the data format, data volume, etc. meet the preset requirements.

[0133] Step 702: If the conditions are met, the data table name pointed to by the view in the view mapping table is updated to the new data table name. If the conditions are not met, an alarm is issued or the new data table is cached in a fixed storage area for manual inspection.

[0134] In a further embodiment, the hbase server data update method further includes:

[0135] Step 703: Count the number of times the detected data does not meet the preset requirements.

[0136] In this step, the number of times the detected data does not meet the preset requirements is the amount of data tables that do not meet the preset requirements. In specific implementation, the amount of data tables that do not meet the preset requirements within a predetermined time period can be counted, or the amount of data tables that do not meet the preset requirements that are continuously detected can be counted.

[0137] Step 704: When the number of times reaches a predetermined number, a data abnormality reminder message is issued.

[0138] In this step, the predetermined number of times can be set according to the actual situation, and the specific value is not limited in this article. The abnormal reminder information method includes but is not limited to text messages, emails, etc.

[0139] This embodiment can ensure the validity of the data in the new data table and can promptly discover data warehouse processing anomalies or hbase storage anomalies.

[0140] In one embodiment of this invention, the view mapping table also stores the historical data table name, such as Figure 8 As shown, the hbase server data update method also includes:

[0141] Step 801: Count the number of historical data table names in the view mapping table.

[0142] Step 802: When the number of historical data table names reaches a predetermined threshold, the top N data tables in the hbase server are deleted in the order of the data table creation time, where N is less than the predetermined threshold.

[0143] This embodiment can release HBase storage space in time and ensure effective storage of data by regularly cleaning up the data table in the HBase server.

[0144] Based on the same inventive concept, this article also provides an HBase server data update device, as described in the following embodiments. Since the principle of solving the problem by the HBase server data update device is similar to that of the HBase server data update method, the implementation of the HBase server data update device can refer to the HBase server data update method, and the repeated parts will not be repeated.

[0145] Specifically, the hbase server data update device is applied to the hbase server, and the hbase server is established with a view mapping table and a data table for storing the view pointing to the data table name, such as Fig. 9 As shown, the hbase server data update device includes: a writing module 90. Specifically, the writing module 90 includes:

[0146] A write request receiving unit 901 is used to receive a data write request sent by a data warehouse;

[0147] A data table creation unit 902, used to create a new data table;

[0148] The data writing unit 903 is used to write the data sent by the data warehouse into a new data table;

[0149] The mapping update unit 904 is used to update the view pointing data table name in the view mapping table to the new data table name, so as to read data according to the view pointing data table name in the view mapping table when responding to a read request.

[0150] In a further embodiment, the hbase server data update device further includes: a reading module 91. Specifically, the reading module 91 includes:

[0151] A read request receiving unit 911, configured to receive a data read request;

[0152] A query unit 912, used to determine the name of the data table pointed to by the view from the view mapping table;

[0153] The reading unit 913 is used to read data from the relevant data table of the HBase server according to the data table name pointed to by the view;

[0154] The monitoring unit 914 is used to monitor in real time whether the view mapping table is updated. If updated, the steps of determining the name of the data table pointed to by the view from the view mapping table and subsequent steps are re-executed.

[0155] This embodiment can produce the following technical effects:

[0156] (1) It can ensure that the latest data is read and there will be no situation where data cannot be read.

[0157] (2) Each time data is generated by the data warehouse, it is stored in a new data table, that is, the data is fully updated, which can avoid the accumulation of update errors and ensure the overall consistency and accuracy of the data.

[0158] (3) Each time a data write operation is performed, there is no traffic accessing the data, which is equivalent to an offline operation. This does not put pressure on the performance of the HBase server. Files can be generated quickly by increasing concurrency without relying on the performance of the HBase server, thus improving the data loading time by 5-10 times.

[0159] (4) Data generation is independent of data table switching, and the view mapping table can be modified at any time, which enhances the flexibility of switching.

[0160] In one embodiment of this invention, an hbase server is provided, such as Fig.10As shown, the business end 1002 may include one or more processors 1004, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The business end 1002 may also include any memory 1006, which is used to store any kind of information such as code, settings, data, etc. Without limitation, for example, the memory 1006 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the business end 1002. In one case, when the processor 1004 executes the associated instructions stored in any memory or combination of memories, the business end 1002 can execute any operation of the associated instructions, specifically, the business end 1002 periodically reads the view pointing to the data table name from the view mapping table of the hbase server; when the read data table name is different from the currently used data table name, start a new read thread, load the data table name into the new read thread, so that the new read thread reads data from the view pointing to the data table of the hbase server. The business end 1002 also includes one or more drive mechanisms 1008 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0161] The business end 1002 may also include an input / output module 1010 (I / O) for receiving various inputs (via input device 1012) and for providing various outputs (via output device 1014). A specific output mechanism may include a presentation device 1016 and an associated graphical user interface 1018 (GUI). In other embodiments, the input / output module 1010 (I / O), input device 1012, and output device 1014 may not be included, and the business end 1002 may only serve as an hbase server in the network. The business end 1002 may also include one or more network interfaces 1020 for exchanging data with other devices via one or more communication links 1022. One or more communication buses 1024 couple the components described above together.

[0162] The communication link 1022 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1022 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0163] The embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0164] The embodiments of this document also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the steps of any of the methods of the aforementioned embodiments.

[0165] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0166] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0167] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0171] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an hbase server (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0173] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A method for updating data on an hbase server. It is characterized in that Applied to an HBase server, the HBase server has a view mapping table and a data table for storing a view pointing to a data table name, and the method includes: Receive data writing requests sent by the data warehouse; Create a new data table; Writing the data sent by the data warehouse into the new data table; The view-pointing data table name in the view mapping table is updated to the new data table name, so that when responding to a read request, data is read according to the view-pointing data table name in the view mapping table.

2. The method according to claim 1, It is characterized in that Also includes: Receive a data read request; Determine the data table name pointed to by the view from the view mapping table; According to the data table name pointed to by the view, read the data from the relevant data table on the hbase server; Monitor in real time whether the view mapping table is updated. If updated, re-execute the steps of determining the data table name pointed to by the view from the view mapping table and the subsequent steps.

3. The method according to claim 2, It is characterized in that After the data sent by the data warehouse is written into the new data table, the following steps are also included: Check whether the data in the new data table meets the preset requirements; If satisfied, the data table name pointed to by the view in the view mapping table is updated to the new data table name.

4. The method according to claim 3, It is characterized in that Also includes: Count the number of times the detected data does not meet the preset requirements; When the number reaches the preset number, a data abnormality reminder message is issued.

5. The method according to claim 1, It is characterized in that The view mapping table also stores the historical data table name, and the method further includes: Count the number of historical data table names in the view mapping table; When the number of historical data table names reaches a predetermined threshold, the top N data tables in the hbase server are deleted in the order of the data table creation time, where N is less than the predetermined threshold.

6. A method for updating data on an hbase server. It is characterized in that Using the hbase server to update the system, the method includes: The service end periodically reads the name of the data table pointed to by the view from the view mapping table of the hbase server; when the read data table name is different from the currently used data table name, a new read thread is started, and the data table name is loaded into the new read thread, so that the new read thread reads data from the data table pointed to by the view of the hbase server; The data warehouse processes data periodically, converts the processed data into an HBase underlying binary data structure file, and sends the HBase underlying binary data structure file to the HBase server; The hbase server creates a new data table, stores the files sent by the data warehouse in the new data table, and updates the data table name pointed to by the view in the view mapping table to the new data table name.

7. The method according to claim 6, It is characterized in that After the business end starts a new read thread, it also includes: Determine whether there is an old read thread. If so, close the old read thread.

8. The method according to claim 7, It is characterized in that The method further comprises: The business end monitors the data read by the reading program, and if the read data is abnormal, sends a read exception request to the hbase server; After receiving the read exception request, the hbase server modifies the view pointing data table name in the view mapping table to the view pointing data table name set last time.

9. A hbase server data update device, It is characterized in that Applied to an HBase server, the HBase server has a view mapping table and a data table for storing a view pointing to a data table name, and the device includes: A write request receiving unit, used for receiving a data write request sent by the data warehouse; A data table creation unit, used to create a new data table; A data writing unit, used for writing the data sent by the data warehouse into the new data table; The mapping update unit is used to update the view pointing data table name in the view mapping table to a new data table name, so as to read data according to the view pointing data table name in the view mapping table when responding to a read request.

10. An hbase server data update system, It is characterized in that include: Business end, hbase server and data warehouse; The business end is used to periodically read the view pointing data table name from the view mapping table of the hbase server; When the name of the data table to be read is different from the name of the data table currently in use, a new reading thread is started, and the name of the data table is loaded into the new reading thread, so that the new reading thread reads data from the data table pointed to by the view of the hbase server; The data warehouse is used to periodically process data, convert the processed data into hbase underlying binary data structure files, and send the hbase underlying binary data structure files to the hbase server; The hbase server is used to create a new data table, store the files sent by the data warehouse in the new data table, and update the view pointing to the data table name in the view mapping table to the new data table name.