Data updating method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data updating method, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the steps of obtaining a to-be-stored file, and setting structure configuration information of an offline data table corresponding to the to-be-stored file based on the to-be-stored file; creating an offline data table in the big data cluster database based on the structure configuration information, and writing the service data in the to-be-stored file into the offline data table based on the structure configuration information; and creating a snapshot of the written off-line data table to obtain a target snapshot, and updating the online data table corresponding to the off-line data table in the big data cluster database based on the target snapshot, so that the updated online data table is utilized to provide a data query service, rapid updating of the online data table is realized, and the data query efficiency is improved. According to the method, the data writing duration of the online data table is shortened, particularly the writing duration of mass data, so that the unavailable duration of the query service of the big data cluster database is effectively shortened, and the stability of the query service is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data updating method, an electronic device, and a storage medium. Background Art

[0002] For a big data cluster database that provides data query services, it is necessary to ensure the timeliness of the data in the big data cluster database, that is, the data tables in the big data cluster database need to be updated in a timely manner.

[0003] In the existing technology, the data table in the big data cluster database is updated by deleting the data table and recreating the data table, and then writing the full amount of business data in batches to the newly created data table. However, the writing process will affect the query performance of the business, resulting in the unavailability of the query service. Therefore, when the above solution writes massive amounts of business data in batches to the data table, the writing time is long, resulting in the query service being unavailable for too long.

[0004] Therefore, the above solution has the problem that the query service is unavailable for a long time due to the batch writing of massive data into the big data cluster database. Summary of the Invention

[0005] The embodiments of the present application provide a data updating method, an electronic device and a storage medium, which realize data updating to solve the problem that the query service is unavailable for a long time due to the batch writing of massive data into a big data cluster database.

[0006] In a first aspect, an embodiment of the present application provides a data updating method, the method comprising:

[0007] Obtain the files to be stored, and set the structure configuration information of the offline data table corresponding to the files to be stored based on the files to be stored;

[0008] Create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the to-be-stored file into the offline data table based on the structure configuration information;

[0009] A snapshot of the offline data table after writing is created to obtain a target snapshot, and an online data table corresponding to the offline data table in the big data cluster database is updated based on the target snapshot to provide data query services using the updated online data table.

[0010] In an embodiment of the present application, a file to be stored can be obtained, and structural configuration information of an offline data table corresponding to the file to be stored can be set based on the file to be stored; an offline data table can be created in a big data cluster database based on the structural configuration information, and business data in the file to be stored can be written into the offline data table based on the structural configuration information; a snapshot of the offline data table after writing is created to obtain a target snapshot, and an online data table corresponding to the offline data table in the big data cluster database is updated based on the target snapshot, so as to provide data query services using the updated online data table. In the above technical solution, offline data tables in the big data cluster database are limited to batch writing of massive data, and online data tables are limited to providing query services. That is, offline data tables do not provide data query services, and only online data tables do. Therefore, business data in the files to be stored can be first written into the offline data tables in the big data cluster database. Then, based on the snapshot mechanism, the data in the offline data tables can be quickly switched to the corresponding online data tables. By using the offline data tables as intermediate data tables, the function of indirectly writing business data in the files to be stored into the online data tables is realized, thereby realizing rapid updates of the online data tables. Compared with directly writing business data in the files to be stored into the online data tables, the data writing time of the online data tables is shortened, especially the writing time of massive data, thereby effectively shortening the unavailability period of the query service of the big data cluster database. This solves the problem in the prior art that the query service is unavailable for a long time due to the batch writing of massive data into the big data cluster database, and improves the stability of the query service. In addition, by limiting the offline and online data tables to provide query services, query conflicts that may occur when the query service switches between offline and online tables can be prevented, thereby reducing application-level development costs.

[0011] In a second aspect, an embodiment of the present application provides a data updating device, the device comprising:

[0012] The setting module is used to obtain the files to be stored and set the structural configuration information of the offline data table corresponding to the files to be stored based on the files to be stored;

[0013] A writing module is used to create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the file to be stored into the offline data table based on the structure configuration information;

[0014] The update module is used to create a snapshot of the offline data table after writing, obtain a target snapshot, and update the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, so as to provide data query services using the updated online data table.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0016] at least one processor; and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the data updating method of any embodiment of the present application.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements a data update method as described in any embodiment of the present application.

[0019] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects described in the second, third and fourth aspects can refer to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0020] In this application, the name of the data update device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalents.

[0021] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is a flow chart of a data updating method provided in an embodiment of the present application;

[0024] Figure 2 This is another flowchart of the data updating method provided in an embodiment of the present application;

[0025] Figure 3 This is a structural diagram of a data updating device provided in an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first," "second," "target," and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatus.

[0029] Figure 1 This is a flow chart of the data update method provided by an embodiment of the present application. This embodiment can be applied to scenarios where data tables in a big data cluster database need to be updated. A data update method provided by this embodiment can be executed by a data update device provided by an embodiment of the present application, which can be implemented by software and / or hardware. In a specific embodiment, the data update device can be integrated into an electronic device, for example, the electronic device can be a computer or other device. The execution subject of this method can be an electronic device, see Figure 1 The data updating method of this embodiment includes but is not limited to the following steps:

[0030] S110: Acquire the file to be stored, and set the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored.

[0031] The files to be imported are raw data files that need to be imported into the big data cluster database. The files to be imported may include business data, such as user behavior data. Optionally, the files to be imported may be compressed files, and the file format may be GZ format.

[0032] Big data cluster databases are data storage and analysis platforms built on a distributed architecture (such as HBase). They support efficient management and parallel computing of massive amounts of data, providing highly available and concurrent query services. Offline data tables are used in big data cluster databases for batch writing of massive amounts of data. Structural configuration information is metadata that defines the physical storage and logical structure of a data table.

[0033] Optionally, the structural configuration information may include at least one of the table name, field name, encoding format, row key, pre-partition, and version; wherein the row key is a unique identifier for each row of data; pre-partitioning is to pre-set the data storage location by formulating partitions when creating a data table in a big data cluster database, so as to achieve data load balancing and reduce resource consumption caused by massive data, thereby improving data query performance.

[0034] Specifically, the to-be-stored files can be obtained, that is, multiple business data files sent by the upstream system can be received. These business data files can be compressed files, and the file format can be GZ format to reduce the consumption of bandwidth resources. Secondly, multiple business data files and the file name of each business data file are saved, and then the name of the to-be-stored file entered by the user is obtained, and the business data file corresponding to the name of the to-be-stored file is queried to obtain the to-be-stored file; then, the structural configuration information of the offline data table corresponding to the to-be-stored file can be set based on the to-be-stored file, that is, the table name, field name, encoding format, row key, pre-partition and / or version of the offline data table can be set based on the to-be-stored file.

[0035] For example, in one implementation method, the table name, field name, encoding format, row key, pre-partition and / or version of the offline data table corresponding to the name of the file to be stored can be queried from the configuration file of the big data cluster database. The configuration file is a pre-configured file, including the mapping relationship between various business data files, offline data tables and online data tables in the upstream system, as well as the structural configuration information of the offline data table, that is, one business data file corresponds to one offline data table, and one offline data table corresponds to one online data table; the online data table is a business application data table in the big data cluster database used to provide query services.

[0036] In another implementation, the name of the offline data table can be set based on the name of the file to be stored in accordance with the preset offline table name naming rule, wherein the preset offline table name naming rule is a table name naming rule set in advance for the offline data table in the big data cluster database to standardize the table name of the offline data table; the business data in the file to be stored can be parsed, and the field name of the offline data table can be determined based on the data type of the business data. For example, for a customer information data file, the field name can be set as customer number, gender, mobile phone number and region, etc.; the unique identifier of each business data in the file to be stored can be determined as a row key. For example, for a customer information data file, The customer number is used as the row key. The number of partitions can be set based on the amount of data in the file to be stored, and the data characteristics of each partition can be set. For example, for a customer information data file, when the number of partitions is 10 and the 12th digit and subsequent digits in the customer number are sequentially generated values, the data with the 12th digit of the customer number being 0 can be stored in partition 1, the data with the 12th digit of the customer number being 1 can be stored in partition 2, and so on, the data with the 12th digit of the customer number being 9 can be stored in partition 10. The encoding format of the offline data table can be set to the default encoding format, such as FAST_DIFF; and / or the version of the offline data table can be set to the current timestamp.

[0037] S120: Create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the to-be-stored file into the offline data table based on the structure configuration information.

[0038] Specifically, after obtaining the structural configuration information of the offline data table, an offline data table can be created in the big data cluster database based on the structural configuration information. That is, an offline data table can be created in the big data cluster database based on the table name, field name, encoding format, row key, pre-partition and / or version to ensure that the table structure of the offline data table is the determined structural configuration information. At this time, there is no data in the offline data table, and it is an empty table.

[0039] Then, the business data in the file to be stored can be written into the offline data table based on the structural configuration information. For example, the business data in the file to be stored can be read line by line according to the structural configuration information, and the read business data can be stored in the offline data table so that the data in the offline data table meets the structural configuration information of the offline data table.

[0040] S130: Create a snapshot of the offline data table after writing to obtain a target snapshot, and update the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot to provide data query services using the updated online data table.

[0041] A snapshot is a mechanism used in large data cluster databases to record and back up data. In this embodiment, a snapshot is a mirror image of the complete logical state of a data table at a specific point in time. This is achieved through metadata mapping rather than full data replication. This means that the snapshots in this embodiment do not involve data copying. The target snapshot is a snapshot of an offline data table.

[0042] Specifically, after the business data in the file to be stored is written into the offline data table, a snapshot of the offline data table after writing can be created to obtain a target snapshot. The target snapshot at this time includes the metadata of the offline data table and the file mapping information of the physical file, that is, the file mapping information is the actual physical address of the table data file corresponding to the offline data table; then, the online data table corresponding to the offline data table in the big data cluster database can be updated based on the target snapshot, so that the data in the online data table is the data in the offline data table, that is, the data in the online data table is the business data in the file to be stored, thereby realizing the function of indirectly writing the business data in the file to be stored into the online data table, thereby using the updated online data table to provide data query services, thereby realizing the real-time query function of the big data cluster database.

[0043] The technical solution of the embodiment of the present application can obtain the file to be stored, and set the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored; create an offline data table in the big data cluster database based on the structural configuration information, and write the business data in the file to be stored into the offline data table based on the structural configuration information; create a snapshot of the offline data table after writing to obtain a target snapshot, and update the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, so as to provide data query services using the updated online data table. In the above technical solution, offline data tables in the big data cluster database are limited to batch writing of massive data, and online data tables are limited to providing query services. That is, offline data tables do not provide data query services, and only online data tables do. Therefore, business data in the files to be stored can be first written into the offline data tables in the big data cluster database. Then, based on the snapshot mechanism, the data in the offline data tables can be quickly switched to the corresponding online data tables. By using the offline data tables as intermediate data tables, the function of indirectly writing business data in the files to be stored into the online data tables is realized, thereby realizing rapid updates of the online data tables. Compared with directly writing business data in the files to be stored into the online data tables, the data writing time of the online data tables is shortened, especially the writing time of massive data, thereby effectively shortening the unavailability period of the query service of the big data cluster database. This solves the problem in the prior art that the query service is unavailable for a long time due to the batch writing of massive data into the big data cluster database, and improves the stability of the query service. In addition, by limiting the offline and online data tables to provide query services, query conflicts that may occur when the query service switches between offline and online tables can be prevented, thereby reducing application-level development costs.

[0044] The following further describes a data updating method provided by an embodiment of the present application. Figure 2 This is another flow chart of the data updating method provided by the embodiment of the present application. The embodiment of the present application is optimized based on the above embodiments. Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0045] S201: Acquire a file to be stored, and set structural configuration information of an offline data table corresponding to the file to be stored based on the file to be stored.

[0046] Optionally, before setting the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored, it can be determined whether the file to be stored is empty; when the file to be stored is not empty, determine whether the offline data table corresponding to the file to be stored exists in the big data cluster database; if the offline data table corresponding to the file to be stored does not exist in the big data cluster database, trigger the execution of setting the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored; if the offline data table corresponding to the file to be stored exists in the big data cluster database, delete the offline data table, and trigger the execution of setting the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored.

[0047] Specifically, after obtaining the file to be stored, it can be determined whether the file to be stored is empty. If the file to be stored is empty, it indicates that there is no need to update the online data table corresponding to the file to be stored, and no operation is required at this time; if the file to be stored is not empty, it indicates that the online data table corresponding to the file to be stored needs to be updated. At this time, it can be determined whether there is an offline data table corresponding to the file to be stored in the big data cluster database.

[0048] If the offline data table corresponding to the file to be stored does not exist in the big data cluster database, the structural configuration information of the offline data table corresponding to the file to be stored can be set directly based on the file to be stored; if the offline data table corresponding to the file to be stored exists in the big data cluster database, it indicates that the offline data table contains business data written in the previous round. At this time, the offline data table needs to be deleted, and then the structural configuration information of the offline data table corresponding to the file to be stored is set based on the file to be stored.

[0049] In an embodiment of the present application, before the business data in the to-be-stored file is written in batches into the offline data table, it is automatically checked whether the to-be-stored file is empty. When the to-be-stored file is empty, the online data table may not be updated. At this time, the data in the online data table is historical data, which avoids the situation where the online data table is empty, thereby ensuring that historical data can be queried when the to-be-stored file is empty, and the query is not empty. When the to-be-stored file is not empty, it is automatically checked whether there is an offline data table corresponding to the to-be-stored file in the big data cluster database. This can avoid the failure of the new and old data to be stored due to duplication of primary keys or logical conflicts, thereby eliminating redundant conflicts, thereby improving the accuracy and efficiency of data writing.

[0050] S202: Create an offline data table in the big data cluster database based on the structure configuration information.

[0051] S203: Convert the format of the file to be stored into an intermediate format to obtain an intermediate storage file.

[0052] The intermediate format is the format in which the business data in the file to be stored is written into the offline data table. It is used to standardize the format of the file to be stored. For example, the intermediate format can be LZO format. The intermediate storage file is the file to be stored in the intermediate format.

[0053] Specifically, after creating an offline data table, the format of the file to be stored can be converted into an intermediate format. For example, the file to be stored in GZ format can be converted into LZO format to obtain an intermediate storage file. The LZO format supports index-based sharding processing. When writing data, multi-threaded parallel loading of data can be started, which improves the data writing efficiency compared to the single thread of the GZ format. In addition, the LZO format file supports decompression and writing at the same time, which reduces memory usage and reduces the risk of memory overflow compared to the full decompression of the GZ format file to memory.

[0054] S204 : Encapsulate the business data in the intermediate storage file based on the structure configuration information to obtain a database file, and load the business data in the database file into the offline data table.

[0055] The database file is a file obtained by reorganizing the business data in the intermediate storage file according to the structural configuration information.

[0056] Specifically, after obtaining the intermediate input file, the business data in the intermediate input file can be encapsulated based on the structural configuration information, that is, the structural configuration information can be parsed and converted into configuration rules, such as constructing configuration rules based on field names, row key setting rules, pre-partitioning partitioning rules, encoding formats and version generation rules; then, the business data in the intermediate input file can be read line by line and standardized according to the configuration rules, such as rearranging each line of business data in the intermediate input file according to the field name and the order of the field names, re-serializing each line of business data after sorting according to the encoding format, setting the row key in each line of business data according to the row key setting rules, and generating the partition identifier of each line of business data based on the pre-partitioning partitioning rules; then, encapsulate the standardized business data to obtain the database file, such as splitting the business data into different files according to the partition identifier of each line of business data to generate the corresponding database file.

[0057] Then, the weight can be determined according to the file size of the intermediate input file, that is, the larger the file, the larger the corresponding weight is set, and resources can be dynamically planned for the offline data table based on the weight, that is, the larger the weight, the more resources are allocated, so as to reasonably allocate memory and CPU resources, make full use of the resources of the big data cluster database, and ensure that the services of the big data cluster database can run stably; after that, the business data in the database file can be loaded into the offline data table, that is, the writing task can be split according to the pre-partition, so as to use multi-threaded parallelism to batch load the business data in the database file into the offline data table, so as to write the business data in the file to be input into the offline data table based on the structure configuration information.

[0058] Optionally, the public attributes involved in warehousing can be templated, including table name, field name, encoding format, row key, pre-partition and version, that is, the business data in the intermediate warehousing file can be encapsulated according to the table structure information in the structure configuration information in combination with the template configuration, and the corresponding database file can be generated according to the pre-partition information in the structure configuration information. This can realize the universal configuration of public logic, and the common logic can be directly reused for different business scenarios, reducing development costs.

[0059] S205: Create a snapshot of the offline data table after writing to obtain a target snapshot.

[0060] Specifically, the file mapping information in the target snapshot is the physical address of the table data file corresponding to the offline data table. At this time, the physical address of the table data file is the physical address in the data directory corresponding to the offline data table in the big data cluster database.

[0061] S206 , deleting the offline data table, and moving the table data file corresponding to the offline data table from the data directory corresponding to the offline data table of the big data cluster database to the archive directory of the big data cluster database, to obtain a first address of the table data file.

[0062] The archive directory is a dedicated storage area for historical data or infrequently accessed data. Its purpose is to optimize storage costs and maintain data accessibility through reasonable data lifecycle management. The first address is the physical address after the table data file corresponding to the offline data table is moved from the data directory corresponding to the offline data table in the big data cluster database to the archive directory.

[0063] Specifically, after creating the target snapshot of the offline data table, you can use a delete statement to delete the offline data table, and then move the physical address of the table data file corresponding to the offline data table from the data directory corresponding to the offline data table of the big data cluster database to the archive directory of the big data cluster database, and determine the moved physical address as the first address of the table data file.

[0064] S207: Update file mapping information of the target snapshot based on the first address.

[0065] Specifically, after obtaining the first address, the file mapping information of the target snapshot can be modified to the first address, that is, the physical address under the data directory corresponding to the offline data table of the big data cluster database can be changed to the physical address under the archive directory of the big data cluster database, so that the target snapshot can successfully reference the table data file of the offline data table.

[0066] S208: Clone an online data table in the big data cluster database based on the target snapshot.

[0067] Specifically, after updating the file mapping information of the target snapshot, the table name of the online data table corresponding to the offline data table can be queried from the configuration file of the big data cluster database. Then, in the big data cluster database, a new table is cloned based on the target snapshot using cloning technology, and the table name of the new table is determined as the table name of the online data table obtained by the query, so as to obtain the online data table. At this time, the cloning operation only copies the metadata and does not migrate the physical data, that is, the metadata of the online data table is the metadata of the offline data table, and the online data table shares the underlying data file with the target snapshot, that is, the file mapping information of the online data table is the file mapping information of the target snapshot, that is, the first address.

[0068] Optionally, before cloning the online data table based on the target snapshot in the big data cluster database, the online data table corresponding to the offline data table can be obtained from the big data cluster database, and it can be determined whether the online data table exists; if the online data table does not exist, the execution of cloning the online data table based on the target snapshot in the big data cluster database is triggered; if the online data table exists, the online data table is deleted, and the execution of cloning the online data table based on the target snapshot in the big data cluster database is triggered.

[0069] Specifically, before cloning the online data table based on the target snapshot in the big data cluster database, the table name of the online data table corresponding to the offline data table can be queried from the configuration file of the big data cluster database, and the online data table corresponding to the table name is obtained in the big data cluster database. If the online data table exists, the online data table is deleted, and then the online data table is cloned based on the target snapshot in the big data cluster database; if the online data table does not exist, the online data table is directly cloned based on the target snapshot in the big data cluster database.

[0070] In an embodiment of the present application, before cloning an online data table based on a target snapshot, it is automatically checked whether the online data table exists, which can avoid table structure conflicts, eliminate interference from historical data, avoid metadata residual risks, and thus ensure data integrity.

[0071] S209: Move the table data file corresponding to the offline data table from the first address to the data directory corresponding to the online data table of the big data cluster database to obtain a second address of the table data file.

[0072] The second address is a physical address obtained after the table data file corresponding to the offline data table is moved from the first address to the data directory corresponding to the online data table of the big data cluster database.

[0073] Specifically, after cloning the online data table based on the target snapshot, the merge function of the big data cluster database can be used to move the table data file corresponding to the offline data table from the first address to the data directory corresponding to the online data table of the big data cluster database, and the physical address after the move is determined as the second address of the table data file.

[0074] S210: Update file mapping information of the online data table based on the second address, so as to provide data query service using the updated online data table.

[0075] Specifically, after obtaining the second address of the table data file, the file mapping information of the online data table can be modified to the second address, that is, the physical address under the archive directory of the big data cluster database can be modified to the physical address under the data directory corresponding to the online data table of the big data cluster database, so that the online data table can successfully reference the table data file of the offline data table, thereby realizing the function of indirectly writing the business data in the file to be stored into the online data table, and completing the data update of the online data table. Therefore, the updated online data table can be used to provide data query services, thereby realizing the real-time query function of the big data cluster database.

[0076] Optionally, after the file mapping information of the online data table is updated based on the second address, the target snapshot may be deleted to release computer resources.

[0077] Optionally, after updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, the offline data table and the online data table can be sorted according to the preset sorting rules, and the sorted offline data table and the sorted online data table can be compared to obtain a comparison result; if the comparison result is that the data is inconsistent, an update failure prompt is generated.

[0078] The preset sorting rule is a pre-set sorting rule used to determine the order and priority of data. The update failure prompt is a message used to prompt the user that the online data table has failed to update.

[0079] Specifically, after updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, the offline data table can be sorted according to the preset sorting rules, and the online data table can be sorted according to the preset sorting rules. For example, the offline data table and the online data table can be sorted in ascending order based on the size of the row keys; then, the sorted offline data table and the sorted online data table are compared, that is, the sampling query conditions input by the user can be obtained, and then the sorted offline data table is queried according to the sampling query conditions to obtain a first query result, and the sorted online data table is queried according to the same sampling query conditions to obtain a second query result, and then the first query result and the second query result are compared. If the data in the first query result and the second query result are consistent, it is determined that the data comparison result is the data. If the data comparison result is consistent, the comparison result is determined to be inconsistent. After that, the data volume of the offline data table and the data volume of the online data table are obtained. If the data volume of the offline data table is the same as that of the online data table, the data volume comparison result is determined to be equal. Otherwise, the data volume comparison result is determined to be unequal. After that, the comparison result is determined based on the data comparison result and the data volume comparison result. If the data comparison result is consistent and the data volume comparison result is equal, the comparison result is determined to be consistent. Otherwise, the comparison result is determined to be inconsistent. At this time, an update failure prompt is generated and displayed. For example, the update failure prompt is displayed in the form of a pop-up window on the display screen of the electronic device, and a prompt sound can be emitted at the same time to remind the user that the data update of the online data table has failed, and to take timely countermeasures, thereby improving the efficiency and accuracy of the data update. Among them, the sampling query condition is used to perform sampling queries on multiple partitions of the data table and ensure that the data in each partition can be sampled, so as to improve the accuracy of the comparison result.

[0080] It should be noted that in the big data cluster database, it supports creating an independent log file for each data table. Therefore, after loading the business data in the database file into the offline data table, the offline data table can be queried according to the sampling query conditions to obtain the first query result and the data volume of the offline data table, and then the first query result and the data volume of the offline data table are stored in the log file corresponding to the offline data table; after updating the file mapping information of the online data table based on the second address, the online data table can be queried according to the same sampling query conditions to obtain the second query result and the data volume of the online data table, and then the second query result and the data volume of the online data table are stored in the log file corresponding to the online data table; then the log file corresponding to the offline data table and the log file corresponding to the online data table are compared to verify whether the data are consistent, thereby obtaining a comparison result.

[0081] The technical solution of the embodiment of the present application can obtain the file to be stored, and set the structural configuration information of the offline data table corresponding to the file to be stored based on the file to be stored. Secondly, based on the structural configuration information, an offline data table is created in the big data cluster database, and the format of the file to be stored is converted into an intermediate format to obtain an intermediate storage file. Then, based on the structural configuration information, the business data in the intermediate storage file is encapsulated to obtain a database file, and the business data in the database file is loaded into the offline data table, which can improve the data writing efficiency and thus improve the data update efficiency.

[0082] Then, a snapshot of the offline data table after writing can be created to obtain the target snapshot, and the offline data table can be deleted, which can release computer resources in time. Then, the table data file corresponding to the offline data table is moved from the data directory corresponding to the offline data table of the big data cluster database to the archive directory of the big data cluster database, and the first address of the table data file is obtained. Based on the first address, the file mapping information of the target snapshot is updated, which can ensure that the target snapshot can successfully reference the table data file of the offline data table, providing an accurate data basis for subsequent updates of the online data table.

[0083] Afterwards, an online data table is cloned in the big data cluster database based on the target snapshot, and the table data file corresponding to the offline data table is moved from the first address to the data directory corresponding to the online data table of the big data cluster database to obtain the second address of the table data file. Then, the file mapping information of the online data table is updated based on the second address, which can ensure that the online data table can successfully reference the table data file and make the table data file independently stored in the data directory of the online data table, thereby making the data in the online data table completely independent and decoupled from the archive directory, thereby achieving the purpose of writing the business data in the to-be-stored file into the online data table. Compared with directly writing the business data in the to-be-stored file into the online data table, the data writing time of the online data table is greatly shortened, and the rapid update of the online data table is further realized. Without increasing storage, the service interruption time caused by batch writing of massive data into the business application table (i.e., online data table) of the big data cluster database is effectively shortened, thereby effectively reducing the unavailability time of the query service of the big data cluster database, improving the stability and reliability of the query service, and enhancing the user experience.

[0084] Figure 3 This is a schematic diagram of the structure of the data updating device provided in the embodiment of the present application, referring to Figure 3 , the data updating device may include:

[0085] A setting module 310 is used to obtain files to be stored and set the structure configuration information of the offline data table corresponding to the files to be stored based on the files to be stored;

[0086] A writing module 320 is used to create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the to-be-stored file into the offline data table based on the structure configuration information;

[0087] The update module 330 is used to create a snapshot of the offline data table after writing, obtain a target snapshot, and update the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot to provide data query services using the updated online data table.

[0088] In one embodiment, the data updating device further includes a deletion module, which is specifically configured to:

[0089] Before updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, deleting the offline data table and moving the table data file corresponding to the offline data table from the data directory corresponding to the offline data table in the big data cluster database to the archive directory of the big data cluster database to obtain a first address of the table data file;

[0090] File mapping information of the target snapshot is updated based on the first address.

[0091] In one embodiment, the updating module 330 updates the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, including:

[0092] Clone online data tables based on target snapshots in a big data cluster database;

[0093] Move the table data file corresponding to the offline data table from the first address to the data directory corresponding to the online data table of the big data cluster database to obtain a second address of the table data file;

[0094] File mapping information of the online data table is updated based on the second address.

[0095] In one embodiment, the data updating device further includes a first judgment module, which is specifically configured to:

[0096] Before cloning an online data table based on the target snapshot in the big data cluster database, obtain the online data table corresponding to the offline data table from the big data cluster database and determine whether the online data table exists;

[0097] If the online data table does not exist, the trigger executes the cloning of the online data table based on the target snapshot in the big data cluster database;

[0098] If the online data table exists, the online data table is deleted, and the online data table is cloned based on the target snapshot in the big data cluster database.

[0099] In one embodiment, the writing module 320 writes the business data in the to-be-stored file into the offline data table based on the structure configuration information, including:

[0100] Convert the format of the file to be stored into an intermediate format to obtain an intermediate storage file;

[0101] The business data in the intermediate storage file is encapsulated based on the structure configuration information to obtain the database file, and the business data in the database file is loaded into the offline data table.

[0102] In one embodiment, the data updating device further includes a verification module, which is specifically configured to:

[0103] After updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, sorting the offline data table and the online data table according to a preset sorting rule, and comparing the sorted offline data table with the sorted online data table to obtain a comparison result;

[0104] If the comparison result shows that the data is inconsistent, an update failure prompt will be generated.

[0105] In one embodiment, the data updating device further includes a second judgment module, which is specifically configured to:

[0106] Before setting the structure configuration information of the offline data table corresponding to the to-be-stored file based on the to-be-stored file, determining whether the to-be-stored file is empty;

[0107] When the file to be stored is not empty, determine whether there is an offline data table corresponding to the file to be stored in the big data cluster database;

[0108] If the offline data table corresponding to the file to be stored does not exist in the big data cluster database, the trigger is to set the structure configuration information of the offline data table corresponding to the file to be stored based on the file to be stored;

[0109] If an offline data table corresponding to the file to be stored exists in the big data cluster database, the offline data table is deleted, and the structure configuration information of the offline data table corresponding to the file to be stored is set based on the file to be stored.

[0110] In one embodiment, the structural configuration information in the setting module 310 includes at least one of a table name, a field name, an encoding format, a row key, a pre-partition, and a version.

[0111] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0112] The data updating device provided in this embodiment can be applied to the data updating method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0113] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 4 A block diagram of an exemplary electronic device 11 suitable for implementing embodiments of the present application is shown. Figure 4 The electronic device 11 shown is only an example and should not bring any limitation to the functions and scope of use of this embodiment.

[0114] like Figure 4 As shown, electronic device 11 is implemented as a general-purpose computing electronic device. Components of electronic device 11 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 connecting various system components (including system memory 28 and processing unit 16).

[0115] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0116] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0117] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4Not shown, often called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present application.

[0118] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0119] The electronic device 11 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the electronic device 11, and / or any device that enables the electronic device 11 to communicate with one or more other computing devices (e.g., a network card and a modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0120] like Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0121] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, for example, implementing a data update method provided in an embodiment of the present application, which includes:

[0122] Obtain the files to be stored, and set the structure configuration information of the offline data table corresponding to the files to be stored based on the files to be stored;

[0123] Create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the to-be-stored file into the offline data table based on the structure configuration information;

[0124] A snapshot of the offline data table after writing is created to obtain a target snapshot, and an online data table corresponding to the offline data table in the big data cluster database is updated based on the target snapshot to provide data query services using the updated online data table.

[0125] Of course, those skilled in the art will understand that the processor can also implement the technical solution of the data updating method provided in any embodiment of the present application.

[0126] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, for example, a data updating method provided in an embodiment of the present application is implemented. The method includes:

[0127] Obtain the files to be stored, and set the structure configuration information of the offline data table corresponding to the files to be stored based on the files to be stored;

[0128] Create an offline data table in the big data cluster database based on the structure configuration information, and write the business data in the to-be-stored file into the offline data table based on the structure configuration information;

[0129] A snapshot of the offline data table after writing is created to obtain a target snapshot, and an online data table corresponding to the offline data table in the big data cluster database is updated based on the target snapshot to provide data query services using the updated online data table.

[0130] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, computer-readable storage media can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0131] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0132] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0133] The computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] Those skilled in the art will appreciate that the modules or steps of the present application described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0135] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with relevant laws and regulations.

[0136] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data updating method, characterized in that: The method comprises: Acquire a file to be stored, and set structural configuration information of an offline data table corresponding to the file to be stored based on the file to be stored; Creating the offline data table in the big data cluster database based on the structure configuration information, and writing the business data in the to-be-stored file into the offline data table based on the structure configuration information; A snapshot of the offline data table after writing is created to obtain a target snapshot, and an online data table corresponding to the offline data table in the big data cluster database is updated based on the target snapshot to provide data query services using the updated online data table.

2. The data updating method according to claim 1, wherein: Before updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, the method further includes: Deleting the offline data table, and moving the table data file corresponding to the offline data table from the data directory corresponding to the offline data table of the big data cluster database to the archive directory of the big data cluster database, to obtain a first address of the table data file; File mapping information of the target snapshot is updated based on the first address.

3. The data updating method according to claim 2, wherein: The updating of the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot includes: Cloning an online data table based on the target snapshot in the big data cluster database; Moving the table data file corresponding to the offline data table from the first address to the data directory corresponding to the online data table of the big data cluster database to obtain a second address of the table data file; The file mapping information of the online data table is updated based on the second address.

4. The data updating method according to claim 3, wherein: Before cloning the online data table based on the target snapshot in the big data cluster database, the method further includes: Obtaining an online data table corresponding to the offline data table from the big data cluster database, and determining whether the online data table exists; If the online data table does not exist, triggering execution to clone the online data table based on the target snapshot in the big data cluster database; If the online data table exists, the online data table is deleted, and a process of cloning an online data table based on the target snapshot in the big data cluster database is triggered.

5. The data updating method according to claim 1, wherein: Writing the business data in the to-be-stored file into the offline data table based on the structural configuration information includes: Convert the format of the file to be stored into an intermediate format to obtain an intermediate storage file; The business data in the intermediate storage file is encapsulated based on the structural configuration information to obtain a database file, and the business data in the database file is loaded into the offline data table.

6. The data updating method according to claim 1, wherein: After updating the online data table corresponding to the offline data table in the big data cluster database based on the target snapshot, the method further includes: Sorting the offline data table and the online data table according to a preset sorting rule, and comparing the sorted offline data table and the sorted online data table to obtain a comparison result; If the comparison result is that the data is inconsistent, an update failure prompt is generated.

7. The data updating method according to claim 1, wherein: Before setting the structure configuration information of the offline data table corresponding to the file to be stored based on the file to be stored, the method further includes: Determine whether the file to be stored is empty; When the file to be stored is not empty, determining whether there is an offline data table corresponding to the file to be stored in the big data cluster database; If the offline data table corresponding to the file to be stored does not exist in the big data cluster database, triggering execution of setting structure configuration information of the offline data table corresponding to the file to be stored based on the file to be stored; If an offline data table corresponding to the file to be stored exists in the big data cluster database, the offline data table is deleted, and the structure configuration information of the offline data table corresponding to the file to be stored is triggered and set based on the file to be stored.

8. The data updating method according to claim 1, wherein: The structural configuration information includes at least one of a table name, a field name, an encoding format, a row key, a pre-partition, and a version.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data updating method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data updating method according to any one of claims 1 to 8 is implemented.

Images