Data updating method and device

Abstract

The invention discloses a data updating method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of determining a to-be-updated field from a data source in response to a condition that a data updating condition is met, and obtaining a to-be-stored partition of each piece of data of the same to-be-updated field in a target database; for the to-be-updated data of the same to-be-updated field in the same to-be-stored partition, generating index identifiers in one-to-one correspondence with the primary key values according to the primary key values of the to-be-updated data, writing the index identifiers into a data table of the target database, and determining the index identifier field as a sorting field of the data table; and reading the data to be updated from the data source according to the index identifier in the data table, converting the data to be updated into a file to be updated, and writing the file to be updated into the data table. According to the implementation mode, the execution time of the data updating task can be guaranteed, the obtained data is the latest data, and calculation resource occupation in the data updating process can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data updating method and device. Background Art

[0002] OLAP (On-Line Analytical Processing)-oriented column-oriented database management systems (hereinafter referred to as specialized databases) are primarily used for analytical processing applications in scenarios involving large data volumes. Currently, updating data in specialized databases typically involves synchronizing the data to be updated from the data source to a wide table in a distributed data warehouse tool. Once the synchronization is complete, the data in the wide table is inserted into a data table in the specialized database.

[0003] This data update method has two drawbacks: First, data in wide tables generally depends on other tables. Existing wide table data update scheduling methods include scheduled scheduling and scheduling based on dependencies. Scheduled scheduling ensures that tasks are completed within a fixed time, but some upstream data may not be ready when the task is executed, resulting in the acquired data not necessarily being the latest data. Scheduling based on dependencies ensures that the acquired data is the latest data, but it requires waiting for all upstream data to be ready before execution, resulting in uncontrollable task execution time. Second, the above data update method requires generating a wide table and inserting the data from the wide table into a data table in a specific database in batches. The entire process consumes a large amount of computing resources and may affect normal query operations. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a data update method and apparatus, which can ensure the execution time of data update tasks and that the acquired data is the latest data, and can also reduce the computing resource usage during the data update process.

[0005] To achieve the above object, according to one aspect of the present invention, a data updating method is provided.

[0006] The data updating method of an embodiment of the present invention includes: in response to a preset data update condition being met, determining a field to be updated from a data source, obtaining each piece of data of the same field to be updated in a partition to be stored in a target database; for the data to be updated of the same field to be updated in the same partition to be stored, generating an index identifier of the data to be updated that corresponds one-to-one with the primary key value of the data to be updated in the data source according to the primary key value of the data to be updated, writing the index identifier into a data table of the target database, and determining the index identifier field as a sorting field of the data table; wherein the index identifier increases in a generation order; based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, sequentially reading the data to be updated of the same field to be updated from the data source, converting the read data to be updated into a file to be updated of the same field to be updated in the same partition to be stored, and writing the file to be updated into the data table.

[0007] Optionally, obtaining the partitions to be stored in the target database for each piece of data of the same field to be updated includes: obtaining the primary key value of each piece of data of the same field to be updated in the data source, and determining the hash value of the primary key value; performing a modulo operation on the total number of partitions of the target database with the hash value, and determining the partition corresponding to the modulo result of any piece of data of the same field to be updated as the partition to be stored for the data.

[0008] Optionally, the method further comprises: before, after or simultaneously with writing the index identifier into the data table of the target database, writing the primary key value of the data to be updated into the data table.

[0009] Optionally, the method further includes: after writing the index identifier and primary key value of the data to be updated into the data table, generating an index file of the data table, a data file and a tag file of the index identifier field, and a data file and a tag file of the primary key field based on the index identifier and the primary key value.

[0010] Optionally, the file to be updated is a file that complies with the target database protocol; and the converting of the read data to be updated into the file to be updated in the same partition to be stored for the same field to be updated includes: performing serialization corresponding to the data type of the read data to be updated on the data to be updated; and compressing the serialization result into the file to be updated based on a pre-configured compressed data block size range and a particle size of the compressed data block.

[0011] Optionally, writing the file to be updated into the data table includes: creating a new segment directory in the same partition to be stored; writing the file to be updated into the new segment directory, and modifying the active segment of the same partition to be stored into the new segment directory.

[0012] Optionally, the file to be updated includes: a column file of the same field to be updated and a tag file of the same field to be updated.

[0013] To achieve the above object, according to another aspect of the present invention, a data updating device is provided.

[0014] The data updating device of an embodiment of the present invention may include: a partition determination unit, configured to: in response to a preset data update condition being met, determine a field to be updated from a data source, and obtain a partition to be stored in a target database for each piece of data of the same field to be updated; an index identifier generation unit, configured to: for the data to be updated of the same field to be updated in the same partition to be stored, generate an index identifier of the data to be updated, corresponding one-to-one to the primary key value of the data to be updated in the data source, write the index identifier into a data table of the target database, and determine the index identifier field as a sorting field of the data table; wherein the index identifier increases in a generation order; and a file generation unit, configured to read the data to be updated of the same field to be updated from the data source in an orderly manner according to the index identifier in the data table and the correspondence between the index identifier and the primary key value, convert the read data to be updated into a file to be updated of the same field to be updated in the same partition to be stored, and write the file to be updated into the data table.

[0015] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0016] An electronic device of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data update method provided by the present invention.

[0017] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0018] A computer-readable storage medium of the present invention stores a computer program, which implements the data updating method provided by the present invention when executed by a processor.

[0019] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects:

[0020] When the preset data update conditions are met, the server first determines the field to be updated from the data source, and obtains the data of the same field to be updated in the partition to be stored in the target database; then, for the data to be updated in the same partition to be stored in the same field to be updated, the server generates an index identifier for the data to be updated that corresponds one-to-one to the primary key value based on the primary key value of the data to be updated in the data source, writes the index identifier into the data table of the target database, and determines the index identifier field as the sorting field of the data table; thereafter, the server reads the data to be updated in the same field to be updated from the data source in an orderly manner based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, converts the read data to be updated into the file to be updated (including the column file and mark file of the field to be updated) of the same field to be updated in the same partition to be stored, and finally writes the file to be updated into the data table. In this way, the above index identifiers that increase in the order of generation are designed as the sorting fields of the target database, thereby accurately mapping the position of each field in the target database storage through the index identifier, and then generating a column file and a tag file for the same field to be updated that conforms to the target database protocol to perform data updates, thereby realizing the overall column-by-column update of the target database through the column file method, without the need to form a wide table of the data warehouse tool, nor the need to perform data insertion from the wide table to the target database, thereby reducing the computing resource usage and ensuring the stability of the normal business operation of the system. Since data in practice is often updated in columns, that is, the same column data is generally updated synchronously, and the update time of different column data may be different, the above column-by-column update method can not only ensure that the data of the field to be updated is the latest data, but also does not have to spend a long time waiting for the upstream data to be ready, so it can take into account both the timely execution of data update tasks and the updated data being the latest data.

[0021] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0023] Figure 1 Schematic diagram of the main steps of the data updating method according to an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the basic flow of the data updating method in an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of specific execution steps of the data updating method according to an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of the components of the data updating device according to an embodiment of the present invention;

[0027] Figure 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0028] Figure 6 2 is a schematic diagram of the structure of an electronic device used to implement the data updating method in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0031] Figure 1 2 is a schematic diagram of the main steps of the data updating method in an embodiment of the present invention.

[0032] like Figure 1 As shown, the data update method of the embodiment of the present invention can be executed by a server for executing data update processing, and the specific execution steps are as follows:

[0033] Step S101: In response to a preset data update condition being met, a field to be updated is determined from a data source, and a partition to be stored in a target database for each piece of data in the same field to be updated is obtained.

[0034] The above data update conditions can be any applicable conditions, such as a preset time, data status, etc. For example, the data update conditions may include: the time reaches midnight each day, and the number of updated data in the same field reaches 1000. When the data update conditions are met, the to-be-updated file can locate one or more to-be-updated fields that need to be synchronized to the target database from the data source (e.g., the business database). It is understood that at least one data item in the to-be-updated fields has changed.

[0035] After determining each field to be updated, the server can perform an update on the target database for the same field to be updated. In an embodiment of the present invention, the target database can be Clickhouse or another database with a similar structure, such as a column-based database like Vertica or Paraccel. The following description uses Clickhouse as an example. It should be understood that the following exemplary description of Clickhouse does not limit the application scenarios of the present invention.

[0036] In this step, the server retrieves the target database partitions where each piece of data for the same field to be updated is stored, and then updates the field in partitions. A partition refers to the storage space divided into sections of the target database's storage unit. A partition can consist of multiple parts. Each piece of data in the field to be updated is a record corresponding to the field value of the field to be updated.

[0037] In specific applications, the server can determine the corresponding partition to be stored based on the field value of the specified field of each piece of data in the field to be updated. For example, the server can determine the partition to be stored based on the primary key value of each piece of data (i.e., the data of the primary key field). That is, the server obtains the primary key value of each piece of data in the same field to be updated in the data source, calculates the hash value of the primary key value, and performs a modulo operation on the total number of partitions of the target database with the hash value. The partition corresponding to the modulo result of any piece of data in the same field to be updated is determined as the partition to be stored for the piece of data. The above steps can prevent the formation of overly large Clickhouse files and single partitions with too many rows of data, thereby optimizing Clickhouse query performance. Figure 2 This is a basic flow chart of the data updating method in the embodiment of the present invention. This step is Figure 2 The "partition by primary key hash" step (S201).

[0038] Step S102: For the data to be updated in the same to-be-updated field and the same to-be-stored partition, an index identifier is generated for the data to be updated, which corresponds one-to-one to the primary key value of the data to be updated in the data source, and the index identifier is written into the data table of the target database, and the index identifier field is determined as the sorting field of the data table.

[0039] In this step, the server can generate an index identifier for each piece of data in the field to be updated to facilitate data positioning. Specifically, for the data to be updated in the same field to be updated and in the same partition to be stored, the server can generate an index identifier based on the primary key value of the data to be updated in the data source, and the index identifier of each piece of data to be updated corresponds to its primary key value one by one. In particular, the index identifier is an integer, and the index identifier can be incremented in the order of its generation (i.e., monotonically increasing). The incrementing method can be continuous incrementing or discontinuous incrementing. Each index identifier needs to correspond to the corresponding primary key value, thereby achieving single-direction data sorting based on the index identifier. For example, the current maximum index identifier of the data in the Clickhouse data table is 100. Currently, three new data to be updated are added. The index identifiers of the three data to be updated can be determined as 101, 102, and 103, or the index identifiers of the three data to be updated can be determined as 102, 104, and 108. Thereafter, the server can store the correspondence between each index identifier and the primary key value. The purpose of generating index identifiers is to align the Clickhouse data with the data source data to ensure the consistency of the data order of the two. The above steps are Figure 2 The "index identifier is generated by partition" step (S202).

[0040] After generating the above index identifiers, the server can write the index identifiers to the data table of the target database to facilitate the subsequent location of the data to be updated. Optionally, before, after, or at the same time as writing the index identifiers to the data table of the target database, the server also writes the primary key value of the data to be updated to the data table to enable queries based on the primary key field in the ClickHouse data table. After writing is completed, the server can perform an optimization operation (optimize table) on the ClickHouse data table to store the index, reduce storage space, and improve access I / O efficiency, and ensure that each partition of the data table has only one active fragment.

[0041] Thereafter, in an optional technical solution, the server can generate the index file of the data table, the data file and tag file of the index identifier field, and the data file and tag file of the primary key field based on the index identifier and primary key value in the Clickhouse data table. The above index file can be the primary.idx, a first-level index file used for data query in the Clickhouse data table, the above data file can be a bin file used to store field data, and the above tag file can be an mrk file used to establish an association between the data file and the index file. The Clickhouse data table can provide data query capabilities based on the above index files, data files and tag files. The above steps are Figure 2 The "target database index file generation" step (S203).

[0042] Step S103: Based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, the data to be updated of the same field to be updated is read from the data source in an orderly manner, the read data to be updated is converted into a file to be updated in the same partition to be stored for the same field to be updated, and the file to be updated is written into the data table.

[0043] Before executing this step, the server may first determine the index identification field as the sorting field of the data table, so as to facilitate the orderly positioning of each data to be updated in the field to be updated. Optionally, the index identification field may also be determined as the index field of the data table at the same time, so as to facilitate the subsequent execution of data queries through the index identification. Then, the server may read each data to be updated of the same field to be updated from the data source based on the index identification in the data table and the correspondence between the index identification and the primary key value, that is, the server first reads each index identification as the sorting field, and determines the corresponding primary key value based on the pre-stored correspondence between the index identification and the primary key value, and finally uses the primary key value to locate the data to be updated from the data source.

[0044] Afterwards, the server can convert the read data to be updated into the updated files for the same field to be updated in the same partition to be stored based on the preset parameters of ClickHouse. Preferably, the above parameters can include the compressed data block size range (including the minimum compressed data block value min_compress_block_size and the maximum compressed data block value max_compress_block_Size) and the granularity of the compressed data block (index_granularity). For example, the following settings can be performed:

[0045] min_compress_block_size=65536B,

[0046] max_compress_block_size=1048576B,

[0047] index_granularity=8192.

[0048] For example, the following settings may also be performed:

[0049] (Minimum number of rows for fragment) min_rows_for_wide_part = 0,

[0050] (Minimum number of bytes for fragment) min_bytes_for_wide_part = 0,

[0051] Set the Data Part type of the ClickHouse wide table to wide so that each column of data is stored in a separate file;

[0052] Set index_granularity_bytes to 0 to disable ClickHouse from adaptively adjusting the granule size.

[0053] In specific applications, the server can perform serialization corresponding to the data type of the data to be updated based on the data type of the data to be updated that is read, that is, use the serialization logic for different data types pre-developed by Clickhouse to perform the above serialization. After the serialization is completed, the server can compress the serialization result into a file to be updated that complies with the target database protocol based on the pre-configured compressed data block size range and the particle size of the compressed data block. The above files to be updated may include column files (i.e., bin files) for the same field to be updated and marker files (i.e., mrk files) for the same field to be updated. It can be understood that the server can also use other applicable serialization methods and compression methods to generate files to be updated. The above steps are Figure 2 The "target database column file generation" step (S204).

[0054] In this step, after obtaining the file to be updated for the field to be updated, the server can write the file to be updated directly into the Clickhouse data table, or delay writing it into the Clickhouse data table to avoid affecting the current query task. Specifically, in the delayed writing step, the server first creates a new segment directory in the same partition to be stored, then writes the file to be updated into the newly created segment directory, and modifies the active segment of the same partition to be stored to the newly created segment directory. The above steps are Figure 2 In practical applications, the above steps S101 to S103 can be executed in a distributed file system.

[0055] In this way, a set of index identifiers that increase in order are designed based on the primary key, and the index identifiers are used as the sorting fields of the Clickhouse table. The index identifiers can be accurately mapped to the position of each field in the Clickhouse storage; the preset Clickhouse configuration parameters can be used to determine the protocol for generating Clickhouse column files and tag files in the distributed file system, ensuring that the files serialized in the distributed file system can be queried normally in Clickhouse.

[0056] Figure 3 This is a schematic diagram of the specific execution steps of the data updating method in an embodiment of the present invention, see Figure 3. Step S301: The server detects that the data update condition is met; Step S302: The server determines the field to be updated from the data source; Step S303: The server obtains the primary key value of each data of the same field to be updated in the data source, and calculates the hash value of the primary key value; Step S304: The server modulo the total number of partitions of the target database with the hash value, and determines the partition to be stored according to the modulo result; Step S305: For the data of the same partition to be stored, the server generates an index identifier that corresponds to the primary key value one by one and increases in the order of generation; Step S306: The server writes the index identifier and the primary key value into the target database data table; Step S307: The server generates the index file of the data table, the data file and tag file of the index identifier field, and the data file of the primary key field. File and mark file; Step S308: The server determines the index identification field as the sorting field of the data table; Step S309: The server reads the to-be-updated data of the same to-be-updated field from the data source in an orderly manner according to the index identification in the data table; Step S310: The server performs serialization corresponding to the data type of the to-be-updated data according to the data type of the read to-be-updated data; Step S311: The server compresses the serialization result into a to-be-updated file according to the pre-configured compressed data block size range and the granularity of the compressed data block; Step S312: The server creates a new segment directory in the same partition to be stored; Step S313: The server writes the to-be-updated file into the newly created segment directory and modifies the active segment of the same partition to be stored to the newly created segment directory. The specific execution details of the above steps have been described above and will not be repeated here.

[0057] In the technical solution of the embodiment of the present invention, the column-by-column update capability of the Clickhouse data table is provided. When the data of a certain column of the data warehouse tool is ready, the corresponding Clickhouse column file is directly generated in the distributed file system, and the column file is pushed to the Clickhouse storage, so as to achieve the purpose of updating the Clickhouse data table by column, eliminating the process of generating the wide table of the data warehouse tool, inserting Clickhouse data and intermediate calculations, and changing the data update method of Clickhouse from pushing and inserting data based on the wide table to directly updating the Clickhouse file, thereby solving the problems of data update timeliness and distributed computing resource occupation.

[0058] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information that may be involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0059] For ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0060] In order to better implement the above solutions of the embodiments of the present invention, relevant devices for implementing the above solutions are also provided below.

[0061] See also Figure 4 As shown, the data updating device 400 provided by the embodiment of the present invention may include: a partition determining unit 401 , an index identifier generating unit 402 and a file generating unit 403 .

[0062] Among them, the partition determination unit 401 can be used to: in response to the preset data update condition being met, determine the field to be updated from the data source, and obtain the partition to be stored of each data of the same field to be updated in the target database; the index identifier generation unit 402 can be used to: for the data to be updated of the same field to be updated in the same partition to be stored, generate an index identifier of the data to be updated, which corresponds one-to-one with the primary key value of the data to be updated in the data source, write the index identifier into the data table of the target database, and determine the index identifier field as the sorting field of the data table; wherein the index identifier increases in the order of generation; the file generation unit 403 can be used to read the data to be updated of the same field to be updated from the data source in an orderly manner based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, convert the read data to be updated into a file to be updated for the same field to be updated in the same partition to be stored, and write the file to be updated into the data table.

[0063] In an embodiment of the present invention, the partition determination unit 401 can be further used to: obtain the primary key value of each piece of data of the same field to be updated in the data source, and determine the hash value of the primary key value; perform a modulo operation on the total number of partitions of the target database with the hash value, and determine the partition corresponding to the modulo result of any piece of data of the same field to be updated as the partition to be stored for the data.

[0064] As a preferred solution, the index identifier generating unit 402 may be further configured to: write the primary key value of the to-be-updated data into the data table before, after or simultaneously with writing the index identifier into the data table of the target database.

[0065] Preferably, the index identifier generation unit 402 can be further used to: after writing the index identifier and primary key value of the data to be updated into the data table, generate the index file of the data table, the data file and tag file of the index identifier field, and the data file and tag file of the primary key field based on the index identifier and the primary key value.

[0066] In a specific application, the file to be updated is a file that complies with the target database protocol; and the file generation unit 403 can be further used to: perform serialization corresponding to the data type of the read data to be updated on the data to be updated; and compress the serialization result into the file to be updated based on a pre-configured compressed data block size range and the particle size of the compressed data block.

[0067] In an exemplary embodiment, the file generation unit 403 may be further configured to: create a new segment directory in the same partition to be stored; write the file to be updated into the new segment directory, and modify the active segment of the same partition to be stored to the new segment directory.

[0068] Furthermore, in an embodiment of the present invention, the to-be-updated file includes: a column file of the same to-be-updated field and a tag file of the same to-be-updated field.

[0069] According to the technical solution of the embodiment of the present invention, a set of index identifiers that increase in order are designed based on the primary key, and the index identifiers are used as the sorting fields of the Clickhouse table. The index identifiers can be accurately mapped to the position of each field in the Clickhouse storage; the preset Clickhouse configuration parameters can be used to determine the protocol for generating Clickhouse column files and tag files in the distributed file system, ensuring that the files serialized in the distributed file system can be queried normally in Clickhouse; in addition, when the index structure is determined, the Clickhouse data can be updated by column by updating the Clickhouse column file, thereby ensuring the execution time of the data update task and the data obtained is the latest data, while reducing the computing resource usage during the data update process.

[0070] Figure 5 An exemplary system architecture 500 is shown to which the data updating method or data updating apparatus according to the embodiment of the present invention can be applied.

[0071] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505 (this architecture is merely an example, and the components included in the specific architecture may be adjusted based on the specific application). Network 504 is used to provide a medium for communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0072] Users can use terminal devices 501, 502, 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, 503, such as big data analysis applications (only as an example).

[0073] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0074] The server 505 may be a server that provides various services, such as a backend server (for example only) that supports big data analysis applications operated by users using the terminal devices 501, 502, and 503. The backend server may process the received data update request and feed back the processing result (for example, the data update result - for example only) to the terminal devices 501, 502, and 503.

[0075] It should be noted that the data updating method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data updating device is generally provided in the server 505 .

[0076] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0077] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data update method provided by the present invention.

[0078] Reference below Figure 6 , which shows a schematic structural diagram of a computer system 600 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0079] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the computer system 600 are also stored in the RAM 603. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed in the storage section 608 as needed.

[0081] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from a removable medium 611. When the computer program is executed by the central processing unit 601, the above-described functions defined in the system of the present invention are performed.

[0082] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor comprising: a partition determination unit, an index identifier generation unit, and a file generation unit. The names of these units do not, in some cases, limit the units themselves. For example, the partition determination unit may also be described as "a unit that provides the index identifier generation unit with partitions to be stored."

[0085] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the device, the device executes the following steps: in response to a preset data update condition being met, determining a field to be updated from a data source, obtaining each piece of data of the same field to be updated in a partition to be stored in a target database; for the data to be updated of the same field to be updated in the same partition to be stored, generating an index identifier of the data to be updated that corresponds one-to-one with the primary key value of the data to be updated in the data source according to the primary key value of the data to be updated, writing the index identifier into a data table of the target database, and determining the index identifier field as a sorting field of the data table; wherein the index identifier increases in a generation order; and based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, sequentially reading the data to be updated of the same field to be updated from the data source, converting the read data to be updated into a file to be updated of the same field to be updated in the same partition to be stored, and writing the file to be updated into the data table.

[0086] In the technical solution of the embodiment of the present invention, when the preset data update conditions are met, the server first determines the field to be updated from the data source, and obtains the data of the same field to be updated in the partition to be stored in the target database; then, for the data to be updated in the same partition to be stored in the same field to be updated, the server generates an index identifier of the data to be updated that corresponds one-to-one with the primary key value of the data to be updated in the data source, writes the index identifier into the data table of the target database, and determines the index identifier field as the sorting field of the data table; thereafter, the server reads the data to be updated in the same field to be updated from the data source in an orderly manner based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, converts the read data to be updated into the file to be updated (including the column file and the mark file of the field to be updated) of the same field to be updated in the same partition to be stored, and finally writes the file to be updated into the data table. In this way, the above index identifiers that increase in the order of generation are designed as the sorting fields of the target database, thereby accurately mapping the position of each field in the target database storage through the index identifier, and then generating a column file and a tag file for the same field to be updated that conforms to the target database protocol to perform data updates, thereby realizing the overall column-by-column update of the target database through the column file method, without the need to form a wide table of the data warehouse tool, nor the need to perform data insertion from the wide table to the target database, thereby reducing the computing resource usage and ensuring the normal business operation stability of the system. Since data in practice is often updated in columns, that is, the same column data is generally updated synchronously, and the update time of different column data may be different, the above column-by-column update method can not only ensure that the data of the field to be updated is the latest data, but also does not have to spend a long time waiting for the upstream data to be ready, so it can take into account both the timely execution of data update tasks and the updated data being the latest data.

[0087] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data updating method, characterized in that: include: In response to a preset data update condition being met, determining a field to be updated from a data source, and obtaining a partition to be stored in a target database for each piece of data of the same field to be updated; For the data to be updated in the same to-be-updated field and the same to-be-stored partition, generate an index identifier for the data to be updated that corresponds one-to-one with the primary key value of the data to be updated according to the primary key value of the data source, write the index identifier into the data table of the target database, and determine the index identifier field as the sorting field of the data table; wherein the index identifiers are incremented in the order of generation; Based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, the data to be updated of the same field to be updated is read in order from the data source, the read data to be updated is converted into a file to be updated of the same field to be updated in the same partition to be stored, and the file to be updated is written into the data table.

2. The method according to claim 1, characterized in that The step of obtaining the partition to be stored in the target database for each piece of data of the same field to be updated includes: Obtain the primary key value of each piece of data of the same field to be updated in the data source, and determine the hash value of the primary key value; A modulo operation is performed on the total number of partitions of the target database using the hash value, and a partition corresponding to the modulo result of any piece of data of the same field to be updated is determined as the partition to be stored for the piece of data.

3. The method according to claim 1, characterized in that The method further comprises: Before, after or simultaneously with writing the index identifier into the data table of the target database, the primary key value of the data to be updated is written into the data table.

4. The method according to claim 3, characterized in that The method further comprises: After the index identifier and primary key value of the data to be updated are written into the data table, an index file of the data table, a data file and a tag file of the index identifier field, and a data file and a tag file of the primary key field are generated based on the index identifier and the primary key value.

5. The method according to claim 1, wherein The file to be updated is a file that complies with the target database protocol; And, converting the read data to be updated into a file to be updated in the same partition to be stored with the same field to be updated, includes: According to the data type of the read data to be updated, serialization corresponding to the data type is performed on the data to be updated; The serialization result is compressed into the file to be updated according to a pre-configured compressed data block size range and a particle size of the compressed data block.

6. The method according to claim 1, characterized in that Writing the file to be updated into the data table includes: Create a new fragment directory in the same partition to be stored; The file to be updated is written into a newly created segment directory, and the active segment of the same partition to be stored is modified to the newly created segment directory.

7. The method according to claim 1, characterized in that The files to be updated include: a column file of the same field to be updated and a tag file of the same field to be updated.

8. A data updating device, characterized in that: include: A partition determination unit is configured to: in response to a preset data update condition being met, determine a field to be updated from a data source, and obtain a partition to be stored in a target database for each piece of data in the same field to be updated; An index identifier generating unit is configured to: for the data to be updated in the same to-be-updated field and the same to-be-stored partition, generate an index identifier for the data to be updated, corresponding one-to-one to the primary key value of the data to be updated, based on the primary key value of the data to be updated in the data source, write the index identifier into a data table of the target database, and determine the index identifier field as a sorting field of the data table; wherein the index identifiers are incremented in the order of generation; A file generation unit is used to sequentially read the data to be updated of the same field to be updated from the data source based on the index identifier in the data table and the correspondence between the index identifier and the primary key value, convert the read data to be updated into a file to be updated for the same field to be updated in the same partition to be stored, and write the file to be updated into the data table.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images