Data update methods, devices, equipment and media

Abstract

This invention discloses a data updating method, apparatus, device, and medium. The method includes: acquiring at least one columnar data file corresponding to original data; the columnar data file includes a data index and at least one data row group; acquiring data to be updated and determining the primary key of the data to be updated; selecting a columnar data file to be updated from each of the columnar data files based on the primary key and data index of the data to be updated, and determining the data row group to be updated in the columnar data file to be updated; updating the data row group to be updated in the columnar data file to be updated based on the data to be updated, thereby obtaining a target columnar data file. This solution improves the updating efficiency of columnar data files.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data update method, apparatus, device and medium. Background Technology

[0002] Columnar storage stores data column-wise. Since data within the same column is of the same data type, efficient compression encoding can be achieved, reducing storage costs. Furthermore, during querying and analysis, only the necessary columns can be read, and identical columns facilitate efficient vector operations for rapid query analysis. However, when a small number of records need to be updated (e.g., modified or deleted), columnar data files require processing the entire file. When a columnar data file stores a large amount of data, the update efficiency becomes relatively low. Summary of the Invention

[0003] This invention provides a data update method, apparatus, device, and medium to improve the update efficiency of columnar data files.

[0004] According to one aspect of the present invention, a data update method is provided, comprising:

[0005] Obtain at least one columnar data file corresponding to the original data; the columnar data file includes a data index and at least one data row group;

[0006] Obtain the data to be updated and determine the primary key of the data to be updated;

[0007] Based on the primary key and data index of the data to be updated, select the columnar data file to be updated from each of the columnar data files, and determine the data row group to be updated in the columnar data file to be updated;

[0008] Based on the data to be updated, the data row groups in the columnar data file to be updated are updated to obtain the target columnar data file.

[0009] According to another aspect of the present invention, a data updating apparatus is provided, comprising:

[0010] A columnar data file acquisition module is used to acquire at least one columnar data file corresponding to the original data; the columnar data file includes a data index and at least one data row group;

[0011] The data to be updated acquisition module is used to acquire the data to be updated and determine the primary key of the data to be updated;

[0012] The module for determining the data row group to be updated is used to select the column data file to be updated from each of the column data files based on the primary key and data index of the data to be updated, and to determine the data row group to be updated in the column data file to be updated.

[0013] The target columnar data file acquisition module is used to update the data row group to be updated in the columnar data file to be updated according to the data to be updated, so as to obtain the target columnar data file.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data update method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data update method described in any embodiment of the present invention.

[0019] The present invention provides a data update scheme that, by determining the data row group to be updated, updates the data in the data row group. This only requires updating the data within the controllable range of the data row group to be updated, without processing the non-data row groups in the columnar data file to be updated, thus improving the update efficiency of the columnar data file.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a data update method provided in Embodiment 1 of the present invention;

[0023] Figure 2A This is a flowchart of a data update method provided in Embodiment 2 of the present invention;

[0024] Figure 2B This is a schematic diagram of a columnar data file provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of a data update device provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device that implements a data update method according to Embodiment 4 of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0028] Example 1

[0029] Figure 1 This is a flowchart of a data update method provided in Embodiment 1 of the present invention. This embodiment is applicable to updating data in a columnar data file. The method can be executed by a data update device, which can be implemented in hardware and / or software and can be configured in an electronic device that carries the data update function.

[0030] See Figure 1 The data update methods shown include:

[0031] S110. Obtain at least one columnar data file corresponding to the original data; the columnar data file includes a data index and at least one data row group.

[0032] In this invention, raw data refers to data that needs to be stored in a columnar format. The embodiments of this invention do not impose any limitations on the quantity and / or content of the raw data; these can be set by those skilled in the art based on experience. It should be noted that the raw data in these embodiments must have its own primary key to uniquely identify the raw data entry.

[0033] In this context, a columnar data file refers to a file used to store raw data. A row group is a collection that includes at least a portion of the raw data. A data index can be used to determine the range of raw data in a columnar data file that needs to be updated. Specifically, a data index can include a row group index and a file index. The row group index is used to identify the row groups in the columnar data file that need to be updated. The file index is used to identify the columnar data file itself that needs to be updated.

[0034] Understandably, by introducing row group indexes and file indexes, the original data in the columnar data file is recorded, which improves the efficiency of subsequent data updates.

[0035] Specifically, for a batch of raw data, according to a preset splitting rule, the raw data is split into at least one data row group; the raw data within each data row group is stored in a columnar manner, and each data row group is compressed and concatenated; while concatenating any data row group, the row group index of that data row group is recorded at the end of the file to which the data row group is concatenated; after all data row groups have been concatenated, the file index of the file is recorded below each row group index, finally obtaining a columnar data file. It should be noted that the embodiments of the present invention do not impose any limitations on the preset splitting rule, which can be set by those skilled in the art based on experience or needs. For example, the preset splitting rule may include the number of rows of raw data and / or the total number of bytes of raw data.

[0036] In this embodiment of the invention, a deletion flag column can be added to the last column of the data row group to mark whether any data has been deleted. It should be noted that this deletion is a logical deletion, not a physical deletion.

[0037] It should be noted that the data row groups in a columnar data file are not actually split files, and when there are a large number of data row groups in a columnar data file, it will put a burden on the file management system.

[0038] S120. Obtain the data to be updated and determine the primary key of the data to be updated.

[0039] In this context, the data to be updated refers to the data used to update the columnar data file. The primary key of the data to be updated is used to uniquely identify the data. It should be noted that this embodiment of the invention does not impose any limitation on the quantity and / or content of the data to be updated, and these can be set by a technician based on experience or needs.

[0040] S130. Based on the primary key and data index of the data to be updated, select the column data file to be updated from each column data file, and determine the data row group to be updated in the column data file to be updated.

[0041] Here, the columnar data file to be updated refers to the columnar data file that needs to be updated. The row group to be updated refers to the group of data rows that needs to be updated.

[0042] Specifically, based on the primary key and data index of the data to be updated, the columnar data file that needs to be updated is determined from at least one columnar data file; then, based on the primary key and data index of the data to be updated, the data row group that needs to be updated is determined from at least one data row group of the columnar data file that needs to be updated.

[0043] S140. Based on the data to be updated, update the data row groups in the columnar data file to be updated to obtain the target columnar data file.

[0044] The target columnar data file refers to the columnar data file after the data update is completed.

[0045] Specifically, after determining the data row group to be updated, the data row group to be updated is decompressed; based on the data to be updated, the data that needs to be updated is determined from the decompressed data row group to be updated; after the data update is completed, the updated data row group to be updated is compressed; the data row groups that do not need to be updated in the column data file to be updated are merged with the updated data row group to be updated to generate a new column data file to replace the column data file before the update, and the updated column data file is the target column data file.

[0046] It should be noted that both the pre-update and post-update columnar data files contain compressed data rows. A data row group is only decompressed when an update is required; once the update is complete, the data row group is compressed again. Data row groups that do not require updates do not need to be decompressed.

[0047] In an alternative embodiment, to ensure that the data index is compatible with the updated columnar data file, the data index can be updated when there are data updates in the data row groups of the columnar data file.

[0048] The present invention provides a data update scheme that, by determining the data row group to be updated, updates the data in the data row group. This only requires updating the data within the controllable range of the data row group to be updated, without processing the non-data row groups in the columnar data file to be updated, thus improving the update efficiency of the columnar data file.

[0049] Example 2

[0050] Figure 2AThis is a flowchart of a data update method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further refines the operation of "selecting a columnar data file to be updated from each columnar data file according to the primary key and data index of the data to be updated, and determining the data row group to be updated in the columnar data file" into "determining the columnar data file to be updated from at least one columnar data file according to the primary key and file index of the data to be updated; determining the data row group to be updated in the columnar data file and its position within the columnar data file according to the primary key and row group index of the data to be updated," thereby improving the mechanism for determining the data row group to be updated. After "updating the data row group to be updated in the columnar data file according to the data to be updated," the operation of "updating the file index of the columnar data file to be updated and the row group index corresponding to the data row group to be updated according to the primary key of each data in the updated data row group to be updated" is added to improve the mechanism for determining the target columnar data file. It should be noted that for parts not detailed in this embodiment, please refer to the descriptions in other embodiments.

[0051] See Figure 2A The data update methods shown include:

[0052] S210. Obtain at least one columnar data file corresponding to the original data; the columnar data file includes a data index and at least one data row group; the data index includes a row group index and a file index.

[0053] In one alternative embodiment, the data index is determined based on the primary key of each original data in the data row group.

[0054] Understandably, determining the data index based on the primary key of each original data avoids the problem of determining the data index directly based on the original data, which would result in a large data index, leading to a large data index space occupation and increasing the operating pressure of the file management system. Reducing the data index space occupation reduces the operating pressure of the file management system.

[0055] Specifically, row group indexes can be determined based on the primary keys of the original data in any data row group; file indexes can be determined based on the primary keys of the original data in any columnar data file.

[0056] S220. Obtain the data to be updated and determine the primary key of the data to be updated.

[0057] S230. Determine the column data file to be updated from at least one column data file based on the primary key and file index of the data to be updated.

[0058] Specifically, for any columnar data file, the file index includes the primary key of each original data in the columnar data file; the primary key of the data to be updated is input into the file index, and the file index can determine whether the corresponding original data exists in the columnar data file; if so, the columnar data file is used as the columnar data file to be updated; if not, the file indexes of other columnar data files are traversed.

[0059] S240. Based on the primary key and row group index of the data to be updated, determine the row group of the data to be updated in the columnar data file, and the position of the row group of the data to be updated in the columnar data file.

[0060] Specifically, row group indexes include conditional indexes and / or range indexes. Both conditional and range indexes can determine whether the original data corresponding to the data to be updated exists in the data row group; that is, both conditional and range indexes can identify the data row group to be updated.

[0061] Understandably, by introducing judgment indexes and range indexes, the diversity of row group indexes is improved, avoiding the situation where the accuracy is not high when determining the data row group to be updated based on a single row group index, and thus improving the accuracy of the determined data row group to be updated.

[0062] In one optional embodiment, determining the row group of data to be updated in the columnar data file to be updated, and the position of the row group of data to be updated in the columnar data file to be updated, based on the primary key and row group index of the data to be updated, includes: determining the row group of data to be updated in the columnar data file to be updated based on the primary key and judgment index of the data to be updated; and determining the position of the row group of data to be updated in the columnar data file to be updated based on the row group position corresponding to the row group index.

[0063] Here, row group position refers to the starting position or offset of a data row group in a columnar data file. Specifically, row group position is used to locate data row groups.

[0064] In another optional embodiment, determining the row group of data to be updated in the columnar data file to be updated, and the position of the row group of data to be updated in the columnar data file to be updated, based on the primary key and row group index of the data to be updated, includes: determining the row group of data to be updated in the columnar data file to be updated based on the primary key and range index of the data to be updated; and determining the position of the row group of data to be updated in the columnar data file to be updated based on the row group position corresponding to the row group index.

[0065] In another optional embodiment, determining the data row group to be updated in the columnar data file to be updated, and the position of the data row group to be updated in the columnar data file to be updated, based on the primary key and row group index of the data to be updated, includes: determining optional data row groups from the columnar data file to be updated based on the range index; determining the data row group to be updated from the optional data row groups based on the judgment index; and determining the position of the data row group to be updated in the columnar data file to be updated based on the row group position corresponding to the row group index.

[0066] In this context, an optional data row group refers to a row group that may contain original data corresponding to the data to be updated. The number of optional data row groups can be at least one, and this embodiment of the invention does not impose any limitation on this.

[0067] Understandably, by using a range index to identify at least one optional data row group, and then using a decision index to identify the data row group to be updated from the at least one optional data row group, the efficiency of identifying the data row group to be updated is improved. At the same time, the combination of range index and decision index improves the accuracy of the identified data row group to be updated.

[0068] It should be noted that when all the original data in the columnar data file to be updated are stored in order of primary key size, the data row group to be updated can be determined based on the range index to reduce the amount of computation and improve the efficiency of determining the data row group to be updated. When at least some of the original data in the columnar data file to be updated are not stored in order of primary key size, the data row group to be updated can be determined based on the judgment index, or based on the judgment index and the range index, to improve the accuracy of the determined data row group to be updated.

[0069] S250. Update the data row groups in the column data file to be updated based on the data to be updated.

[0070] S260. Based on the primary key of each data in the updated data row group to be updated, update the file index of the columnar data file to be updated and the row group index corresponding to the data row group to be updated to obtain the target columnar data file.

[0071] Specifically, for any data row group in the columnar data file that has been updated, after the update of that data row group is completed, the row group index of that data row group is updated according to the primary key of each data in the data row group, and the row group positions corresponding to each row group index in the columnar data file are updated. For any columnar data file that has been updated, after the update of that columnar data file is completed, the file index of that columnar data file is updated according to the primary key of each data in the columnar data file.

[0072] For example, the target columnar data file may include the updated file index, the updated data row groups, the updated row group index, and the data row groups that do not have data updates.

[0073] This invention provides a data update scheme that uses a file index to determine the columnar data file to be updated and a row group index to determine the data row group to be updated, thereby improving the efficiency of data updates. At the same time, updating the file index and row group index improves the compatibility with the columnar data file. Timely updating of the file index and row group index facilitates subsequent data updates.

[0074] In one alternative embodiment, the index can be a Bloom filter; the range index can be a primary key range; and the file index can be a Bloom filter.

[0075] Specifically, the decision index can be a Bloom filter determined based on the primary keys of the original data in the data row group; the range index can be a primary key range determined based on the primary keys of the original data in the data row group; and the file index can be a Bloom filter determined based on the primary keys of the original data in the columnar data file. The Bloom filter used as a file index can be generated by efficiently merging the various Bloom filters used as decision indices in the columnar data file.

[0076] For example, see Figure 2B The diagram illustrates a columnar data file. This file contains data row groups A, B, and C, as well as a row group index A' corresponding to data row group A, which includes Bloom filter a; a row group index B' corresponding to data row group B, which includes Bloom filter b; a row group index C' corresponding to data row group C, which includes Bloom filter c; and the file index of the columnar data file, i.e., Bloom filter d. The Bloom filters are used to determine the columnar data file to be updated and the data row groups to be updated.

[0077] Understandably, by introducing a Bloom filter as a file index, it's possible to quickly determine whether any columnar data file contains original data corresponding to the data to be updated; similarly, by introducing a Bloom filter as a judgment index, it's possible to quickly determine whether any data row group contains original data corresponding to the data to be updated, thus improving the efficiency of data updates. Introducing a primary key range as a range index can reduce the amount of computation.

[0078] It's important to note that Bloom filters can produce false positives. For example, using a Bloom filter as a decision index, if a given group of rows doesn't actually contain the original data corresponding to the data to be updated, but the Bloom filter incorrectly identifies it as present, this is a false positive. False positives impact performance but not accuracy, and because Bloom filters have a low false positive rate, the impact on performance is minimal. Therefore, using Bloom filters as file indexes or decision indexes is generally more effective.

[0079] For example, when there is a batch of data to be updated, for any data in that batch, the status of the data to be updated is determined, i.e., modified or added. Specifically, this determination can be made in response to an operation command. If the operation command is an add operation, then in response to the add operation, the columnar data file to be updated is determined, and the data to be updated is added to the data row group to be updated that meets the preset addition conditions. Here, the add operation refers to the command to control the addition of data. This embodiment of the invention does not limit the preset addition conditions in any way, and can be set by a technician based on experience or needs. For example, the preset addition condition can be a data threshold for a data row group. The data threshold refers to the upper limit of the number of data that a data row group can contain. This embodiment of the invention does not limit the size of the data threshold in any way, and can be set by a technician based on experience. When there is no data row group in the columnar data file to be updated that meets the preset addition conditions, the data to be updated can be added to the columnar data file as a separate data row group.

[0080] Continuing the previous example, if the operation command is a modification operation, then in response to the modification operation, the columnar data file to be updated is determined based on the primary key and file index of the data to be updated; then, the row groups to be updated within the columnar data file are determined based on the primary key and row group index of the data to be updated; after determining the row groups to be updated, they are decompressed; the corresponding original data in the decompressed row groups is updated according to the data to be updated; finally, the updated row groups are compressed. Here, a modification operation refers to an instruction that controls the modification of data.

[0081] Continuing from the previous example, for any columnar data file to be updated, after the data update is completed, the data index in the columnar data file to be updated is updated to obtain the target columnar data file.

[0082] It should be noted that when updating data (such as modifying or adding data), only the data row groups that have been updated need to be decompressed, while the data row groups that have not been updated do not need to be decompressed, in order to improve the efficiency of data updates.

[0083] It should be noted that when any data row group in the columnar data file to be updated is updated, the row group position of each data row group in the columnar data file to be updated may change. Therefore, it is necessary to update the row group position corresponding to the row group index of each data row group.

[0084] It should be noted that in this embodiment of the invention, multiple pieces of data to be updated can be processed simultaneously to improve the efficiency of data updates. For example, multiple pieces of data to be updated can be processed simultaneously using a multi-threaded concurrent approach.

[0085] It should be noted that, in this embodiment of the invention, the status of the data to be updated can be determined through file indexes. For example, for any columnar data file, if the file index corresponding to the data to be updated indicates that the original data exists within that columnar data file, then the status of the data to be updated is "modified." If the file index corresponding to the data to be updated indicates that the original data does not exist within that columnar data file, then the file indexes corresponding to other columnar data files are traversed. If all file indexes determine that the original data corresponding to the data to be updated does not exist in their own columnar data files, then the status of the data to be updated is "added."

[0086] Example 3

[0087] Figure 3 This is a schematic diagram of a data update device provided in Embodiment 3 of the present invention. This embodiment is applicable to updating data in a columnar data file. The method can be executed by a data update device, which can be implemented in hardware and / or software and can be configured in an electronic device that carries the data update function.

[0088] like Figure 3 As shown, the device includes: a columnar data file acquisition module 310, a data to be updated acquisition module 320, a data row group to be updated determination module 330, and a target columnar data file acquisition module 340. Among them,

[0089] The columnar data file acquisition module 310 is used to acquire at least one columnar data file corresponding to the original data; the columnar data file includes a data index and at least one data row group;

[0090] The data to be updated acquisition module 320 is used to acquire the data to be updated and determine the primary key of the data to be updated;

[0091] The module 330 for determining the data row group to be updated is used to select the column data file to be updated from each column data file based on the primary key and data index of the data to be updated, and to determine the data row group to be updated in the column data file to be updated.

[0092] The target columnar data file acquisition module 340 is used to update the data row groups to be updated in the columnar data file to be updated based on the data to be updated, so as to obtain the target columnar data file.

[0093] The present invention provides a data update scheme that, by determining the data row group to be updated, updates the data in the data row group. This only requires updating the data within the controllable range of the data row group to be updated, without processing the non-data row groups in the columnar data file to be updated, thus improving the update efficiency of the columnar data file.

[0094] Optionally, data indexes include row group indexes and file indexes.

[0095] Optionally, the data row group determination module 330 to be updated includes:

[0096] The columnar data file to be updated determination unit is used to determine the columnar data file to be updated from at least one columnar data file based on the primary key and file index of the data to be updated;

[0097] The data row group determination unit is used to determine the data row group to be updated in the columnar data file to be updated, and the position of the data row group to be updated in the columnar data file to be updated, based on the primary key and row group index of the data to be updated.

[0098] Accordingly, after updating the data row groups in the columnar data file to be updated based on the data to be updated, the device further includes:

[0099] The data index update module is used to update the file index of the columnar data file to be updated and the row group index corresponding to the data row group to be updated, based on the primary key of each data in the updated data row group.

[0100] Optionally, row group indexes include conditional indexes and / or range indexes.

[0101] Optionally, the unit for determining the data row group to be updated is specifically used for:

[0102] Based on the range index, determine the optional data row groups from the columnar data file to be updated;

[0103] Based on the judgment index, determine the data row group to be updated from the optional data row groups;

[0104] Determine the position of the row group to be updated in the columnar data file based on the row group position corresponding to the row group index.

[0105] Optionally, the data index is determined based on the primary key of each original data in the data row group.

[0106] Optionally, determine if the index is a Bloom filter; if the range index is a primary key range; if the file index is a Bloom filter.

[0107] The data update apparatus provided in the embodiments of the present invention can execute the data update method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each data update method.

[0108] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision and disclosure of the original data, the primary key of the original data, the data to be updated and the primary key of the data to be updated, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] Example 4

[0110] Figure 4 This is a schematic diagram of the structure of an electronic device for implementing a data update method according to Embodiment 4 of the present invention. The electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0112] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as data update methods.

[0114] In some embodiments, the data update method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the data update method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the data update method by any other suitable means (e.g., by means of firmware).

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data updating method characterized by comprising: include: Obtain at least one columnar data file corresponding to the original data; The columnar data file includes a data index and at least one data row group; wherein, the data row group has a deletion flag column, used to mark whether the original data in the data row group has been logically deleted; Obtain the data to be updated and determine the primary key of the data to be updated; Based on the primary key and data index of the data to be updated, select the columnar data file to be updated from each of the columnar data files, and determine the data row group to be updated in the columnar data file to be updated; Based on the data to be updated, the data row groups in the columnar data file to be updated are updated to obtain the target columnar data file; The data index includes row group index and file index; After updating the data row group in the columnar data file according to the data to be updated, the method further includes: Based on the primary key of each data in the updated data row group to be updated, update the file index of the columnar data file to be updated and the row group index corresponding to the data row group to be updated; The step of updating the data row group in the columnar data file to be updated based on the data to be updated, to obtain the target columnar data file, includes: In response to the operation that the data to be updated is to modify existing data, the row group of the data to be updated in the columnar data file to be updated is updated according to the data to be updated, and the data index in the columnar data file to be updated is updated to obtain the target columnar data file; In response to the operation that the data to be updated is newly added data, the columnar data file to be updated is determined, and the data to be updated is added to the row group of data to be updated that meets the preset addition conditions. The data index in the columnar data file to be updated is then updated to obtain the target columnar data file. The preset addition conditions are data thresholds for the row groups, where the data threshold is the upper limit of the number of data that a row group can contain. Alternatively, if no row group of data to be updated meets the preset addition conditions exists in the columnar data file to be updated, the data to be updated is added as a new row group to the columnar data file to be updated, and the data index in the columnar data file to be updated is then updated to obtain the target columnar data file.

2. The method of claim 1, wherein, The step of selecting the columnar data file to be updated from each of the columnar data files based on the primary key and data index of the data to be updated, and determining the data row group to be updated in the columnar data file to be updated, includes: Based on the primary key of the data to be updated and the file index, determine the columnar data file to be updated from at least one columnar data file; Based on the primary key of the data to be updated and the row group index, determine the row group of the data to be updated in the columnar data file, and the position of the row group of the data to be updated in the columnar data file.

3. The method of claim 2, wherein, The row group index includes a judgment index and / or a range index.

4. The method of claim 3, wherein, The step of determining the row group of data to be updated in the columnar data file and the position of the row group of data to be updated in the columnar data file, based on the primary key of the data to be updated and the row group index, includes: Based on the interval index, determine the optional data row group from the columnar data file to be updated; Based on the judgment index, determine the data row group to be updated from the optional data row group; The position of the data row group to be updated in the columnar data file to be updated is determined based on the row group position corresponding to the row group index.

5. The method according to any of claims 1-2, characterized by, The data index is determined based on the primary key of each original data in the data row group.

6. The method of claim 3, wherein, The judgment index is a Bloom filter; the interval index is a primary key interval; and the file index is a Bloom filter.

7. A data updating apparatus characterized by comprising: include: The columnar data file acquisition module is used to acquire at least one columnar data file corresponding to the original data; The columnar data file includes a data index and at least one data row group; wherein, the data row group has a deletion flag column, used to mark whether the original data in the data row group has been logically deleted; The data to be updated acquisition module is used to acquire the data to be updated and determine the primary key of the data to be updated; The module for determining the data row group to be updated is used to select the column data file to be updated from each of the column data files based on the primary key and data index of the data to be updated, and to determine the data row group to be updated in the column data file to be updated. The target columnar data file acquisition module is used to update the data row group to be updated in the columnar data file to be updated according to the data to be updated, so as to obtain the target columnar data file; The data index includes row group index and file index; After updating the row group of data to be updated in the columnar data file according to the data to be updated, the apparatus further includes: The data index update module is used to update the file index of the columnar data file to be updated and the row group index corresponding to the data row group to be updated based on the primary key of each data in the updated data row group to be updated. The step of updating the data row group in the columnar data file to be updated based on the data to be updated, to obtain the target columnar data file, includes: In response to the operation that the data to be updated is to modify existing data, the row group of the data to be updated in the columnar data file to be updated is updated according to the data to be updated, and the data index in the columnar data file to be updated is updated to obtain the target columnar data file; In response to the operation that the data to be updated is newly added data, the columnar data file to be updated is determined, and the data to be updated is added to the row group of data to be updated that meets the preset addition conditions. The data index in the columnar data file to be updated is then updated to obtain the target columnar data file. The preset addition conditions are data thresholds for the row groups, where the data threshold is the upper limit of the number of data that a row group can contain. Alternatively, if no row group of data to be updated meets the preset addition conditions exists in the columnar data file to be updated, the data to be updated is added as a new row group to the columnar data file to be updated, and the data index in the columnar data file to be updated is then updated to obtain the target columnar data file.

8. An electronic device, comprising: include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a data update method as described in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements a data update method as described in any one of claims 1-6.

Images