A data refreshing method, device, apparatus and storage medium

Abstract

The application discloses a data refreshing method, device and equipment and a storage medium. The method comprises the following steps: in response to receiving an item detail table and an item dimension table, determining cluster shard nodes corresponding to each shard field value in the item dimension table; for each cluster shard node, storing a sub-item detail table split from the item detail table and a sub-item dimension table split from the item dimension table into the cluster shard node based on the shard field value corresponding to the cluster shard node; in response to detecting an updated sub-item dimension table, performing a data refreshing operation on a sub-item detail table in a target cluster shard node corresponding to the updated sub-item dimension table based on the updated sub-item dimension table, and obtaining an updated sub-item detail table. The application solves the problem of low data refreshing efficiency in the prior art and meets the application requirement of mass data updating.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data refresh method, apparatus, device, and storage medium. Background Technology

[0002] In e-commerce, databases store item detail tables and item dimension tables. Item detail tables record various item data, such as SKU_ID (Stock Keeping Unit), purchasing department, sales department, equipment number, brand, store ID, and item owner. Item dimension tables record item data related to a specific data dimension. For example, an item dimension table can include SKU_ID, brand, and item owner related to the SKU dimension, or store ID and item category related to the user dimension.

[0003] In practical applications, the organizational structure, brands, and product categories of e-commerce organizations change daily, causing the product information in the product dimension table to be updated accordingly. Existing query methods are primarily based on the product detail table, requiring frequent data refreshes of the product detail table based on the product dimension table to ensure the accuracy of query results.

[0004] In the process of realizing this invention, at least the following technical problems were found in the prior art:

[0005] Existing technologies primarily refresh the item details table within the source database (such as Hive). However, due to limitations in the source database's Spark (computing engine) resources and output time, the source database-based refresh method cannot meet the application requirements of daily large-scale data updates, resulting in low refresh efficiency. Summary of the Invention

[0006] This invention provides a data refresh method, apparatus, device, and storage medium to solve the problem of low efficiency in performing refresh operations in the source database and to meet the application requirements of large-scale data updates.

[0007] According to one aspect of the present invention, a data refresh method is provided, the method comprising:

[0008] In response to receiving the item details table and the item dimension table, the cluster shard node corresponding to each shard field value in the item dimension table is determined; wherein, the shard field value is the field value of the shard field that represents the data dimension of the item dimension table;

[0009] For each cluster shard node, the sub-item detail table split from the item detail table and the sub-item dimension table split from the item dimension table, based on the shard field value corresponding to the cluster shard node, are stored in the cluster shard node.

[0010] In response to the detection of an updated sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, based on the updated sub-item dimension table, to obtain the updated sub-item detail table.

[0011] According to another aspect of the present invention, a data refresh device is provided, the device comprising:

[0012] The cluster sharding node determination module is used to determine the cluster sharding node corresponding to each sharding field value in the item dimension table in response to receiving the item detail table and the item dimension table; wherein, the sharding field value is the field value of the sharding field that represents the data dimension of the item dimension table;

[0013] The item dimension table splitting module is used to store, for each cluster shard node, sub-item detail tables split from the item detail table and sub-item dimension tables split from the item dimension table, based on the sharding field value corresponding to the cluster shard node, into the cluster shard node.

[0014] The sub-item detail table determination module is used to respond to the detection of an updated sub-item dimension table, and in the target cluster shard node corresponding to the updated sub-item dimension table, perform a data refresh operation on the sub-item detail table in the target cluster shard node based on the updated sub-item dimension table to obtain the updated sub-item detail table.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data refresh method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data refresh method according to any embodiment of the present invention.

[0020] The technical solution of this invention stores sub-item dimension tables and sub-item detail tables, obtained by splitting the item dimension table based on the sharding field values, in cluster shard nodes corresponding to the respective sharding field values. In response to detecting an update to the sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, resulting in an updated sub-item detail table. This ensures that sub-item detail tables and sub-item dimension tables with the same sharding field dimension are stored locally on the cluster distribution nodes, avoiding network overhead caused by data transmission between cluster distribution nodes during data refresh. This solves the problem of low efficiency in performing refresh operations in existing technologies, meets the application requirement of completing large-scale data updates within a specified timeframe, and thus ensures a stable and accurate online query environment.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a data refresh method provided according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram illustrating the distribution of an item detail table and an item dimension table according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of another data refresh method provided according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a data refresh device according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0030] This embodiment is applicable to the situation of refreshing the item detail table based on the item dimension table. The method can be executed by a data refresh device, which can be implemented in hardware and / or software and can be configured in a columnar data management system.

[0031] Specifically, a data management system (DBMS) is software used to manipulate and manage databases, specifically for creating, using, and maintaining them. A columnar data management system is management software for columnar databases, which store data using a column-dependent storage architecture, primarily suitable for large-scale data processing and real-time querying. For example, a columnar data management system could be the ClickHouse system. ClickHouse allows tables and databases to be created, data loaded, and queries run at runtime without requiring server reconfiguration or restarts.

[0032] Figure 1 This is a flowchart of a data refresh method provided according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0033] S110. In response to receiving the item details table and the item dimension table, determine the cluster shard node corresponding to each shard field value in the item dimension table.

[0034] Specifically, the item details table records various item data, such as SKU_ID, purchasing department, sales department, equipment number, brand, store ID, and item owner. SKU is the basic unit representing the quantity of inventory entering and leaving the store; it can be a piece, box, or pallet, etc., and is now used as a shorthand for a unique product identification number. Each product has a unique SKU_ID.

[0035] Specifically, the item dimension table records item data related to a specific data dimension. For example, an item dimension table can include SKU_ID, brand, and item owner information related to the SKU dimension, or user account, store ID, and item category information related to the user dimension. The item detail table can contain some or all of the item data from the item dimension tables.

[0036] In this embodiment, the sharding field value is the field value of the sharding field representing the data dimension of the item dimension table. For example, assuming the data dimension corresponding to the item dimension table is the SKU dimension, the sharding field is SKU_ID, and the corresponding sharding field value can be Male_S, Male_M, Male_L, Male_XL, and Male_XXL, etc. Assuming the data dimension corresponding to the item dimension table is the user dimension, the sharding field is user_ID, and the corresponding sharding field value can specifically be each user's ID card number, medical insurance number, or passport number, etc. Here, the data dimension corresponding to the item dimension table is not limited; users can set it according to their actual needs.

[0037] In one embodiment, optionally, determining the cluster shard node corresponding to each sharding field in the item dimension table includes: determining at least one cluster shard node based on a preset sharding function and at least one sharding field value in the item dimension table; wherein, the preset sharding function is constructed based on the sharding field value, the number of cluster shards, and a hash distribution algorithm. In one embodiment, optionally, the preset sharding function satisfies the formula:

[0038] P = Coalesce((Cast(Abs(Hash(sharding field value)))as int)%%M,0)

[0039] Where P represents the node identifier of the cluster shard node, and M represents the number of cluster shard nodes.

[0040] Specifically, %% represents the modulo operation, Hash represents the hash value calculation operation, Abs represents the absolute value operation, Cast(x)as int represents the operation of converting x to int data type, and the Coalesce function means that if the expression is empty, it returns 0, and if the expression is not empty, it returns the corresponding non-empty value.

[0041] S120. For each cluster shard node, the sub-item detail table split from the item detail table and the sub-item dimension table split from the item dimension table, based on the shard field value corresponding to the cluster shard node, are stored in the cluster shard node.

[0042] Specifically, the sub-item detail table contains the detail data corresponding to the sharding field values ​​in the item detail table, and the sub-item dimension table contains the dimension table data corresponding to the sharding field values ​​in the item dimension table.

[0043] Figure 2 This is a schematic diagram illustrating the distribution of an item detail table and an item dimension table according to an embodiment of the present invention. Specifically, the ClickHouse system responds to the item detail table and the SKU item dimension table, wherein the data dimension corresponding to the SKU item dimension table is the SKU dimension. Based on the two sharding field values ​​Male_S and Male_M in the SKU item dimension table, sub-item detail tables and SKU sub-item dimension tables are determined. Specifically, cluster shard node 1 stores sub-item detail table 1 corresponding to Male_S and SKU sub-item dimension table 1, and cluster shard node 2 stores sub-item detail table 1 corresponding to Male_M and SKU sub-item dimension table 2.

[0044] Because the ClickHouse system requires processing large batches of data in a distributed cluster, data updates necessitate data transfer between multiple machines, resulting in significant network overhead. This invention addresses this by storing sub-item detail tables and sub-item dimension tables with the same sharding field dimension within the same cluster shard node. This ensures data distribution consistency, preventing subsequent data updates from traversing the distributed tables and effectively reducing network overhead.

[0045] S130. In response to detecting an updated sub-item dimension table, in the target cluster shard node corresponding to the updated sub-item dimension table, perform a data refresh operation on the sub-item detail table in the target cluster shard node based on the updated sub-item dimension table to obtain the updated sub-item detail table.

[0046] Specifically, based on a preset detection period, data checks are performed on the sub-item dimension tables in at least one cluster shard node to determine whether each cluster shard node has updated its sub-item dimension tables. For example, the preset detection period could be one hour, one day, or one week, etc.

[0047] In one embodiment, optionally, based on updating the sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node to obtain an updated sub-item detail table. This includes: based on updating the sub-item dimension table, performing a full data refresh operation on the sub-item detail table in the target cluster shard node to obtain an updated sub-item detail table. In this embodiment, the sub-item detail table contains all the data from the updated sub-item dimension table.

[0048] Specifically, data refresh operations can be either deletion or insertion, because for the ClickHouse system, full insertion is more efficient than updating.

[0049] The technical solution of this embodiment stores sub-item dimension tables and sub-item detail tables, obtained by splitting the item dimension table based on the sharding field values, in cluster shard nodes corresponding to the respective sharding field values. In response to detecting an update to the sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, resulting in an updated sub-item detail table. This ensures that both the sub-item detail table and the sub-item dimension table with the same sharding field dimension are stored locally on the cluster distribution nodes, avoiding network overhead caused by data transmission between cluster distribution nodes during data refresh. This solves the problem of low efficiency in performing refresh operations in existing technologies, meets the application requirement of completing large-scale data updates within a specified timeframe, and thus ensures a stable and accurate online query environment.

[0050] In another embodiment, the update method for the sub-item dimension table in the cluster shard node is further defined. Figure 3 This is a flowchart of another data refresh method provided according to an embodiment of the present invention, such as... Figure 3 As shown, the method includes:

[0051] S210. In response to receiving the item details table and the item dimension table, determine the cluster shard node corresponding to each shard field value in the item dimension table.

[0052] S220. For each cluster shard node, the sub-item detail table split from the item detail table and the sub-item dimension table split from the item dimension table, based on the shard field value corresponding to the cluster shard node, are stored in the cluster shard node.

[0053] S230. In response to receiving the updated data table, determine the target cluster shard node corresponding to at least one shard field value in the updated data table.

[0054] The updated data in the updated data table consists of data that needs to be updated in the sub-item dimension tables of each cluster shard node. In this embodiment, the updated data table includes the updated data, the shard field values ​​corresponding to each updated data, and the update identifier.

[0055] Specifically, the update identifier for different updated data can be the same, such as "update". However, the update identifier for different updated data can be different. If the update operation is a modification operation, the corresponding update identifier can be "update"; if the update operation is an insert operation, the corresponding update identifier can be "insert"; if the update operation is a delete operation, the corresponding update identifier can be "delete", and so on. There is no limitation on the update identifier here. For example, the update identifier can be composed of at least one of the following: uppercase letters, lowercase letters, text, special symbols, and punctuation marks.

[0056] For example, the updated data table can be sent by the source data storage system, which can be a Hive system.

[0057] It should be noted that the target cluster shard node here and the target cluster shard node that performs the refresh operation belong to the same batch of cluster shard nodes. In other words, the cluster shard node that performs the update operation on the stored sub-item dimension table is the same cluster shard node that performs the data refresh operation later.

[0058] S240. For each target cluster shard node, based on the update data and update identifier corresponding to the target cluster shard node in the update data table, perform an update operation on the sub-item dimension table in the target cluster shard node to obtain the updated sub-item dimension table.

[0059] Specifically, based on the target shard field value corresponding to the target cluster shard node, the update data table is split to obtain a sub-update data table corresponding to the target shard node. Then, based on the sub-update data table, an update operation is performed on the sub-item dimension table to obtain the updated sub-item dimension table. The sub-update data table contains the target shard field value, the updated data corresponding to the target shard field value, and an update identifier.

[0060] S250. In response to detecting an updated sub-item dimension table, in the target cluster shard node corresponding to the updated sub-item dimension table, perform a data refresh operation on the sub-item detail table in the target cluster shard node based on the updated sub-item dimension table to obtain the updated sub-item detail table.

[0061] In one embodiment, optionally, based on the updated sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node to obtain the updated sub-item detail table, including: obtaining the updated data in the updated sub-item dimension table based on the update identifier in the updated sub-item dimension table, and determining the data to be refreshed based on the updated data; and performing a data refresh operation on the sub-item detail table in the target cluster shard node based on the data to be refreshed to obtain the updated sub-item detail table.

[0062] Considering that only a portion of the data in the sub-item dimension table is updated, performing a full update of the entire sub-item detail table due to only partial updates would not meet timeliness requirements and would be highly uneconomical. The technical solution in this embodiment updates the sub-item dimension table based on the updated data and update identifiers in the updated data table during the update process, resulting in an updated sub-item dimension table. During data refresh, the sub-item detail table is refreshed based on the updated data corresponding to the update identifiers in the updated sub-item dimension table, thus solving the problem of excessive data volume during refresh and further improving data refresh efficiency.

[0063] Based on the above embodiments, optionally, obtaining the updated data in the updated sub-item dimension table based on the update identifier in the updated sub-item dimension table includes: loading the data in the updated sub-item dimension table stored in the hard disk storage area of ​​the target cluster shard node into a dictionary table stored in the memory storage area of ​​the target cluster shard node; and obtaining the updated data in the updated sub-item dimension table based on the update identifier in the dictionary table.

[0064] Specifically, the dictionary table is a storage medium provided by the ClickHouse system, stored in the memory storage area of ​​the target cluster shard nodes. The dictionary table is a data table defined in the form of key-value and attribute mapping.

[0065] Existing technologies use dictionary tables at the distributed cluster level, where a single dictionary table loads the entire item dimension table. This causes a spike in memory usage when retrieving updated data, impacting cluster stability. In contrast, this embodiment sets up a dictionary table in each cluster shard node. This dictionary table only loads the sub-item dimension tables from its respective shard node. On one hand, retrieving updated data from the dictionary table in the in-memory storage area significantly improves efficiency. On the other hand, since this dictionary table only loads sub-item dimension tables, it effectively ensures the stability of the distributed cluster.

[0066] Based on the above embodiments, optionally, after obtaining the updated sub-item details table, the method further includes: deleting the dictionary table in the memory storage area of ​​the target cluster shard node. From the perspective of the entire distributed cluster, considering the effective amount of memory in the distributed cluster, deleting the dictionary table can release memory, thereby further maintaining the stability of the distributed cluster.

[0067] Based on the above embodiments, optionally, determining the data to be refreshed based on the updated data includes: obtaining at least one preset data type, and for each preset data type, determining whether there are at least two updated data belonging to the preset data type; if so, performing an aggregation operation on the at least two updated data based on a preset aggregation method corresponding to the preset data type to obtain aggregated data; and taking the aggregated data and the remaining updated data other than the at least two updated data corresponding to each preset data type as the data to be refreshed.

[0068] For example, the preset data types include, but are not limited to, visitor count, page views, page dwell time, etc. The preset aggregation methods include, but are not limited to, at least one of summation aggregation, deduplication aggregation, and mean aggregation.

[0069] The advantage of this setup is that it effectively reduces the amount of data during the data refresh process, thereby further improving the efficiency of data refresh. Furthermore, since the data in the sub-item dimension tables all belong to the same data dimension, this embodiment can omit the step of filtering the item dimension table based on the data dimension before the aggregation operation, as required by existing technologies, thus simplifying the aggregation process.

[0070] Based on the above embodiments, optionally, a data refresh operation is performed on the sub-item detail table in the target cluster shard node based on the data to be refreshed to obtain an updated sub-item detail table, including: adding the data to be refreshed to the item version table and obtaining the target updated data corresponding to the target cluster shard node in the updated data table; verifying the item version table based on the target updated data; if the verification passes, copying the data to be refreshed in the item version table to the sub-item detail table in the target cluster shard node to obtain an updated sub-item detail table.

[0071] Specifically, the item version table can be understood as an intermediate table. All data to be refreshed is first pushed to the item version table for data temporary storage and verification.

[0072] Specifically, the target update data is compared with the data to be refreshed in the item version table. If the data to be refreshed is the same as its corresponding target update data, the verification passes. If at least one piece of data to be refreshed is different from its corresponding target update data, the verification fails.

[0073] The advantage of this setup is that, because the ClickHouse system uses a data refresh operation that deletes data before inserting it, the sub-item detail tables of the target cluster distributed nodes synchronized on the display page will have a window of inactivity during the data refresh period, resulting in noticeable data fluctuations for the user. This embodiment addresses this by adding the data to be refreshed to the item version table. After the item version table is validated, the data to be refreshed is copied to the sub-item detail tables in the target cluster shard nodes, resulting in an updated sub-item detail table. This avoids noticeable data fluctuations on the display page and improves the user experience. Furthermore, by validating the item version table, this embodiment maintains consistency between updated and unupdated data, preventing errors in the updated data due to insufficient system resources or system failures. This improves the accuracy of updating the sub-item detail tables and further ensures a stable and accurate online query environment.

[0074] Based on the above embodiments, optionally, a data refresh operation is performed on the sub-item detail table in the target cluster shard node based on the updated sub-item dimension table to obtain the updated sub-item detail table, including: determining the load rate corresponding to at least two cluster replica nodes in the target cluster shard node respectively; and in the cluster replica node with the lowest load rate, a data refresh operation is performed on the sub-item detail table in the cluster replica node based on the updated sub-item dimension table to obtain the updated sub-item detail table.

[0075] In this embodiment, the columnar data management system includes a processor, at least one cluster distribution node, and at least one cluster replica node corresponding to each cluster distribution node.

[0076] Specifically, the cluster replica nodes store backup data corresponding to the target cluster distribution nodes, meaning that each of the two cluster replica nodes stores a sub-item dimension table and a sub-item detail table.

[0077] For example, load rate can refer to the utilization rate of CPU (central processing unit) and / or the utilization rate of memory storage area.

[0078] The advantage of this configuration is that, considering the ClickHouse system's data refresh method involves migrating data to a designated cluster replica node (usually the first cluster replica node) of the cluster shard nodes, the performance of data refresh is limited by the performance of that shard node. This can lead to excessive load on a single cluster replica node, impacting the stability of the distributed cluster. This embodiment monitors the load of the cluster replica nodes and uses the cluster replica node with the lowest load to perform the data refresh operation. Based on concurrent execution logic, it dynamically adjusts the cluster replica nodes performing the data refresh operation, avoiding the performance bottleneck of a single cluster replica node affecting the overall data refresh performance and further improving the efficiency of data refresh.

[0079] Figure 4 This is a schematic diagram of a data refresh device according to an embodiment of the present invention. Figure 4 As shown, the device includes: a cluster shard node determination module 310, an item dimension table splitting module 320, and an update sub-item detail table determination module 330.

[0080] The cluster sharding node determination module 310 is used to determine the cluster sharding node corresponding to each sharding field value in the item dimension table in response to receiving the item detail table and the item dimension table; wherein, the sharding field value is the field value of the sharding field that represents the data dimension of the item dimension table.

[0081] The item dimension table splitting module 320 is used to store the sub-item detail tables split from the item detail table and the sub-item dimension tables split from the item dimension table, based on the sharding field values ​​corresponding to the cluster sharding node, into the cluster sharding node for each cluster sharding node.

[0082] The update sub-item detail table determination module 330 is used to respond to the detection of an updated sub-item dimension table, and perform a data refresh operation on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, based on the updated sub-item dimension table, to obtain the updated sub-item detail table.

[0083] The technical solution of this embodiment stores sub-item dimension tables and sub-item detail tables, obtained by splitting the item dimension table based on the sharding field values, in cluster shard nodes corresponding to the respective sharding field values. In response to detecting an update to the sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, resulting in an updated sub-item detail table. This ensures that both the sub-item detail table and the sub-item dimension table with the same sharding field dimension are stored locally on the cluster distribution nodes, avoiding network overhead caused by data transmission between cluster distribution nodes during data refresh. This solves the problem of low efficiency in performing refresh operations in existing technologies, meets the application requirement of completing large-scale data updates within a specified timeframe, and thus ensures a stable and accurate online query environment.

[0084] Optionally, based on the above embodiments, the device further includes:

[0085] The target cluster shard node determination module is used to determine the target cluster shard node corresponding to at least one shard field value in the update data table in response to receiving the update data table, before detecting the update of the sub-item dimension table; wherein, the update data table contains the updated data, the shard field value corresponding to each updated data, and the update identifier.

[0086] The module for determining the updated sub-item dimension table is used to perform update operations on the sub-item dimension table in the target cluster shard node for each target cluster shard node, based on the update data and update identifier corresponding to the target cluster shard node in the update data table, to obtain the updated sub-item dimension table.

[0087] Based on the above embodiments, optionally, the sub-item details table determination module 330 includes:

[0088] The data to be refreshed determination unit is used to obtain the updated data in the updated sub-item dimension table based on the update identifier in the updated sub-item dimension table, and determine the data to be refreshed based on the updated data;

[0089] The updated sub-item detail table determination unit is used to perform a data refresh operation on the sub-item detail table in the target cluster shard node based on the data to be refreshed, so as to obtain the updated sub-item detail table.

[0090] Based on the above embodiments, optionally, the data to be refreshed determination unit includes:

[0091] The dictionary table loading sub-unit is used to load the data from the updated sub-item dimension table stored in the hard disk storage area of ​​the target cluster shard node into the dictionary table stored in the memory storage area of ​​the target cluster shard node; and to obtain the updated data from the updated sub-item dimension table based on the update identifier in the dictionary table.

[0092] Based on the above embodiments, optionally, the data to be refreshed determination unit includes:

[0093] The update data aggregation subunit is used to obtain at least one preset data type, and for each preset data type, to determine whether there are at least two update data belonging to the preset data type; if so, to perform an aggregation operation on the at least two update data based on the preset aggregation method corresponding to the preset data type to obtain aggregated data; and to take the aggregated data and the remaining update data other than the at least two update data corresponding to each preset data type as the data to be refreshed.

[0094] Based on the above embodiments, optionally, the sub-item detail table determination unit is used for:

[0095] Add the data to be refreshed to the item version table and obtain the target update data corresponding to the target cluster shard node from the update data table; verify the item version table based on the target update data; if the verification passes, copy the data to be refreshed in the item version table to the sub-item detail table in the target cluster shard node to obtain the updated sub-item detail table.

[0096] Based on the above embodiments, optionally, the sub-item details table determination module 330 includes:

[0097] The cluster replica node selection unit is used to determine the load rates of at least two cluster replica nodes in the target cluster shard nodes. In the cluster replica node with the lowest load rate, a data refresh operation is performed on the sub-item detail table in the cluster replica node based on the updated sub-item dimension table to obtain the updated sub-item detail table.

[0098] Based on the above embodiments, optionally, the cluster shard node determination module 310 is specifically used for:

[0099] At least one cluster shard node is determined based on a preset sharding function and at least one sharding field value in the item dimension table; wherein, the preset sharding function is constructed based on the sharding field value, the number of cluster shards, and a hash distribution algorithm.

[0100] Based on the above embodiments, optionally, the preset slice function satisfies the formula:

[0101] P = Coalesce((Cast(Abs(Hash(sharding field value)))as int)%%M,0)

[0102] Where P represents the node identifier of the cluster shard node, and M represents the number of cluster shard nodes.

[0103] The data refresh device provided in the embodiments of the present invention can execute the data refresh method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0104] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0105] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor 11. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data refresh methods.

[0108] In some embodiments, the data refresh method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data refresh method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data refresh method by any other suitable means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), load-rate programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs used to implement the data refresh method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] This invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data refresh method, the method comprising:

[0112] In response to receiving the item details table and the item dimension table, determine the cluster shard node corresponding to each shard field value in the item dimension table; where the shard field value is the field value of the shard field that represents the data dimension of the item dimension table;

[0113] For each cluster shard node, the sub-item detail table split from the item detail table and the sub-item dimension table split from the item dimension table will be stored in the cluster shard node based on the shard field value corresponding to the cluster shard node.

[0114] In response to the detection of an updated sub-item dimension table, a data refresh operation is performed on the sub-item detail table in the target cluster shard node corresponding to the updated sub-item dimension table, based on the updated sub-item dimension table, to obtain the updated sub-item detail table.

[0115] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0119] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0120] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data refresh method, characterized by, The method comprises: in response to receiving the item specification table and the item dimension table, determining the cluster shard node corresponding to each shard field value in the item dimension table; wherein the shard field value is the field value of the shard field representing the data dimension of the item dimension table; for each cluster shard node, storing the sub-item specification table split from the item specification table and the sub-item dimension table split from the item dimension table based on the shard field value corresponding to the cluster shard node into the cluster shard node; in response to detecting the updated sub-item dimension table, performing a data refresh operation on the sub-item specification table in the target cluster shard node corresponding to the updated sub-item dimension table based on the updated sub-item dimension table to obtain an updated sub-item specification table, wherein the sub-item specification table contains all data in the updated sub-item dimension table; wherein, before responding to detecting the updated sub-item dimension table, the method further comprises: in response to receiving the update data table, determining the target cluster shard node corresponding to each shard field value in the update data table; wherein the update data table contains updated update data, shard field values corresponding to each update data, and update identifiers; for each target cluster shard node, performing an update operation on the sub-item dimension table in the target cluster shard node based on the update data and update identifiers corresponding to the target cluster shard node in the update data table to obtain an updated sub-item dimension table.

2. The method of claim 1, wherein, The method for performing a data refresh operation on the sub-item specification table in the target cluster shard node based on the updated sub-item dimension table to obtain an updated sub-item specification table comprises: based on the update identifier in the updated sub-item dimension table, obtaining the update data in the updated sub-item dimension table, and based on the update data, determining the data to be refreshed; based on the data to be refreshed, performing a data refresh operation on the sub-item specification table in the target cluster shard node to obtain an updated sub-item specification table.

3. The method of claim 2, wherein, The method for obtaining the update data in the updated sub-item dimension table based on the update identifier in the updated sub-item dimension table comprises: loading the data in the updated sub-item dimension table stored in the hard disk storage area in the target cluster shard node into a dictionary table stored in the memory storage area in the target cluster shard node; based on the update identifier in the dictionary table, obtaining the update data in the updated sub-item dimension table.

4. The method of claim 2, wherein, The method for determining the data to be refreshed based on the update data comprises: obtaining at least one preset data type, and for each preset data type, determining whether there are at least two update data belonging to the preset data type; if so, performing an aggregation operation on the at least two update data based on a preset aggregation mode corresponding to the preset data type to obtain aggregated data; the aggregated data and the remaining update data other than the at least two update data corresponding to each preset data type are taken as the data to be refreshed.

5. The method of claim 2, wherein, The data refreshing operation is performed on the sub-item detail table in the target cluster shard node based on the to-be-refreshed data, and an updated sub-item detail table is obtained, including: adding the to-be-refreshed data to the item version table, and obtaining target update data corresponding to the target cluster shard node in the update data table; verifying the item version table based on the target update data; if the verification is passed, copying the to-be-refreshed data in the item version table to the sub-item detail table in the target cluster shard node to obtain an updated sub-item detail table.

6. The method of claim 1, wherein, The data refreshing operation is performed on the sub-item detail table in the target cluster shard node based on the to-be-refreshed data, and an updated sub-item detail table is obtained, including: determining the load rates of at least two cluster replica nodes corresponding to the target cluster shard node respectively; in the cluster replica node with the smallest load rate, performing a data refreshing operation on the sub-item detail table in the cluster replica node based on the updated sub-item table to obtain an updated sub-item detail table.

7. The method according to any one of claims 1 to 6, characterized in that, The cluster shard node corresponding to each shard field value in the item table is determined, including: determining at least one cluster shard node based on a preset shard function and at least one shard field value in the item table; wherein the preset shard function is constructed based on a shard field value, a cluster shard number and a hash distribution algorithm.

8. The method of claim 7, wherein, The preset shard function satisfies the formula: ； wherein, represents a node identity of the cluster shard node, represents a number of the cluster shard nodes.

9. A data refresh apparatus, characterized by comprising: including: a cluster shard node determination module, configured to determine the cluster shard node corresponding to each shard field value in the item table in response to receiving an item detail table and an item table; wherein the shard field value is a field value of a shard field representing a data dimension of the item table; an item table splitting module, configured to split, for each cluster shard node, a sub-item detail table split from the item detail table based on the shard field value corresponding to the cluster shard node, and a sub-item table split from the item table, and store them in the cluster shard node; an updated sub-item detail table determination module, configured to perform a data refreshing operation on the sub-item detail table in the target cluster shard node based on the updated sub-item table in the target cluster shard node corresponding to the updated sub-item table in response to detecting the updated sub-item table, wherein the sub-item detail table contains all data in the updated sub-item table. The device further includes: a target cluster shard node determination module, configured to determine the target cluster shard node corresponding to at least one shard field value in the update data table in response to receiving the update data table in response to detecting the updated sub-item table; wherein the update data table contains updated update data, a shard field value corresponding to each update data, and an update identifier. An update sub-item dimension table determination module is configured to, for each target cluster shard node, perform an update operation on a sub-item dimension table in the target cluster shard node based on the update data corresponding to the target cluster shard node in the update data table and the update identifier, to obtain an updated sub-item dimension table.

10. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data refreshing method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the data refreshing method of any one of claims 1-8 when executed.

Images