Data updating method and device, equipment and storage medium

Abstract

The present disclosure provides a data updating method and device, equipment and storage medium, relates to the field of artificial intelligence, in particular to the technical field of big data, machine learning and the like, and can be applied to the field of smart finance. The specific implementation scheme is: obtaining to-be-updated data, performing sharding processing on the to-be-updated data to obtain a plurality of first data pieces; the first data piece represents a part of the to-be-updated data; storing the first data piece into a preset database; the preset database stores initial data, and the initial data is data before the to-be-updated data is updated; if it is determined that each first data piece is stored into the preset database, then the data query version is switched from the version of the initial data to the version of the to-be-updated data; the data query version represents the version of the data to be obtained when data query is performed on the preset database. The present disclosure can avoid mixing of new and old versions of data in the database, and improve data updating efficiency and data query accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data, machine learning and the like in the field of artificial intelligence, and can be applied to the field of smart finance and the like, and particularly relates to a data updating method and device, equipment and a storage medium. BACKGROUND

[0002] A large amount of data can be stored in a database, for example, a large amount of feature data of users can be stored, and the feature data can represent the behavior of the users. An artificial intelligence model can query the feature data from the database and perform model inference according to the feature data.

[0003] However, the behavior of the users changes every day, and therefore, the data stored in the database needs to be updated in a timely and efficient manner to ensure the effect of data query. SUMMARY

[0004] The present disclosure provides a data updating method, device, equipment and a storage medium.

[0005] According to a first aspect of the present disclosure, a data updating method is provided, comprising:

[0006] obtaining to-be-updated data, performing sharding processing on the to-be-updated data to obtain a plurality of first data pieces; wherein the first data piece represents a part of the to-be-updated data;

[0007] storing the first data pieces into a preset database; wherein the preset database stores initial data, and the initial data is data before the to-be-updated data is updated;

[0008] if it is determined that each first data piece is stored into the preset database, switching a data query version from a version of the initial data to a version of the to-be-updated data; wherein the data query version represents a version of data to be obtained when data query is performed on the preset database.

[0009] According to a second aspect of the present disclosure, a data updating device is provided, comprising:

[0010] a sharding unit, configured to obtain to-be-updated data, perform sharding processing on the to-be-updated data to obtain a plurality of first data pieces; wherein the first data piece represents a part of the to-be-updated data;

[0011] a storage unit, configured to store the first data pieces into a preset database; wherein the preset database stores initial data, and the initial data is data before the to-be-updated data is updated;

[0012] An updating unit is used to switch the data query version from the version of the initial data to the version of the data to be updated if it is determined that all first data slices are stored in the preset database; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the data updating efficiency in the database is improved and the data query effect is enhanced.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 It is a database update logic diagram in the related art;

[0023] Figure 2 is a flow chart of a data updating method provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of a data updating method provided according to an embodiment of the present disclosure;

[0025] Figure 4 This is a database update logic diagram provided according to an embodiment of the present disclosure;

[0026] Figure 5 is a process architecture diagram of a database update method provided according to an embodiment of the present disclosure;

[0027] Figure 6 is a structural block diagram of a data updating device provided according to an embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of a data updating device provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a block diagram of an electronic device for implementing the data updating method according to an embodiment of the present disclosure;

[0030] Figure 9 It is a block diagram of an electronic device used to implement the data updating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In model inference scenarios, the database can store a large amount of user feature data, allowing the model to directly perform model inference by obtaining feature data from the database. Feature data can be extracted from user behavior data, but user behavior data can change daily, so the feature data in the database needs to be frequently updated.

[0033] Currently, data updates for key-value (KV) databases like Redis (Remote Dictionary Server) and RocksDB (RocksDatabase) are performed online, one-by-one, in real time. This is inefficient, and an update of billions of records can take days. During the import process, new and old data coexist, making it impossible to completely clear differences, hindering overall atomic validation and rapid rollback, and impacting model inference performance.

[0034] Figure 1 Updates the schematic diagram for the database in the related technology. Figure 1As shown, the original file is the data to be updated, the storage node is a computer instance, and there are n storage nodes. The storage file is located in the database, and each node corresponds to a storage file, that is, there are n storage files. The data to be updated is hashed and sharded, and the sharded data can be imported into the corresponding storage file through different nodes. During the import process, the imported data is in the original KV format, that is, the data imported into the database is in KV format, but the underlying file format of the database is mostly not in KV format, so format conversion is also required in the database, resulting in low database data update efficiency, mixed storage of new and old data in the database, and inability to accurately query and obtain data in the database in a timely manner.

[0035] The present disclosure provides a data update method, device, equipment and storage medium, which are applied to technical fields such as big data and machine learning in the field of artificial intelligence, and can be applied to fields such as smart finance to improve data update efficiency and enhance data query effects.

[0036] It should be noted that the data in this embodiment is not targeted at a specific user and does not reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0037] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figure 2-Figure 9 对实施例进行进一步细化。

[0039] Figure 2 FIG. 1 is a flow chart of a data updating method according to an embodiment of the present disclosure, which can be executed by a data updating device. Figure 2 As shown, the method includes the following steps:

[0040] S201 , obtaining data to be updated, and performing sharding processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated.

[0041] For example, a database is pre-set as a preset database, which can store a large amount of data. The data in the preset database may need to be updated at any time. The data currently stored in the preset database is used as the initial data, that is, the data to be updated. The data that replaces the initial data is used as the data to be updated, that is, the data after the initial data is updated.

[0042] When the preset database needs to be updated, the data to be updated is obtained, for example, the data to be updated can be managed or uploaded by a staff member. If the data to be updated is received, the update operation of the preset database can be started, that is, the data to be updated needs to be updated into the preset database.

[0043] After obtaining the data to be updated, a sharding operation can be performed on the data to be updated, that is, the data to be updated is divided into multiple data slices as first data slices. Each first data slice can contain different content in the data to be updated, that is, each first data slice can represent a local portion of the data to be updated, that is, a portion of the data to be updated. In this embodiment, the processing process of the sharding operation is not specifically limited. For example, a maximum amount of data that each first data slice can contain can be preset, and the data to be updated can be divided according to the preset maximum amount of data so that the maximum amount of data in each first data slice is the preset maximum amount of data, thereby avoiding excessive data allocation differences between the first data slices, which may affect subsequent calculations.

[0044] For example, if the data to be updated is "abcdef", when sharding, each first data slice can contain a character in the data to be updated, then there can be six first data slices, with the contents being "a", "b", "c", "d", "e", and "f" respectively.

[0045] In this embodiment, the data to be updated is sharded to obtain multiple first data slices, including: caching the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing; and the data to be updated in the preset distributed computing cluster is sharded to obtain multiple first data slices.

[0046] Specifically, the pre-set distributed computing cluster may be MR (Map Reduce, a distributed big data computing cluster). Users can upload the data to be updated to a pre-set big data processing control platform, retrieve the data to be updated from the big data processing control platform through a pre-set interface, and cache the data to be updated in the pre-set distributed computing cluster. In this embodiment, the type of the pre-set distributed computing cluster is not specifically limited.

[0047] The pre-set distributed computing cluster may be a distributed cluster for data processing. The data to be updated is cached in the distributed computing cluster, allowing the data to be updated to be processed in shards within the pre-set distributed computing cluster. Specifically, the data to be updated in the pre-set distributed computing cluster is sharded to be divided into a plurality of first data slices. Data format conversion rules may also be configured within the pre-set distributed computing cluster for subsequent data format conversion.

[0048] The beneficial effect of this setting is that by processing in a distributed computing cluster, the loss of data to be updated can be avoided, and the distributed computing cluster can improve the efficiency of sharding processing of data to be updated, thereby improving the update efficiency of the database.

[0049] In this embodiment, the data to be updated in the preset distributed computing cluster is sharded to obtain multiple data slices, including: evenly distributing the data to be updated in the preset distributed computing cluster according to a preset number to obtain multiple first data slices.

[0050] Specifically, a number of data slices is pre-set as a preset number. Based on the preset number, the data to be updated in the preset distributed computing cluster is evenly distributed so that the amount of data contained in each first data slice is consistent, thereby generating multiple data slices. That is, each time the database is updated, the number of first data slices generated is fixed, but the amount of data in each first data slice may vary.

[0051] After the updated data is sharded in the preset distributed computing cluster, each first data slice can be cached from the distributed big data computing cluster to the distributed big data storage cluster. The distributed big data storage cluster can be HDFS (Hadoop Distributed File System), so as to avoid data loss after sharding and improve subsequent processing efficiency and accuracy.

[0052] The beneficial effect of such a setting is that by fixing the number of first data slices and evenly distributing the contents of the first data slices, the generation efficiency of the first data slices can be improved, thereby improving the data update efficiency of the database.

[0053] S202: Store the first data slice in a preset database; wherein the preset database stores initial data, which is data before the data to be updated is updated.

[0054] For example, after obtaining the first data slices, each first data slice can be stored in a preset database. Each first data slice can be stored sequentially, or all first data slices can be stored in the preset database in parallel. After all first data slices are stored in the preset database, the data to be updated is also stored in the preset database. For example, each first data slice can be stored in the preset database, or after all first data slices are generated, a response to a data storage instruction can be waited for. If a data storage instruction is received, each first data slice can be stored in the preset database.

[0055] The preset database is empty when storing data in the preset database for the first time. When the preset database is updated subsequently, at least initial data corresponding to data before the update of the to-be-updated data, i.e., data of a previous version of the to-be-updated data, is stored in the preset database. After the to-be-updated data is stored in the preset database, the initial data in the preset database is kept unchanged, i.e., the initial data is retained in the preset database, and at least two versions of data exist in the preset database at this time.

[0056] In S203, if it is determined that each first data piece is stored in the preset database, the data query version is switched from the version of the initial data to the version of the to-be-updated data. The data query version represents a version of data to be acquired when data in the preset database is queried.

[0057] For example, during storage of the first data pieces in the preset database, it is determined in real time whether all the first data pieces are stored in the preset database. If there is a first data piece that is not stored in the preset database, storage and determination are continued. If it is determined that all the first data pieces are stored in the preset database, the data query version can be switched from the version of the initial data to the version of the to-be-updated data. The data query version represents a version of data to be acquired when data in the preset database is queried, i.e., only data of the data query version can be queried when data is queried.

[0058] The data stored in the preset database corresponds to its own version identifier, for example, V1 is marked in the initial data and V2 is marked in the to-be-updated data. After it is determined that each first data piece is stored in the preset database, the data query version is switched from the version of the initial data to the version of the to-be-updated data, i.e., V1 is switched to V2, and only data marked with V2 is queried when data is queried subsequently, thereby realizing switching of the version.

[0059] In this embodiment, the following steps are further included: in response to a data query instruction, the to-be-updated data is acquired from the preset database based on the data query version; and data is queried from the to-be-updated data to obtain a query result.

[0060] Specifically, after the preset database is updated, the latest version of data in the preset database can be queried. A user can issue a data query instruction to the preset database, and in response to the data query instruction, a current data query version is determined. According to the data query version, data is queried from the preset database.

[0061] The data represented by the current data query version is the data to be updated, that is, the data to be updated is obtained from the preset database. The data to be updated is queried to obtain the query results. For example, if the data query instruction indicates that the user A's profile information is to be queried, the user A can be searched in the data to be updated, and then the profile information associated with user A can be found as the query result.

[0062] The beneficial effect of this setting is that after version conversion, query data can be queried from the latest version of data, avoiding the mixing of new and old data and improving the accuracy of data query.

[0063] This embodiment also includes: if it is determined that there is an abnormality in the query result, the data query version is switched from the version of the data to be updated back to the version of the initial data, and a prompt message is issued; wherein the prompt message is used to remind the staff to check the data to be updated.

[0064] Specifically, after receiving the query results, the user can determine whether there are any anomalies in the query results. For example, the user can determine whether the query results are empty, contain garbled characters, or report an error. If the query results are determined to be normal, the data to be updated in the preset database is considered to be correct, and the user can continue to query the latest version of the data to be updated.

[0065] If an anomaly is detected in the query result, the data query version can be switched from the version of the data to be updated back to the version of the initial data, that is, from the latest version back to the previous version. This allows users to perform normal queries from the previous version of the data when performing data queries. A prompt message can also be issued to remind database management staff to check the data to be updated. Once the data to be updated is confirmed to be correct, they can switch back to the latest version.

[0066] The beneficial effect of this setting is that if an error occurs in the latest version of the data during the query process, you can switch back to the old version of the data to achieve rapid data rollback and improve the effect of data query.

[0067] This embodiment further includes: if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, deleting the initial data from the preset database; wherein the new version data represents data after updating the data to be updated.

[0068] Specifically, after the data to be updated is stored in the preset database, the initial data remains in the preset database until a newer version of data is stored in the preset database. In other words, it is possible to determine in real time whether the data query version in the preset database has switched from the version of the data to be updated to the new version of the data. The new version of the data refers to the data that has been updated after the data to be updated, i.e., the data to be updated is the previous version of the new version of the data.

[0069] If it is determined that the data query version of the preset database is still the version of the data to be updated, the query will continue with the data to be updated. If it is determined that the data query version of the preset database has switched from the version of the data to be updated to the new version of the data, the initial data can be deleted from the preset database. At this time, the preset database stores both the data to be updated and the new version of the data. The query can be performed based on the new version of the data. If there is an anomaly in the new version of the data, the query can be switched back to the version of the data to be updated.

[0070] The beneficial effect of this setting is that if the latest version of the data in the preset database is updated, the original old version of the data can be deleted, which can not only save the capacity of the preset database, but also ensure the new version of the preset data query, thereby improving the data query effect.

[0071] In the disclosed embodiment, for a massive amount of data to be updated, the data to be updated can be sliced ​​and processed, and the data to be updated can be divided into multiple first data slices to improve the efficiency of data update in the database. Each first data slice is stored in a preset database, and it is determined whether the storage is complete. Only when all the first data slices are updated to the database will the version switch be performed, so that when querying the data, the query is performed from the latest version of the data. If the data storage is not completed, the query is still performed from the previous version of the data. Batch updates to the database are realized, and the overall atomicity of the data to be updated is realized, avoiding the coexistence and mixing of new and old data during the data import process, and improving the accuracy and efficiency of data queries.

[0072] Figure 3 A flowchart of a data updating method provided in an embodiment of the present disclosure is provided.

[0073] In this embodiment, storing the first data slice in a preset database includes: performing format conversion processing on the first data slice according to a preset data format to obtain target format data; wherein the target format data represents the first data slice in the preset data format; and storing the target format data in the preset database.

[0074] This embodiment is based on the above embodiment. Figure 3 As shown, the method includes the following steps:

[0075] S301 , obtaining data to be updated, and performing sharding processing on the data to be updated to obtain multiple first data slices; wherein the first data slice represents a part of the data to be updated.

[0076] For example, this step may refer to the above-mentioned step S201 and will not be described in detail.

[0077] S302 : Perform format conversion processing on the first data slice according to a preset data format to obtain target format data; wherein the target format data represents the first data slice in the preset data format.

[0078] Exemplarily, a database underlying data format is pre-set. For each first data piece, format conversion processing is performed on the first data piece according to the pre-set data format, converting the first data piece into data in the data format as target format data. Each first data piece corresponds to a target format data, and the data content represented by the target format data is consistent with that of the data piece.

[0079] A data format conversion rule may be set in a preset distributed computing cluster, and the first data slice may be converted into target format data in a preset data format according to the preset data format conversion rule. For example, if the content of the first data slice is the lowercase English letter "a" and the preset data format is uppercase English letters, then the target format data corresponding to the first data slice is "A", that is, "A" is the uppercase format of "a".

[0080] In this embodiment, the format of the first data slice is a key-value pair KV format, the preset data format is a sorted sequence table SST format, and the first data slice includes multiple KV data; according to the preset data format, the first data slice is format converted to obtain target format data, including: sorting the KV data in the data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein, the second data slice represents the sorted first data slice; converting the second data slice into the SST format to obtain target format data.

[0081] Specifically, the data to be updated may be in KV format, i.e., the data slice may be in KV format. Each first data slice includes multiple KV data items, each KV data item includes a key and a corresponding value. The underlying file format in the database is SST (Sorted Sequence Table) format. The preset data format may be SST format, which is a file format for storing sorted KV key-value pairs.

[0082] For each first data slice, during format conversion, the key in each KV data item in the first data slice can be obtained and the KV data in the first data slice can be sorted according to the size of the key. For example, if the key is a number, the KV data in the first data slice can be sorted from small to large according to the numerical value of the key to obtain the second data slice corresponding to the first data slice.

[0083] The second data slice is still in KV format and can be converted to SST format to obtain ordered SST data, that is, to obtain target format data. In other words, the target format data is the ordered SST data corresponding to the first data slice. In this embodiment, the method for converting KV to SST is not specifically limited.

[0084] The beneficial effect of this setting is that the data to be updated in KV format is first preprocessed into the SST format of the underlying file, so that the data in the underlying file format can be directly loaded into the database, reducing the conversion process after the data is stored in the database and improving the update efficiency of data loading.

[0085] S303: Store the target format data in a preset database.

[0086] For example, the target format data corresponding to each data slice is determined and each target format data is stored in a preset database. That is, each data slice is stored in the preset database. Each target format data may be stored in the preset database each time it is generated, or all target format data may be stored in batches in the preset database after all target format data is obtained.

[0087] In this embodiment, format conversion is performed before the data to be updated is stored in the database to obtain data in the underlying file format, thereby avoiding conversion operations on the data to be updated in the database, reducing the waste of database resources, and achieving real-time format conversion processing of the data to be updated during the database filling process, thereby improving the update efficiency of the database.

[0088] In this embodiment, storing the target format data in a preset database includes: caching the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage; according to a preset timed storage task, storing the target format data in the preset distributed storage cluster in the preset database.

[0089] Specifically, a distributed storage cluster is pre-set, and the pre-set distributed storage cluster is a distributed cluster for data storage, and the distributed storage cluster can be a distributed object storage cluster. After obtaining the target format data, the target format data can be temporarily stored in the pre-set distributed storage cluster.

[0090] A scheduled storage task is pre-configured, instructing the storage of target format data from the distributed storage cluster into a preset database. Based on the pre-set scheduled storage task, a storage instruction can be issued periodically. If a storage instruction is received, the target format data can be retrieved from the pre-set distributed storage cluster and stored in the pre-set database. The pre-set database includes storage nodes, which are computer instances. Storage instructions can be issued to the storage nodes, causing them to pull the target format data from the pre-set distributed storage cluster, thereby storing the target format data in the storage nodes, and thus in the pre-set database.

[0091] The beneficial effect of this setting is that the target format data is first stored in the distributed storage cluster and then stored in the preset database, avoiding the loss of target format data during data processing and storage, achieving orderly update of the database, and improving the update efficiency and accuracy of the database.

[0092] In this embodiment, multiple storage nodes are deployed in the preset database, and the storage nodes correspond to the target format data one by one; according to the preset timed storage task, the target format data in the preset distributed storage cluster is stored in the preset database, including: according to the preset timed storage task, obtaining the target format data from the preset distributed storage cluster; storing the target format data in the corresponding storage node in the preset database.

[0093] Specifically, a preset database can be deployed with multiple storage nodes, each corresponding to the target format data. That is, the number of storage nodes is consistent with the number of target format data, that is, the number of storage nodes is consistent with the number of data slices. When performing data sharding operations, the number of data slices can be preset, and the preset number can be the same as the number of storage nodes.

[0094] According to the preset scheduled storage tasks, each target format data can be periodically obtained from the preset distributed storage cluster. Each target format data can be stored in the corresponding storage node and then stored in the preset database. That is, each target format data corresponds to a storage node, and can be assigned to the target format data in sequence according to the preset number of the storage node, or can be assigned to the target format data through pre-configuration. The storage instruction triggered by the scheduled storage task can be sent to each storage node, and each storage node can pull a copy of the target format data from the preset distributed storage cluster. The target format data pulled by each storage node is different, thereby realizing the storage of the target format data.

[0095] The beneficial effect of this setting is that, through the preset storage nodes, the target format data can be stored in the preset database, so that the data to be updated can be stored in fragments, thereby improving the efficiency of data filling.

[0096] S304: If it is determined that all first data pieces are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0097] For example, this step may refer to the above-mentioned step S203 and will not be described in detail.

[0098] Figure 4 Update the logical diagram for the database in this example. Figure 4 In the example, the original file is the data to be updated, and the computing nodes and storage nodes are different computer instances. After the data to be updated is sharded, each computing node corresponds to a first data slice, and different computing nodes can be used to perform format conversion processing on each first data slice, that is, each computing node can convert its corresponding first data slice in KV format into target format data in SST format. The converted target format data is used as a storage file, and each computing node corresponds to a storage file. The number of computing nodes and storage nodes is the same, that is, the number of computing nodes, storage nodes, and data slices is the same, which is n. The storage nodes are located in the preset database, and the corresponding storage files, that is, the target format data, can be pulled through the storage nodes. The corresponding storage files are written into the preset database to complete the update of the preset database.

[0099] In the disclosed embodiment, for a massive amount of data to be updated, the data to be updated can be sliced ​​and processed, and the data to be updated can be divided into multiple first data slices to improve the efficiency of data update in the database. Each first data slice is stored in a preset database, and it is determined whether the storage is complete. Only when all the first data slices are updated to the database will the version switch be performed, so that when querying the data, the query is performed from the latest version of the data. If the data storage is not completed, the query is still performed from the previous version of the data. Batch updates to the database are realized, and the overall atomicity of the data to be updated is realized, avoiding the coexistence and mixing of new and old data during the data import process, and improving the accuracy and efficiency of data queries.

[0100] Figure 5 This is a process architecture diagram of the database update method provided according to an embodiment of the present disclosure. This embodiment is based on the above embodiment, such as Figure 5 As shown, during the database update process:

[0101] Users can upload data to be updated to a pre-set big data processing control platform. The distributed computing cluster retrieves the data and shards it, generating multiple first data slices that are stored in HDFS. The distributed computing cluster can also pre-set data format conversion rules. Based on the pre-set data format conversion rules, the first data slices in KV format are converted to ordered SST format as target format data. Each first data slice corresponds to a target format data piece. Each target format data piece is stored in the distributed storage cluster.

[0102] Through the Open Application Programming Interface (OpenAPI), you can publish scheduled storage tasks to the Database Control Service (DCS). DCS is a control service that can be applied to key-value databases. Scheduled storage tasks can be used to store data in a target format in the database on a scheduled basis.

[0103] A plurality of storage nodes are provided in the database, and the number of the storage nodes is consistent with the number of the first data slices. Figure 5 In the example, the data to be updated is divided into two first data slices, and there are two storage nodes in the database. The DCS can send storage instructions to each storage node at a regular interval based on the scheduled storage task. The storage node receives the storage instruction and pulls the corresponding target format data from the distributed storage cluster. The pulled target format data is the V2 version data, and each storage node also stores the corresponding data of the previous version, that is, the V1 version data. The V2 version data in each storage node can replace the V1 version data. When all storage nodes have completed storing the V2 version data, the data query version can be switched from V1 to V2.

[0104] Users can use the Redis client to send data query commands to the proxy. After receiving the data query command, the proxy can obtain version 2 data from the database storage node. If there is a problem with the query result, the database will be notified to replace the data query version from V2 to V1. If the storage node also stores version 3 data, the V1 data can be deleted.

[0105] The database updating method provided in the embodiments of the present disclosure can solve the database updating problem in various scenarios, especially for the financial joint modeling scenario which needs to store a large amount of KV data, frequently update the full amount and perform real-time query. For this scenario, the stored content in the database can include full amount user features, full amount user standard model scores, customer customized model scores and the like according to the database updating method provided in the embodiments of the present disclosure. The data level can be more than 1 billion rows in a single table, and the maximum can be 2.5 billion rows. The single table data size can be about 1T, and the maximum can be 13T. The Key length is within 64B, and the Value is up to 100K level. There are currently 35 tables, occupying 60T of resources, and it is expected to grow to 100 tables in the future, with resources growing to 200T. The database data updating method and cycle can be full amount coverage updating, and the updating cycle can be daily, weekly or monthly. The single table full amount database updating time efficiency can be up to hours, supporting batch database updating and atomic updating. For query efficiency, the single table average QPS (Query Per Second) is greater than or equal to 1000, the peak value is 3000, the 99th percentile response time is within 10ms, and the service stability index is 99.95%. The embodiments of the present disclosure can also realize fast rollback, that is, when the error data affects the online service, it can be rolled back to the historical version within 1 minute.

[0106] Figure 6 A structural block diagram of a data updating apparatus provided by the embodiments of the present disclosure is shown in FIG. 6. For ease of illustration, only the parts related to the embodiments of the present disclosure are shown. For details, refer to Figure 6 The data updating apparatus 600 includes a sharding unit 601, a storage unit 602 and an updating unit 603.

[0107] The sharding unit 601 is configured to obtain to-be-updated data, perform sharding processing on the to-be-updated data, and obtain a plurality of first data pieces; wherein the first data piece represents a part of the to-be-updated data.

[0108] The storage unit 602 is configured to store the first data pieces into a preset database; wherein the preset database stores initial data, and the initial data is data before the to-be-updated data is updated.

[0109] The updating unit 603 is configured to, if it is determined that each first data piece is stored into the preset database, switch a data query version from a version of the initial data to a version of the to-be-updated data; wherein the data query version represents a version of data to be obtained when data in the preset database is queried.

[0110] Figure 7 A structural block diagram of a data updating apparatus provided by the embodiments of the present disclosure is shown in FIG. 6. For ease of illustration, only the parts related to the embodiments of the present disclosure are shown. For details, refer to Figure 7As shown, the data updating device 700 includes a sharding unit 701 , a storage unit 702 and an updating unit 703 , wherein the storage unit 702 includes a format conversion module 7021 and a data storage module 7022 .

[0111] In one example, the format of the first data slice is a key-value pair (KV) format, the preset data format is a sorted sequence table (SST) format, and the first data slice includes multiple KV data. The format conversion module 7021 includes:

[0112] a data sorting submodule, configured to sort the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein the second data slice represents the sorted first data slice;

[0113] The data conversion submodule is used to convert the second data slice into the SST format to obtain the target format data.

[0114] In one example, the data storage module 7022 includes:

[0115] A data caching submodule, configured to cache the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster for data storage;

[0116] The data storage submodule is used to store the target format data in the preset distributed storage cluster into the preset database according to the preset timed storage task.

[0117] In one example, a plurality of storage nodes are deployed in a preset database, and the storage nodes correspond one-to-one to the target format data; the data storage submodule is specifically configured to:

[0118] According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster;

[0119] The target format data is stored in a corresponding storage node in the preset database.

[0120] In one example, it also includes:

[0121] a data query unit, configured to obtain the to-be-updated data from the preset database based on the data query version in response to a data query instruction;

[0122] The result obtaining unit is used to perform data query from the data to be updated to obtain query results.

[0123] In one example, it also includes:

[0124] The version switching unit is used to switch the data query version from the version of the data to be updated back to the version of the initial data if it is determined that there is an abnormality in the query result, and issue a prompt message; wherein the prompt message is used to remind the staff to check the data to be updated.

[0125] In one example, it also includes:

[0126] The data deleting unit is configured to delete the initial data from the preset database if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data; wherein the new version data represents the data after the data to be updated is updated.

[0127] In one example, the sharding unit 701 includes:

[0128] A cache module, configured to cache the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster for data processing;

[0129] The sharding module is used to perform sharding processing on the data to be updated in the preset distributed computing cluster to obtain multiple first data slices.

[0130] In one example, the sharding module includes:

[0131] The average distribution submodule is used to perform average distribution processing on the data to be updated in the preset distributed computing cluster according to a preset number to obtain the multiple first data slices.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0133] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 8 As shown, the electronic device 800 includes: at least one processor 802; and a memory 801 communicatively connected to the at least one processor 802; wherein the memory stores instructions that can be executed by the at least one processor 802, and the instructions are executed by the at least one processor 802 to enable the at least one processor 802 to execute the data update method disclosed herein.

[0134] The electronic device 800 further includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by other devices, and the transmitter 804 is used to send instructions and data to external devices.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0137] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0138] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0139] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the data updating method. For example, in some embodiments, the data updating method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the data updating method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the data updating method by any other appropriate means, such as by means of firmware.

[0141] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0142] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0147] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0148] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data updating method, comprising: Acquire data to be updated, and perform sharding processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a portion of the data to be updated; Storing the first data slice in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated; If it is determined that all first data pieces are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database; The format of the first data piece is a key-value pair KV format, and the first data piece includes a plurality of KV data. The storing the first data piece in a preset database includes: sorting the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein the second data slice represents the sorted first data slice; Converting the second data piece into a preset data format to obtain target format data, wherein the preset data format is a sorted sequence table SST format; The target format data is stored in the preset database.

2. The method according to claim 1, wherein Storing the target format data in the preset database includes: caching the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage; According to a preset timed storage task, the target format data in the preset distributed storage cluster is stored in the preset database.

3. The method according to claim 2, wherein: The preset database is deployed with a plurality of storage nodes, and the storage nodes correspond one-to-one to the target format data; The storing of the target format data in the preset distributed storage cluster into the preset database according to the preset timed storage task includes: According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster; The target format data is stored in a corresponding storage node in the preset database.

4. The method according to claim 1, further comprising: In response to a data query instruction, obtaining the to-be-updated data from the preset database based on the data query version; Perform a data query on the data to be updated to obtain a query result.

5. The method according to claim 4, further comprising: If it is determined that the query result is abnormal, the data query version is switched from the version of the data to be updated back to the version of the initial data, and a prompt message is issued; wherein, the prompt message is used to remind the staff to check the data to be updated.

6. The method according to any one of claims 1 to 5, further comprising: If it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, the initial data is deleted from the preset database; wherein the new version data represents data after updating the data to be updated.

7. The method according to any one of claims 1 to 5, wherein The slicing of the data to be updated to obtain a plurality of first data slices includes: caching the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing; The data to be updated in the preset distributed computing cluster is sliced ​​to obtain a plurality of first data slices.

8. The method according to claim 7, wherein: The sharding of the data to be updated in the preset distributed computing cluster to obtain a plurality of first data slices includes: According to a preset number, the data to be updated in the preset distributed computing cluster is evenly distributed to obtain the multiple first data slices.

9. A data updating device, comprising: A sharding unit, configured to obtain data to be updated, and shard the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a portion of the data to be updated; A storage unit, configured to store the first data slice in a preset database; wherein the preset database stores initial data, which is data before the data to be updated is updated; an updating unit, configured to switch a data query version from the version of the initial data to the version of the data to be updated if it is determined that all first data slices are stored in the preset database; wherein the data query version represents a version of data to be obtained when performing a data query on the preset database; The format of the first data slice is a key-value pair (KV) format. The first data slice includes multiple pieces of KV data. The storage unit includes: a data sorting submodule, configured to sort the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein the second data slice represents the sorted first data slice; A data conversion submodule, configured to convert the second data slice into a preset data format to obtain target format data, wherein the preset data format is a sorted sequence table SST format; The data storage module is used to store the target format data in the preset database.

10. The device according to claim 9, wherein The data storage module includes: A data caching submodule, configured to cache the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster for data storage; The data storage submodule is used to store the target format data in the preset distributed storage cluster into the preset database according to the preset timed storage task.

11. The device according to claim 10, wherein The preset database is deployed with multiple storage nodes, each of which corresponds to the target format data one by one; the data storage submodule is specifically used to: According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster; The target format data is stored in a corresponding storage node in the preset database.

12. The apparatus according to claim 9, further comprising: A data query unit, configured to obtain the to-be-updated data from the preset database based on the data query version in response to a data query instruction; The result obtaining unit is used to perform data query from the data to be updated to obtain query results.

13. The apparatus according to claim 9, further comprising: The version switching unit is used to switch the data query version from the version of the data to be updated back to the version of the initial data if it is determined that there is an abnormality in the query result, and issue a prompt message; wherein the prompt message is used to remind the staff to check the data to be updated.

14. The apparatus according to any one of claims 9 to 13, further comprising: The data deleting unit is configured to delete the initial data from the preset database if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data; wherein the new version data represents the data after the data to be updated is updated.

15. The device according to any one of claims 9 to 13, wherein: The sharding unit includes: A cache module, configured to cache the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster for data processing; The sharding module is used to perform sharding processing on the data to be updated in the preset distributed computing cluster to obtain multiple first data slices.

16. The device according to claim 15, wherein The sharding module includes: The average distribution submodule is used to perform average distribution processing on the data to be updated in the preset distributed computing cluster according to a preset number to obtain the multiple first data slices.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, wherein The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 8 when the computer program is executed by a processor.

Images