Data updating method and device, equipment and storage medium

Abstract

The invention provides a data updating method and device, equipment and a storage medium, relates to the field of artificial intelligence, in particular to the technical fields of big data, machine learning and the like, and can be applied to the fields of smart finance and the like. According to the specific implementation scheme, to-be-updated data is obtained, and the to-be-updated data is fragmented to obtain a plurality of first data fragments; the first data piece represents a part of the data to be updated; storing the first data piece into a preset database; initial data is stored in the preset database, and the initial data is data before the to-be-updated data is updated; if it is determined that all the first data pieces are stored in the preset database, switching the data query version from the version of the initial data to the version of the to-be-updated data; the data query version represents the version of the data to be acquired when the preset database is subjected to data query. According to the invention, mixed use of new and old versions of data in the database can be avoided, and the data updating efficiency and the data query precision are improved.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as big data and machine learning in the field of artificial intelligence, and can be applied to fields such as smart finance, and in particular to a data updating method, device, equipment and storage medium. Background Art

[0002] The database can store a large amount of data, for example, a large amount of user feature data, which can characterize the user's behavior. The artificial intelligence model can query the feature data from the database and perform model inference based on the feature data.

[0003] However, user behavior changes every day, so the data stored in the database needs to be updated promptly and efficiently to ensure the effectiveness of data queries. Summary of the invention

[0004] The present disclosure provides a data updating method, apparatus, device and storage medium.

[0005] According to a first aspect of the present disclosure, there is provided a data updating method, comprising:

[0006] Acquire data to be updated, and perform slice processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated;

[0007] The first data slice is stored in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated;

[0008] If it is determined that all first data slices are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0009] According to a second aspect of the present disclosure, there is provided a data updating device, comprising:

[0010] A slicing unit, used to obtain data to be updated, and perform slicing processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated;

[0011] A storage unit, used to store the first data slice in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated;

[0012] An updating unit is used to switch the data query version from the version of the initial data to the version of the data to be updated if it is determined that all first data slices are stored in the preset database; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the data updating efficiency in the database is improved and the data query effect is enhanced.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 It is a database update logic diagram in the related technology;

[0023] Figure 2 is a flow chart of a data updating method provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of a data updating method provided according to an embodiment of the present disclosure;

[0025] Figure 4 It is a database update logic diagram provided according to an embodiment of the present disclosure;

[0026] Figure 5 is a process architecture diagram of a database updating method provided according to an embodiment of the present disclosure;

[0027] Figure 6 is a structural block diagram of a data updating device provided according to an embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of a data updating device provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a block diagram of an electronic device for implementing the data updating method of the embodiment of the present disclosure;

[0030] Fig. 9 The block diagram is a block diagram of an electronic device used to implement the data updating method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] In the scenario of model inference, a large amount of user feature data can be stored in the database, so that the model can obtain the feature data from the database and directly perform model inference. Feature data can be obtained by extracting features from user behavior data, but user behavior data may change every day, so the feature data in the database needs to be updated frequently.

[0033] At present, data updates of KV (Key-Value) databases such as Redis (Remote Dictionary Server) and RocksDB (RocksDatabase) are all imported online in real time, which is inefficient. It takes days to update 1 billion data. During the import process, new and old data coexist and are mixed, and the difference data cannot be completely cleared, and the overall atomicity and fast rollback cannot be achieved, which affects the reasoning effect of the model.

[0034] Figure 1 Updates the schematic diagram for the database in the related technology. Figure 1As shown, the original file is the data to be updated, the storage node is a computer instance, and there are n storage nodes. The storage file is located in the database, and each node corresponds to a storage file, that is, there are n storage files. The data to be updated is hashed, and the sharded data can be imported into the corresponding storage file through different nodes. During the import process, the imported data is in the original KV format, that is, the data imported into the database is in KV format, but the underlying file format of the database is mostly not in KV format, so format conversion is also required in the database, resulting in low data update efficiency in the database, mixed storage of new and old data in the database, and inability to accurately query and obtain data in the database in a timely manner.

[0035] The present invention discloses a data updating method, device, equipment and storage medium, which are applied to technical fields such as big data and machine learning in the field of artificial intelligence, and can be applied to fields such as smart finance to improve data updating efficiency and enhance data query effects.

[0036] It should be noted that the data in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0037] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 2-Figure 9 The embodiment is further refined.

[0039] Figure 2 FIG. 1 is a flow chart of a data updating method provided according to an embodiment of the present disclosure, and the method can be executed by a data updating device. Figure 2 As shown, the method comprises the following steps:

[0040] S201, obtaining data to be updated, and performing slice processing on the data to be updated to obtain multiple first data slices; wherein the first data slice represents a part of the data to be updated.

[0041] Exemplarily, a database is pre-set as a preset database, and a large amount of data may be stored in the preset database. The data in the preset database may need to be updated at any time, and the data currently stored in the preset database is used as the initial data, that is, the data to be updated. The data that replaces the initial data is used as the data to be updated, that is, the data after the initial data is updated.

[0042] When the preset database needs to be updated, the data to be updated is obtained, for example, the data to be updated can be managed or uploaded by a staff member. If the data to be updated is received, the update operation of the preset database can be started, that is, the data to be updated needs to be updated into the preset database.

[0043] After obtaining the data to be updated, the data to be updated can be sharded, that is, the data to be updated is divided into multiple data slices as first data slices. Each first data slice can contain different contents in the data to be updated, that is, each first data slice can represent local data to be updated, that is, a part of the data to be updated. In this embodiment, the processing process of the sharding operation is not specifically limited. For example, the maximum amount of data that each first data slice can contain can be preset, and the data to be updated can be divided according to the preset maximum amount of data, so that the maximum amount of data in each first data slice is the preset maximum amount of data, so as to avoid excessive data allocation differences between the first data slices, which affects subsequent calculations.

[0044] For example, if the data to be updated is "abcdef", when sharding, each first data slice may contain a character in the data to be updated, then there may be six first data slices, with contents of "a", "b", "c", "d", "e", and "f" respectively.

[0045] In this embodiment, the data to be updated is sliced ​​to obtain multiple first data slices, including: caching the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing; the data to be updated in the preset distributed computing cluster is sliced ​​to obtain multiple first data slices.

[0046] Specifically, the preset distributed computing cluster may be MR (Map Reduce, distributed big data computing cluster). The user may upload the data to be updated to the preset big data processing control platform, obtain the data to be updated from the big data processing control platform through the preset interface, and cache the data to be updated in the preset distributed computing cluster. In this embodiment, the type of the preset distributed computing cluster is not specifically limited.

[0047] The preset distributed computing cluster may be a distributed cluster for data processing, and the data to be updated is cached in the distributed computing cluster, so that the data to be updated can be processed in slices in the preset distributed computing cluster. That is, the data to be updated in the preset distributed computing cluster is processed in slices, and the data to be updated is divided into a plurality of first data slices. A data format conversion rule may also be set in the preset distributed computing cluster for subsequent data format conversion.

[0048] The beneficial effect of such a setting is that by processing in a distributed computing cluster, the loss of data to be updated can be avoided, and the distributed computing cluster can improve the efficiency of sharding processing of data to be updated, thereby improving the update efficiency of the database.

[0049] In this embodiment, the data to be updated in the preset distributed computing cluster is sliced ​​to obtain multiple data slices, including: according to a preset number, the data to be updated in the preset distributed computing cluster is evenly distributed to obtain multiple first data slices.

[0050] Specifically, the number of data slices is preset as the preset number. According to the preset number, the data to be updated in the preset distributed computing cluster is evenly distributed so that the amount of data contained in each first data slice is consistent, thereby obtaining multiple data slices. That is, each time the database is updated, the number of first data slices generated is fixed, but the amount of data in the first data slice may change.

[0051] After sharding the data to be updated in the preset distributed computing cluster, each first data slice can be cached from the distributed big data computing cluster to the distributed big data storage cluster. The distributed big data storage cluster can be HDFS (Hadoop Distributed File System), so as to avoid data loss after sharding and improve subsequent processing efficiency and accuracy.

[0052] The beneficial effect of such a setting is that by fixing the number of first data slices and evenly distributing the contents of the first data slices, the generation efficiency of the first data slices can be improved, thereby improving the data update efficiency of the database.

[0053] S202, storing the first data slice in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated.

[0054] Exemplarily, after obtaining the first data slices, each first data slice may be stored in a preset database. Each first data slice may be stored sequentially, or all first data slices may be stored in parallel in the preset database. After all first data slices are stored in the preset database, the data to be updated is also stored in the preset database. For example, each time a first data slice is generated, the first data slice may be stored in the preset database, or after generating all first data slices, a response instruction for data storage may be waited, and if an instruction for data storage is received, each first data slice may be stored in the preset database.

[0055] When data is stored in the preset database for the first time, the preset database is empty. When the preset database is subsequently updated, at least initial data is stored in the preset database, and the initial data is the data before the update corresponding to the data to be updated, that is, the data of the previous version of the data to be updated. After the data to be updated is stored in the preset database, the initial data in the preset database is kept unchanged, that is, the initial data is retained in the preset database, and at this time, there are at least two versions of data in the preset database.

[0056] S203: If it is determined that all first data pieces are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0057] Exemplarily, in the process of storing the first data slice in the preset database, it is determined in real time whether all the first data slices are stored in the preset database. If there are first data slices that have not been stored in the preset database, the storage and determination are continued; if it is determined that all the first data slices are stored in the preset database, the data query version can be switched from the version of the initial data to the version of the data to be updated. The data query version represents the version of the data to be obtained when performing a data query on the preset database, that is, when performing a data query, only the data of the data query version can be queried.

[0058] The data stored in the preset database has its own version identifier. For example, the initial data is marked with V1, and the data to be updated is marked with V2. After determining that all first data pieces are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated, that is, V1 is switched to V2. When querying data subsequently, only the data marked with V2 is queried, thereby realizing the version switching.

[0059] In this embodiment, it also includes: in response to the data query instruction, based on the data query version, obtaining the data to be updated from the preset database; and performing data query on the data to be updated to obtain the query result.

[0060] Specifically, after updating the preset database, the latest version of the data in the preset database can be queried. The user can issue a data query instruction to the preset database, respond to the data query instruction, and determine the current data query version. According to the data query version, data is queried from the preset database.

[0061] The data represented by the current data query version is the data to be updated, that is, the data to be updated is obtained from the preset database. The data to be updated is queried to obtain the query result. For example, if the data query instruction indicates that the portrait information of user A is to be queried, user A can be searched from the data to be updated, and then the portrait information associated with user A can be searched as the query result.

[0062] The beneficial effect of such a setting is that after version conversion, query data can be queried from the latest version of data, avoiding mixing of new and old data and improving the accuracy of data query.

[0063] In this embodiment, it also includes: if it is determined that there is an abnormality in the query result, the data query version is switched from the version of the data to be updated back to the version of the initial data, and a prompt message is issued; wherein the prompt message is used to remind the staff to check the data to be updated.

[0064] Specifically, after obtaining the query result, the user can determine whether there is an abnormality in the query result. For example, it can be determined whether the query result is empty, whether garbled characters appear, whether an error is reported, etc. If it is determined that there is no abnormality in the query result, it is considered that there is no problem with the data to be updated in the preset database, and the data query can continue according to the latest version of the data to be updated.

[0065] If it is determined that the query result is abnormal, the data query version can be switched from the version of the data to be updated back to the version of the initial data, that is, from the latest version to the previous version. This allows users to perform normal queries from the previous version of the data when performing data queries. A prompt message can also be issued to remind the staff who manage the database to check the data to be updated. After confirming that there is no problem with the data to be updated, it can be switched back to the latest version.

[0066] The beneficial effect of this setting is that if an error occurs in the latest version of the data during the query process, you can switch back to the old version of the data to achieve rapid data rollback and improve the effect of data query.

[0067] In this embodiment, it also includes: if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, deleting the initial data from the preset database; wherein the new version data represents the data after the data to be updated is updated.

[0068] Specifically, after the data to be updated is stored in the preset database, the initial data is retained in the preset database until an updated version of data is stored in the preset database. That is, it is possible to determine in real time whether the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, where the new version data refers to the data after the data to be updated is updated, that is, the data to be updated is the data of the previous version of the new version data.

[0069] If it is determined that the data query version of the preset database is still the version of the data to be updated, continue to query with the data to be updated; if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, the initial data can be deleted from the preset database. At this time, the preset database stores the data to be updated and the new version data. The query can be based on the new version data. If there is an abnormality in the new version data, it can be switched back to the version of the data to be updated.

[0070] The beneficial effect of such a setting is that if the latest version of the data in the preset database is updated, the original old version of the data can be deleted, which can not only save the capacity of the preset database, but also ensure the new version of the data query of the preset data, thereby improving the data query effect.

[0071] In the disclosed embodiment, for a large amount of data to be updated, the data to be updated can be sliced ​​and processed, and the data to be updated can be divided into multiple first data slices to improve the data update efficiency of the database. Each first data slice is stored in a preset database, and it is determined whether the storage is complete. Only when all the first data slices are updated to the database will the version switch be performed, so that when querying the data, the query is performed from the latest version of the data. If the data storage is not completed, the query is still performed from the previous version of the data. Batch updates to the database are realized, and the overall atomicity of the data to be updated is implemented, avoiding the coexistence and mixing of new and old data during the data import process, and improving the accuracy and efficiency of data queries.

[0072] Figure 3 A flowchart of a data updating method provided in an embodiment of the present disclosure is provided.

[0073] In this embodiment, storing the first data piece in a preset database includes: performing format conversion processing on the first data piece according to a preset data format to obtain target format data; wherein the target format data represents the first data piece of the preset data format; and storing the target format data in the preset database.

[0074] This embodiment is based on the above embodiment. Figure 3 As shown, the method comprises the following steps:

[0075] S301, obtaining data to be updated, and performing slice processing on the data to be updated to obtain multiple first data slices; wherein the first data slice represents a part of the data to be updated.

[0076] Exemplarily, this step may refer to the above-mentioned step S201 and will not be described in detail.

[0077] S302: Perform format conversion processing on the first data slice according to a preset data format to obtain target format data; wherein the target format data represents the first data slice in the preset data format.

[0078] Exemplarily, a data format of the underlying database is preset, and for each first data piece, a format conversion process is performed on the first data piece according to the preset data format, and the first data piece is converted into data of the data format as target format data. Each first data piece corresponds to a target format data, and the data content represented by the target format data is consistent with the data piece.

[0079] A data format conversion rule may be set in the preset distributed computing cluster, and the first data slice may be converted into target format data of the preset data format according to the preset data format conversion rule. For example, if the content of the first data slice is a lowercase English letter "a", and the preset data format is an uppercase English letter, then the target format data corresponding to the first data slice is "A", that is, "A" is an uppercase "a".

[0080] In this embodiment, the format of the first data slice is a key-value pair KV format, the preset data format is a sorted sequence table SST format, and the first data slice includes multiple KV data; according to the preset data format, the first data slice is format converted to obtain target format data, including: according to the keywords of the KV data in the first data slice, the KV data in the data slice is sorted to obtain a second data slice; wherein the second data slice represents the sorted first data slice; the second data slice is converted into the SST format to obtain the target format data.

[0081] Specifically, the data to be updated may be in KV format, that is, the format of the data slice may be in KV format, each first data slice includes multiple KV data, and each KV data includes a Key (keyword) and a corresponding Value (value). The underlying format of the file in the database is SST (Sorted Sequence Table) format, and the preset data format can be determined as SST format. SST format is a file format for storing sorted KV key-value pair data.

[0082] For each first data slice, when performing format conversion, the Key in each KV data in the first data slice can be obtained, and the KV data in the first data slice can be sorted according to the size of the Key. For example, if the Key is a number, the KV data in the first data slice can be sorted from small to large according to the numerical size of the Key to obtain the second data slice corresponding to the first data slice.

[0083] The second data slice is still in KV format, and the second data slice can be converted into SST format to obtain ordered SST data, that is, to obtain target format data. In other words, the target format data is the ordered SST data corresponding to the first data slice. In this embodiment, the method of converting KV to SST is not specifically limited.

[0084] The beneficial effect of such a setting is that the data to be updated in KV format is first preprocessed into the SST format of the underlying file, so that the data in the underlying file format can be directly loaded into the database, reducing the conversion process after the data is stored in the database, and improving the update efficiency of data loading.

[0085] S303: Store the target format data in a preset database.

[0086] Exemplarily, the target format data corresponding to each data piece is determined, and each target format data is stored in a preset database, that is, each data piece is stored in the preset database. Each time a target format data is generated, the target format data is stored in the preset database, or after all the target format data are obtained, all the target format data are stored in batches in the preset database.

[0087] In this embodiment, before the data to be updated is stored in the database, format conversion is performed to obtain data in the underlying file format, thereby avoiding conversion operations on the data to be updated in the database, reducing the waste of database resources, and realizing real-time format conversion processing of the data to be updated during the database filling process, thereby improving the updating efficiency of the database.

[0088] In this embodiment, the target format data is stored in a preset database, including: caching the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage; according to a preset timed storage task, the target format data in the preset distributed storage cluster is stored in the preset database.

[0089] Specifically, a distributed storage cluster is pre-set, and the pre-set distributed storage cluster is a distributed cluster for data storage, and the distributed storage cluster may be a distributed object storage cluster. After obtaining the target format data, the target format data may be temporarily stored in the pre-set distributed storage cluster.

[0090] A scheduled storage task is pre-set, and the scheduled storage task is used to instruct the target format data in the distributed storage cluster to be stored in the preset database. According to the preset scheduled storage task, a storage instruction can be issued regularly. If the storage instruction is responded to, the target format data can be obtained from the preset distributed storage cluster and stored in the preset database. Storage nodes are deployed in the preset database. The storage nodes are computer instances. Storage instructions can be issued to the storage nodes, and the storage nodes pull the target format data from the preset distributed storage cluster, so that the target format data is stored in the storage nodes, that is, stored in the preset database.

[0091] The beneficial effect of this setting is that the target format data is first stored in the distributed storage cluster and then stored in the preset database, avoiding the loss of target format data during data processing and storage, achieving orderly update of the database, and improving the update efficiency and accuracy of the database.

[0092] In this embodiment, multiple storage nodes are deployed in the preset database, and the storage nodes correspond to the target format data one by one; according to the preset scheduled storage task, the target format data in the preset distributed storage cluster is stored in the preset database, including: according to the preset scheduled storage task, the target format data is obtained from the preset distributed storage cluster; the target format data is stored in the corresponding storage node in the preset database.

[0093] Specifically, a plurality of storage nodes may be deployed in the preset database, and the storage nodes correspond to the target format data one by one, that is, the number of storage nodes is consistent with the number of target format data, that is, the number of storage nodes is consistent with the number of data slices. When performing a data slicing operation, the number of data slices may be preset, and the preset number may be the same as the number of storage nodes.

[0094] According to the preset scheduled storage task, each target format data can be obtained from the preset distributed storage cluster on a regular basis, and each target format data can be stored in the corresponding storage node, and thus stored in the preset database. That is, each target format data corresponds to a storage node, which can be assigned to the target format data in sequence according to the preset number of the storage node, or can be assigned to the target format data through pre-configuration. The storage instruction triggered by the scheduled storage task can be sent to each storage node, and each storage node can pull a copy of the target format data from the preset distributed storage cluster. The target format data pulled by each storage node is different, thereby realizing the storage of the target format data.

[0095] The beneficial effect of such a setting is that, through the preset storage nodes, the target format data can be stored in the preset database accordingly, so that the data to be updated can be stored in fragments, thereby improving the efficiency of data filling.

[0096] S304: If it is determined that all first data slices are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0097] Exemplarily, this step may refer to the above-mentioned step S203 and will not be described in detail.

[0098] Figure 4 The logical diagram is updated for the database in this embodiment. Figure 4 In the example, the original file is the data to be updated, and the computing nodes and storage nodes are different computer instances. After the data to be updated is sharded, each computing node corresponds to a first data slice, and different computing nodes can be used to perform format conversion processing on each first data slice, that is, each computing node can convert its corresponding first data slice in KV format into target format data in SST format. The converted target format data is used as a storage file, and each computing node corresponds to a storage file. The number of computing nodes and storage nodes is the same, that is, the number of computing nodes, storage nodes, and data slices is the same, which is n. The storage nodes are located in the preset database, and the corresponding storage files, that is, the target format data, can be pulled through the storage nodes. The corresponding storage files are written into the preset database to complete the update of the preset database.

[0099] In the disclosed embodiment, for a large amount of data to be updated, the data to be updated can be sliced ​​and processed, and the data to be updated can be divided into multiple first data slices to improve the data update efficiency of the database. Each first data slice is stored in a preset database, and it is determined whether the storage is complete. Only when all the first data slices are updated to the database will the version switch be performed, so that when querying the data, the query is performed from the latest version of the data. If the data storage is not completed, the query is still performed from the previous version of the data. Batch updates to the database are realized, and the overall atomicity of the data to be updated is implemented, avoiding the coexistence and mixing of new and old data during the data import process, and improving the accuracy and efficiency of data queries.

[0100] Figure 5 1 is a process architecture diagram of a database update method provided according to an embodiment of the present disclosure. This embodiment is based on the above embodiment. Figure 5 As shown, during the database update process:

[0101] The user can upload the data to be updated to the preset big data processing control platform. The distributed computing cluster obtains the data to be updated, slices the data to be updated, obtains multiple first data slices, and stores them in HDFS. The distributed computing cluster can also preset data format conversion rules. According to the preset data format conversion rules, the first data slice in KV format is converted into an ordered SST format as the target format data. Each first data slice corresponds to a target format data. Each target format data is stored in the distributed storage cluster.

[0102] Through OpenAPI (Open Application Programming Interface), you can publish scheduled storage tasks to DCS (Database Control Service). DCS is a control service that can be applied to KV databases. Scheduled storage tasks can be used to store target format data in the database on a regular basis.

[0103] A plurality of storage nodes are arranged in the database, and the number of the storage nodes is consistent with the number of the first data slices. Figure 5 In the example, the data to be updated is divided into two first data slices, and there are two storage nodes in the database. DCS can send storage instructions to each storage node regularly according to the scheduled storage task. The storage node receives the storage instruction and pulls the corresponding target format data from the distributed storage cluster. The pulled target format data is the V2 version of the data, and each storage node also stores the corresponding data of the previous version, that is, the V1 version of the data. The V2 version of the data in each storage node can replace the V1 version of the data. When all storage nodes have completed the storage of the V2 version of the data, the data query version can be switched from V1 to V2.

[0104] Users can send data query instructions to the proxy through the Redis Client. After receiving the data query instructions, the proxy can obtain the V2 version of the data from the database storage node. If there is a problem with the query result, the database will be notified to replace the data query version from V2 to V1. If the storage node also stores V3 version of the data, the V1 version of the data can be deleted.

[0105] The disclosed embodiments can solve the database update problem in various scenarios, especially for the financial joint modeling scenario, which requires the storage, frequent full update and real-time query of a large amount of KV data. For this scenario, through the database update method of the disclosed embodiment, the stored content in the database may include full user features, full user standard model points, customer customized model points, etc. The data volume can be more than 1 billion rows in a single table, with a maximum of 2.5 billion rows. The data size of a single table can be around 1T, with a maximum of 13T. The Key length is within 64B, and the maximum Value is 100K. There are currently 35 tables in stock, occupying 60T of resources, and it is expected to grow to 100 tables in the future, and the resources will grow to 200T. The data update method and cycle in the database can be a full coverage update, with update cycles at the daily, weekly, and monthly levels. The maximum acceptable time for a single table full database update is at the hour level, and batch database updates and atomic updates are supported. For query timeliness, the average QPS (Query Per Second) of a single table is greater than or equal to 1000, the peak is 3000, the 99th percentile response time is within 10ms, and the service stability index is 99.95%. The disclosed embodiment can also achieve fast rollback, that is, when the wrong data is injected and affects the online service, it can be rolled back to the historical version within 1 minute.

[0106] Figure 6 This is a structural block diagram of a data updating device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6 The data updating device 600 includes: a sharding unit 601, a storage unit 602 and an updating unit 603.

[0107] The slicing unit 601 is used to obtain data to be updated, and to slice the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated;

[0108] The storage unit 602 is used to store the first data slice in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated;

[0109] The updating unit 603 is used to switch the data query version from the version of the initial data to the version of the data to be updated if it is determined that all first data slices are stored in the preset database; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

[0110] Figure 7 A structural block diagram of a data updating device provided in an embodiment of the present disclosure, such as Figure 7As shown, the data updating device 700 includes a segmentation unit 701 , a storage unit 702 and an updating unit 703 , wherein the storage unit 702 includes a format conversion module 7021 and a data storage module 7022 .

[0111] In one example, the format of the first data piece is a key-value pair KV format, the preset data format is a sorted sequence table SST format, and the first data piece includes multiple KV data; the format conversion module 7021 includes:

[0112] A data sorting submodule, configured to sort the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein the second data slice represents the sorted first data slice;

[0113] The data conversion submodule is used to convert the second data slice into the SST format to obtain the target format data.

[0114] In one example, the data storage module 7022 includes:

[0115] A data cache submodule, used to cache the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage;

[0116] The data storage submodule is used to store the target format data in the preset distributed storage cluster into the preset database according to the preset timed storage task.

[0117] In one example, a plurality of storage nodes are deployed in a preset database, and the storage nodes correspond to the target format data one by one; the data storage submodule is specifically used for:

[0118] According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster;

[0119] The target format data is stored in a corresponding storage node in the preset database.

[0120] In one example, it also includes:

[0121] A data query unit, configured to obtain the to-be-updated data from the preset database based on the data query version in response to a data query instruction;

[0122] The result obtaining unit is used to perform data query from the data to be updated to obtain the query result.

[0123] In one example, it also includes:

[0124] A version switching unit is used to switch the data query version from the version of the data to be updated back to the version of the initial data if it is determined that the query result is abnormal, and issue a prompt message; wherein the prompt message is used to remind the staff to check the data to be updated.

[0125] In one example, it also includes:

[0126] The data deleting unit is used to delete the initial data from the preset database if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data; wherein the new version data represents the data after the data to be updated is updated.

[0127] In one example, the slicing unit 701 includes:

[0128] A cache module, used to cache the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing;

[0129] The sharding module is used to perform sharding processing on the data to be updated in the preset distributed computing cluster to obtain multiple first data slices.

[0130] In one example, the sharding module includes:

[0131] The average distribution submodule is used to perform average distribution processing on the to-be-updated data in the preset distributed computing cluster according to a preset number to obtain the multiple first data slices.

[0132] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0133] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 8 As shown, the electronic device 800 includes: at least one processor 802; and a memory 801 communicatively connected to the at least one processor 802; wherein the memory stores instructions executable by the at least one processor 802, and the instructions are executed by the at least one processor 802 so that the at least one processor 802 can execute the data update method disclosed herein.

[0134] The electronic device 800 further includes a receiver 803 and a transmitter 804. The receiver 803 is used to receive instructions and data sent by other devices, and the transmitter 804 is used to send instructions and data to external devices.

[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0136] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0137] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0138] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0139] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a data update method. For example, in some embodiments, the data update method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the data update method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the data update method in any other appropriate manner (e.g., by means of firmware).

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data updating method, comprising: Acquire data to be updated, and perform slice processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated; The first data slice is stored in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated; If it is determined that all first data slices are stored in the preset database, the data query version is switched from the version of the initial data to the version of the data to be updated; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

2. The method according to claim 1, wherein: The storing the first data slice into a preset database includes: According to a preset data format, format conversion processing is performed on the first data slice to obtain target format data; wherein the target format data represents the first data slice of the preset data format; The target format data is stored in the preset database.

3. The method according to claim 2, wherein: The format of the first data piece is a key-value pair KV format, the preset data format is a sorted sequence table SST format, and the first data piece includes a plurality of KV data; The step of performing format conversion processing on the first data slice according to a preset data format to obtain target format data includes: Sorting the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; wherein the second data slice represents the sorted first data slice; Convert the second data slice into SST format to obtain the target format data.

4. The method according to claim 2 or 3, wherein: The storing the target format data into the preset database comprises: Cache the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage; According to a preset timed storage task, the target format data in the preset distributed storage cluster is stored in the preset database.

5. The method according to claim 4, wherein: A plurality of storage nodes are deployed in the preset database, and the storage nodes correspond to the target format data one by one; According to the preset timed storage task, storing the target format data in the preset distributed storage cluster into the preset database includes: According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster; The target format data is stored in a corresponding storage node in the preset database.

6. The method according to any one of claims 1 to 5, further comprising: In response to the data query instruction, based on the data query version, acquiring the data to be updated from the preset database; A data query is performed from the data to be updated to obtain a query result.

7. The method according to claim 6, further comprising: If it is determined that the query result is abnormal, the data query version is switched from the version of the data to be updated back to the version of the initial data, and a prompt message is issued; wherein the prompt message is used to remind the staff to check the data to be updated.

8. The method according to any one of claims 1 to 7, further comprising: If it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data, the initial data is deleted from the preset database; wherein the new version data represents the data after the data to be updated is updated.

9. The method according to any one of claims 1 to 8, wherein: The step of performing slicing processing on the data to be updated to obtain a plurality of first data slices includes: Cache the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing; The data to be updated in the preset distributed computing cluster is sliced ​​to obtain a plurality of first data slices.

10. The method according to claim 9, wherein: The slicing of the data to be updated in the preset distributed computing cluster to obtain a plurality of first data slices includes: According to a preset number, the data to be updated in the preset distributed computing cluster is evenly distributed to obtain the multiple first data slices.

11. A data updating device, comprising: A slicing unit, used to obtain data to be updated, and perform slicing processing on the data to be updated to obtain a plurality of first data slices; wherein the first data slice represents a part of the data to be updated; A storage unit, used to store the first data slice in a preset database; wherein the preset database stores initial data, and the initial data is data before the data to be updated is updated; An updating unit is used to switch the data query version from the version of the initial data to the version of the data to be updated if it is determined that all first data slices are stored in the preset database; wherein the data query version represents the version of the data to be obtained when performing a data query on the preset database.

12. The device according to claim 11, wherein The storage unit comprises: A format conversion module, configured to perform format conversion processing on the first data slice according to a preset data format to obtain target format data; wherein the target format data represents the first data slice of the preset data format; The data storage module is used to store the target format data in the preset database.

13. The device according to claim 12, wherein: The format of the data slice is a key-value pair KV format, the preset data format is a sorted sequence table SST format, and the data slice includes a plurality of KV data; The format conversion module comprises: a data sorting submodule, configured to sort the KV data in the first data slice according to the keywords of the KV data in the first data slice to obtain a second data slice; The data conversion submodule is used to convert the second data slice into the SST format to obtain the target format data.

14. The device according to claim 12 or 13, wherein: The data storage module comprises: A data cache submodule, used to cache the target format data in a preset distributed storage cluster; wherein the preset distributed storage cluster is a distributed cluster used for data storage; The data storage submodule is used to store the target format data in the preset distributed storage cluster into the preset database according to the preset timed storage task.

15. The device according to claim 14, wherein: The preset database is deployed with a plurality of storage nodes, and the storage nodes correspond to the target format data one by one; the data storage submodule is specifically used for: According to the preset scheduled storage task, obtaining target format data from the preset distributed storage cluster; The target format data is stored in a corresponding storage node in the preset database.

16. The apparatus according to any one of claims 11 to 15, further comprising: A data query unit, configured to obtain the to-be-updated data from the preset database based on the data query version in response to a data query instruction; The result obtaining unit is used to perform data query from the data to be updated to obtain the query result.

17. The method according to claim 6, further comprising: A version switching unit is used to switch the data query version from the version of the data to be updated back to the version of the initial data and issue a prompt message if it is determined that the query result is abnormal; wherein the prompt message is used to remind the staff to check the data to be updated.

18. The apparatus according to any one of claims 11 to 17, further comprising: The data deleting unit is used to delete the initial data from the preset database if it is determined that the data query version of the preset database is switched from the version of the data to be updated to the version of the new version data; wherein the new version data represents the data after the data to be updated is updated.

19. The device according to any one of claims 11 to 18, wherein: The slicing unit comprises: A cache module, used to cache the data to be updated in a preset distributed computing cluster; wherein the preset distributed computing cluster is a distributed cluster used for data processing; The sharding module is used to perform sharding processing on the data to be updated in the preset distributed computing cluster to obtain multiple first data slices.

20. The device according to claim 19, wherein The sharding module comprises: The average distribution submodule is used to perform average distribution processing on the to-be-updated data in the preset distributed computing cluster according to a preset number to obtain the multiple first data slices.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product, wherein: The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 10 when being executed by a processor.

Images