Data management system, data storage method and data query method

By introducing in-store service modules and distributed proxy modules into the data management system, the unified management of multiple KV storage clusters is solved, and the problem of frequent modification of online application codes in the existing technology is improved, and the management efficiency of big data KV storage clusters is improved and maintainability and scalability of business systems.

CN118964480APending Publication Date: 2024-11-15DUXIAOMAN TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410967568.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing big data KV storage cluster needs to adaptively modify the online application code when it is changed, resulting in poor scalability and maintainability of the business system.

Method used

By introducing in-store service modules and distributed proxy modules into the data management system, multiple KV storage clusters are managed uniformly, and unified calls and management of KV storage clusters of different structures are realized without modifying online application codes, reducing data in-store and The coupling of query improves the maintainability and scalability of the system.

Benefits of technology

It realizes efficient management and data processing of KV storage clusters, reduces the coupling between online applications and data storage, improves the maintainability and scalability of business systems, and avoids modifying online when adding or decreasing KV storage clusters. Requirements for application code.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964480A_ABST
    Figure CN118964480A_ABST
Patent Text Reader

Abstract

The invention provides a data management system, a data storage method and a data query method, and the system comprises a storage service module which receives a data storage request through a first data interface, calls a calculation cluster to process to-be-stored data and stores the to-be-stored data in a target table of a target cluster, metadata of the to-be-stored data is sent to the distributed agent module; the distributed agent module is used for routing a data query request to a storage cluster for processing based on pre-stored metadata of to-be-queried data, and the storage module comprises multiple KV storage clusters. The distributed agent module is arranged between the application and the data table to manage the metadata, only the metadata needs to be sent to the distributed agent module after data storage, code logic of the online application does not need to be modified, coupling of the online application and data storage is reduced, when a KV storage cluster needs to be changed, online application codes do not need to be modified, and data storage efficiency is improved. And the maintainability and the expandability of the service system are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data cluster technology, and in particular to a data management system, a data warehousing method and a data query method. Background Art

[0002] With the rapid development of distributed technology, big data clusters have been widely used in various industries. Big data clusters used by enterprises usually include big data KV storage clusters, which are systems used to process and store large-scale key-value data. Among them, the key and value can be any data type, such as text, numbers, images, etc.

[0003] Currently, after changing the big data KV storage cluster, it is usually necessary to adaptively modify the code of the online application, resulting in poor scalability and maintainability of the business system. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data management system, a data storage method and a data query method to improve the scalability and maintainability of the business system.

[0005] According to one aspect of the present invention, there is provided a data management system, the system comprising:

[0006] The warehousing service module is used to receive a data warehousing request through a first data interface, wherein the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, and the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored; call a computing cluster to process the data to be warehousing to obtain the data to be stored; store the data to be stored in the target table of the target cluster; and send the metadata of the data to be stored to the distributed proxy module;

[0007] The distributed proxy module is used to receive a data query request through a second data interface, wherein the data query request includes the data to be queried; based on the pre-stored metadata of the data to be queried, route the data query request to a storage cluster for processing, obtain a query result returned by the storage cluster, wherein the metadata includes the storage cluster of the data to be queried; and return the query result to the second data interface;

[0008] The storage module includes multiple KV storage clusters and is used to store the data to be stored and the data to be queried.

[0009] In a possible embodiment, the logic table information further includes: a merge quantity, where the merge quantity is the quantity of data of the same type to be merged; the computing cluster processes the data to be stored in the database through the following steps:

[0010] Determine the type of data to be stored, the type of data to be stored includes real-time data and batch data;

[0011] Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on the number of merged versions preset for the batch data, and converting the merged data into data in a preset format.

[0012] In a possible embodiment, a whitelist is pre-set for each cluster and each table, the whitelist includes user identifiers with storage and / or query permissions, the data storage request includes the storage user identifier, and the data query request includes the query user identifier;

[0013] The storage service module is further used to perform permission verification on the storage user ID based on the target cluster and the white list of the target table; if the storage user ID passes the permission verification, the computing cluster is called to process the data to be stored to obtain the data to be stored;

[0014] The distributed proxy module is also used to verify the authority of the query user identifier based on the storage cluster and the storage table whitelist in the storage cluster; if the query user identifier passes the authority verification, the data query request is routed to the storage cluster for processing to obtain the query result returned by the storage cluster, wherein the storage table in the storage cluster is used to store the data to be queried.

[0015] In a possible embodiment, the logical table information also includes the number of version tables that the business needs to save, and different version tables correspond to data generated at different times; the warehousing service module is used to disable the current version table when receiving data exception information, and use the previous version table as the new current version table. The new current version table is used to provide a basis for data query, and the previous version table is a table whose data generation time is before the current version table.

[0016] According to another aspect of the present invention, a data warehousing method is provided, which is applied to a warehousing service module in a data management system, wherein the data management system also includes a distributed agent module; the method comprises:

[0017] Receive a data entry request through a first data interface, wherein the data entry request includes the data to be entered and the logical table information of the business to which the data to be entered belongs, and the logical table information includes the target cluster and target table information where the data to be entered needs to be stored, and the first data interface is used to receive the data entry request;

[0018] Calling a computing cluster to process the data to be stored to obtain data to be stored;

[0019] Storing the data to be stored in the target table of the target cluster;

[0020] The metadata of the data to be stored is sent to the distributed proxy module, so that the distributed proxy module stores the metadata of the data to be stored and processes the data query request based on the metadata.

[0021] In a possible embodiment, the logic table information further includes: a merge quantity, where the merge quantity is the quantity of data of the same type to be merged; the computing cluster processes the data to be stored in the database through the following steps:

[0022] Determine the type of data to be stored, the type of data to be stored includes real-time data and batch data;

[0023] Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on the number of merged versions preset for the batch data, and converting the merged data into data in a preset format.

[0024] In a possible embodiment, a whitelist is pre-set for each cluster and each table, the whitelist includes a user identifier with storage query authority, and the data storage request includes the storage user identifier; the method further includes:

[0025] The authority verification is performed on the storage user ID based on the target cluster and the white list of the target table; if the storage user ID passes the authority verification, the computing cluster is called to process the data to be stored to obtain the data to be stored.

[0026] According to another aspect of the present invention, a data query method is provided, which is applied to a distributed agent module of a data management system, wherein the data management system also includes a warehousing service module; the method comprises:

[0027] Receiving a data query request through a second data interface, wherein the data query request includes data to be queried;

[0028] Based on the pre-stored metadata of the data to be queried, routing the data query request to the storage cluster for processing, and obtaining the query result returned by the storage cluster;

[0029] The metadata includes the storage cluster of the data to be queried; the metadata is: the warehousing service module receives a data warehousing request through a first data interface, the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored, the first data interface is used to receive the data warehousing request; call the computing cluster to process the data to be warehousing to obtain the data to be stored; store the data to be stored in the target table of the target cluster and then send it to the data of the distributed proxy module;

[0030] The query result is returned to the second data interface.

[0031] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0032] Processor; and

[0033] Memory for storing programs,

[0034] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned data storage methods or data query methods.

[0035] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned data warehousing methods or data query methods.

[0036] One or more technical solutions provided in the embodiments of the present invention,

[0037] By unifying each KV storage cluster as the underlying storage layer, the warehousing service module receives data warehousing requests through the first data interface, and executes warehousing operations on the corresponding KV storage cluster based on the data warehousing requests; the distributed agent module receives data query requests through the second data interface, and calls the corresponding KV storage cluster for data query based on the data query request, thereby achieving unified calling and unified management of KV storage clusters with different structures. There is no need to use different engines or different methods to manage different KV storage clusters, thereby improving the management efficiency of the KV storage cluster and the data warehousing and query efficiency based on the KV storage cluster, that is, improving the cluster data processing efficiency.

[0038] Furthermore, by setting up a distributed proxy module between the application and the database, and having the distributed proxy module manage the metadata of the data and route the data requests of the online application, after the data is entered into the warehouse, only the metadata of the entered data needs to be sent to the distributed proxy module, without the need to modify the code logic of the online application. This reduces the coupling between the online application and the data entry, and improves the independence of the online application and the data entry. When it is necessary to increase or decrease the KV storage cluster, there is no need to modify the online application code, thereby improving the maintainability and scalability of the business system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 A schematic diagram of the structure of a data management system provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the architecture of a data management system provided by an embodiment of the present invention;

[0042] Figure 3 A flow chart of data storage and data query in a data management system provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a processing flow of a data management system provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of a flow chart of a data storage method provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of a flow chart of a data query method provided by an embodiment of the present invention;

[0046] Figure 7 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0048] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0049] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0050] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0051] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0052] In order to improve the efficiency of data processing, the embodiments of the present invention provide a data storage method, a data query method and a data management system. The data management system provided by the embodiments of the present invention is first exemplified below. The data management system provided by the embodiments of the present invention can be applied to a big data cluster. The scheme of the present invention is described below with reference to the accompanying drawings:

[0053] like Figure 1 As shown, Figure 1 A schematic diagram of a data management system provided in an embodiment of the present invention is provided. The data management system 100 may include:

[0054] The warehousing service module 101 is used to receive a data warehousing request through a first data interface, wherein the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, and the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored; call a computing cluster to process the data to be warehousing to obtain the data to be stored; store the data to be stored in the target table of the target cluster; and send the metadata of the data to be stored to the distributed proxy module;

[0055] The distributed proxy module 102 is used to receive a data query request through a second data interface, wherein the data query request includes the data to be queried; based on the pre-stored metadata of the data to be queried, route the data query request to a storage cluster for processing, obtain a query result returned by the storage cluster, wherein the metadata includes the storage cluster of the data to be queried; and return the query result to the second data interface;

[0056] The storage module 103 includes multiple KV storage clusters for storing the data to be stored and the data to be queried.

[0057] By applying the embodiments of the present invention, various KV storage clusters are unified as the underlying storage layer, and the warehousing service module receives data warehousing requests through the first data interface, and executes warehousing operations on the corresponding KV storage cluster based on the data warehousing requests; the distributed proxy module receives data query requests through the second data interface, and calls the corresponding KV storage cluster for data query based on the data query request, thereby realizing unified calling and unified management of KV storage clusters with different structures, without the need to use different engines or different methods to manage different KV storage clusters, thereby improving the management efficiency of the KV storage cluster and the data warehousing and query efficiency based on the KV storage cluster, that is, improving the cluster data processing efficiency.

[0058] Furthermore, by setting up a distributed proxy module between the application and the data table, and having the distributed proxy module manage the metadata of the data and route the data requests of the online application, after the data is entered into the warehouse, only the metadata of the entered data needs to be sent to the distributed proxy module without modifying the code logic of the online application. This reduces the coupling between the online application and the data entry and improves the independence of the online application and the data entry. When it is necessary to increase or decrease the KV storage cluster, there is no need to modify the online application code, thereby improving the maintainability and scalability of the business system.

[0059] The following is an exemplary description of the data management system provided by the embodiment of the present invention:

[0060] In the embodiment of the present invention, different interfaces can be set for the warehousing service and the query service of the online application. As a possible implementation method, a first data interface is set for the warehousing service, such as the data warehousing service required by the feature management platform, other task scheduling platforms, and operations such as version rollback, and a second data interface is set for the online query service, such as model online scoring or other online query services. The above data interface can be any type of API (Application Programming Interface, application programming interface), which can be written in programming languages ​​such as C language and Java.

[0061] Platforms that have data storage requirements, such as feature management platforms, can send data storage requests through the above-mentioned first data interface, and the first data interface can send the data storage request to the storage service module, which is used to store the data. The above-mentioned data storage request may include the data to be stored and the logical table information of the business described by the data to be stored. There may be multiple data to be stored, and the above-mentioned logical table information may include the data type of the data generated by the business, the storage cluster identifier of the data corresponding to each data type, the identifier of the table storing the business data, and the number of hot backups of the table, etc.

[0062] Among them, the storage cluster identifier can be the name of the storage cluster, and the table identifier can be the name of the table or a preset code, etc. In a possible embodiment, the business data generated by the same business may be stored in different subtables of the same table, and each subtable is stored in a different cluster. Therefore, the identifier of the above table can be the subtable identifier for storing business data. Exemplarily, for business data of the same business with different usage rates, a subtable of a SATA type KV storage cluster can be set to store data with a usage frequency of less than 500 within 24 hours, and data with a usage frequency of more than 500 within 24 hours can be stored in a subtable of an SSD type KV storage cluster. Data types with a usage frequency of more than 500 and less than 500 can be pre-configured according to actual business scenarios. Accordingly, the business logic table can include two storage cluster identifiers and subtable identifiers.

[0063] In a possible embodiment, a table can be established in the required cluster before configuring the logical table. Specifically, cluster information with a specific name, type and person in charge can be created through the operation and maintenance API (OP-API) to provide an infrastructure for subsequent data storage and access. The above name can be set according to the actual business scenario, and the cluster type can include the type of the KV storage cluster itself and the processing engine of the KV storage cluster, wherein the KV storage cluster type can include SSD (Solid State Disk, solid state hard disk) cluster and SATA (Serial Advanced Technology Attachment, serial hard disk) cluster, etc., and the above processing engine can include RocksDB engine and Rainbow engine, etc., wherein the RocksDB engine is widely used in the storage of real-time features to achieve high concurrency, low latency data storage and processing, and the Rainbow engine is specifically used for the storage of batch features, thereby providing a variety of data processing methods for the business, and the type and processing engine of each cluster can be flexibly set according to actual business needs, and the present invention does not make specific limitations on this.

[0064] The above-mentioned number of hot backups refers to the number of replicas of the table stored in the master-slave cluster. The master-slave cluster is the default cluster for processing data requests. The above-mentioned replicas can be stored in the master-slave cluster or in other KV storage clusters. The number of replicas can be configured according to actual business needs to replace the default table in the master-slave cluster when an exception occurs to process data requests.

[0065] For multi-version tables, the above logical table may also include the number of versions and the number of merges of the multi-version table. Tables in a KV storage cluster usually store data in KV (key-value) format, where the key is usually a data type, which can be pre-set according to the actual application scenario, and the value may include data information corresponding to the data type, such as data value, data production time, etc. The above version is the production time of the data corresponding to the same key value, and the number of merges refers to the number of data of the same type that needs to be merged. The merge here can refer to deleting data that exceeds the merge number in the order of generation time.

[0066] In a possible embodiment, the same table is stored in different KV storage clusters in the form of multiple sub-tables, so the key may also include a sub-table identifier.

[0067] When the warehousing service module receives a data warehousing request through the first data interface, it can process the data to be stored in the data warehousing request. Specifically, it can call a computing cluster to process the data to be stored. The computing cluster can be any cluster with computing functions, such as Kafka, Flink, etc., and the present invention does not specifically limit this.

[0068] In a possible embodiment, the above-mentioned computing cluster can perform calculations through the Eggroll framework, and accordingly, the warehousing service module can call the Eggroll entity table construction API to request the computing cluster to process the data warehousing request. In Eggroll, building an entity table API usually refers to creating or registering a table in Eggroll's graph computing framework for subsequent computing operations. Since Eggroll is a data processing framework based on graph computing, it does not directly provide an API for creating a table. Therefore, as a possible implementation method, the creation or registration of a table can be simulated by calling the graph computing-related API provided by Eggroll.

[0069] In a possible embodiment, the computing cluster may perform storage processing on the data to be stored by the following process:

[0070] S11, determining the type of data to be stored, where the type of data to be stored includes real-time data and batch data;

[0071] S12. Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain key-value data of the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on a preset number of merged versions for the batch data, and converting the merged data into data in a preset format.

[0072] The batch processing flow and the streaming processing flow in this step can be pre-set according to the actual business scenario. In a possible embodiment, the streaming processing and batch processing flow can both include extracting key value information of the data to be stored, wherein the key information can be the data type of the data to be stored and the sub-table identifier corresponding to the data to be stored, and the value information can include the value of the data to be stored, the production time of the data to be stored, etc. As a possible embodiment, the batch processing flow can also include merging multiple versions of the data to be stored according to the merge quantity.

[0073] The preset format can be pre-set according to the actual business scenario, such as binary, hexadecimal, etc., and the KV data format is unified by converting the KV data into the preset format, thereby improving the efficiency of subsequent data queries.

[0074] In a possible embodiment, a whitelist may be pre-configured for each table and each KV storage cluster, and the whitelist includes user identifiers that can store data in the table or cluster. Accordingly, the above-mentioned data entry request may also include the user identifier that sends the data entry request. The entry service request may determine the location where the data to be entered needs to be stored based on the logical table information included in the data entry request, including the table and cluster information that need to be stored, and perform permission verification on the user identifier based on the whitelist of the table and cluster. If the user identifier is found in the whitelist, it means that the user identifier has passed the permission verification, and the processing flow of the data entry request may be executed. If the user identifier is not found in the whitelist, it means that the user identifier has not passed the permission verification, so an error message may be returned or the request may be discarded.

[0075] In a possible embodiment, the warehousing service module can also provide a rollback function. Specifically, the above-mentioned logical table information also includes the number of version tables that need to be saved by the business, and different version tables correspond to data generated at different times; the warehousing service module is used to disable the current version table when receiving data exception information, and use the previous version table as the new current version table. The new current version table is used to provide a basis for data query, and the previous version table is a table whose data generation time is before the current version table.

[0076] When an anomaly is found in the current data or a certain version of data, the function of rolling back to the hot standby version in seconds, rolling back to the cold standby version in minutes or disabling the version can be provided. Users can quickly recover and handle data anomalies by calling the corresponding API capabilities. The hot standby version refers to the version table stored in the hot standby server, and the cold standby version refers to the version table stored in the cold standby server.

[0077] In a possible embodiment, the data management system may scan each version table according to a preset time period, and when an invalid table appears, the invalid table may be deleted. The invalidation may be that the data has not been used for a long time or the data production time exceeds the preset time.

[0078] In a possible embodiment, the above-mentioned warehousing service module may include multiple APIs, such as comprehensive APIs including table creation, warehousing, rollback, re-warehousing, and interaction with query services. By calling APIs with different functions, the warehousing service module can be driven to implement corresponding functions.

[0079] After the warehousing service module completes the warehousing operation, the metadata of the above-mentioned data to be stored can be sent to the distributed proxy module. Metadata is the description information of the data, which can include the query path, storage location and other information of the data. The distributed proxy module stores the metadata of the data in each KV storage cluster. As a possible implementation method, the distributed proxy module can store the data identifier and the metadata of the data in correspondence. When a data query request is received, the storage location of the data to be queried can be determined based on the metadata of the data to be queried included in the data query request, and the data query request can be routed to the KV storage cluster corresponding to the storage location.

[0080] In a possible embodiment, the distributed agent module may receive a data query request through the above-mentioned second data interface. The data query request is sent by an online application. The data query request may include an identifier of the data to be queried. The identifier of the data to be queried may be a data type and a data production time, etc.

[0081] The distributed proxy module can determine the metadata of the data to be queried based on the identifier of the data to be queried included in the data query request, and then determine the KV storage cluster where the data to be queried is located. After that, the distributed proxy module can send the data query request to the corresponding KV storage cluster for processing, and obtain the query result and return it through the second data interface.

[0082] In a possible embodiment, the above-mentioned pre-configured whitelist for each table and each KV storage cluster may also include query permissions for data in the corresponding KV storage cluster or table. Accordingly, the above-mentioned data query request may include a user identifier for sending the data query request. The distributed proxy module may determine the KV storage cluster and table that needs to be scheduled based on the data to be queried, and query the query user identifier in the whitelist of the corresponding KV storage cluster or table. If the query is found, it means that the query user has passed the permission verification and the data query request can be processed. Otherwise, an error message is returned.

[0083] In a possible embodiment, the distributed proxy module can use the Netty framework to process data query requests. Netty is a Java open source framework that provides asynchronous, event-driven network application frameworks and tools for quickly developing high-performance, high-reliability network servers and client programs. By using the Netty framework to process data query requests, efficient and stable query capabilities are provided, so that the data management system can process a large number of data requests at the same time, improve the speed and efficiency of data processing, reduce the waiting time of requests, and improve the throughput of the system. And by implementing the proxy layer in front of the distributed proxy module, the pressure on the server can be reduced and the query efficiency can be improved. The proxy layer can be linked with the warehousing service to switch the query mode and realize lossless table changes. At the same time, the distributed proxy introduces the concept of logical tables. The upstream service only needs to provide the logical table name to realize easy query, which reduces the complexity of the query and improves the query efficiency.

[0084] like Figure 2 As shown, Figure 2A schematic diagram of the architecture of a data management system provided by an embodiment of the present invention, the data management system can be divided into an upper application layer, an intermediate functional layer and an underlying storage layer, the underlying storage layer can include various KV storage clusters, such as SSD type Rainbow engine KV storage cluster, SATA type Rainbow engine KV storage cluster, RocksDB engine real-time KV storage cluster, the intermediate functional layer includes a storage service module and a distributed agent module, and the upper application layer can include a feature management platform or other submission entry for tasks with data storage requirements and online applications such as online scoring. Specifically, the feature management platform, etc. can send a data storage request to the storage service module through a first data interface, and the storage service module calls a computing cluster to process the data to be stored, including batch processing or streaming processing of the data to be stored and conversion of data formats, etc., and then the data to be stored in a unified format can be stored in the corresponding KV storage cluster. The distributed agent module can receive a data query request sent by an online application such as model online scoring through a second data interface, and route the data query request to the corresponding KV storage cluster based on the stored metadata, and the corresponding KV storage cluster processes the data query request.

[0085] like Figure 3 As shown, Figure 3 The process architecture diagram for data warehousing and data query in the data management system provided for the embodiment of the present invention is as follows: after batch data or real-time data is produced, a warehousing request is submitted, and the warehousing service module performs corresponding batch data processing, multi-version merging or real-time data processing on the data to be stored based on the data type of the data to be stored, and stores the processed data in the underlying storage, and notifies the distributed proxy module of the warehousing operation, specifically sending the metadata of the data to be stored to the distributed proxy module. The distributed proxy module receives the query request, and determines the KV storage cluster storing the data to be queried based on the metadata of the data to be queried included in the query request, queries the data in the KV storage cluster storing the data to be queried and returns it.

[0086] like Figure 4 As shown, Figure 4The schematic diagram of the process architecture of table building, warehousing and query services in the embodiment of the present invention, wherein table building includes calling OP-API, sending a table building request to the corresponding cluster based on the business logic table information and cluster information, and the corresponding cluster can establish a business table based on the table building request. The warehousing process is executed by the warehousing service module, including receiving a data warehousing request, obtaining the data address, cluster and logic table information included in the data warehousing request, calling the computing cluster to convert the data into a KV storage format, and warehousing the data into the corresponding table of the corresponding cluster. And notify the distributed proxy module to update the data metadata, and the distributed proxy module can manage the metadata of all data. The query process may include a distributed proxy module determining the table contained in the data query request and the cluster corresponding to the primary key based on the stored metadata, and routing the data query request to the corresponding cluster for processing. The above cluster may include SSD-type Rainbow engine KV storage, SATA-type Rainbow engine KV storage, and RocksDB engine real-time KV storage, etc.

[0087] By applying the embodiments of the present invention, unified management and optimization of multiple KV storage clusters with different structures can be achieved, and the warehousing and online applications of multiple KV storage clusters with different structures can be effectively managed, thereby improving the efficiency of data processing and the effect of online applications. By using the Netty framework to process data query requests, due to the characteristics of the Netty framework itself, the data management system can process a large number of data requests at the same time, thereby improving the speed and efficiency of data processing, reducing the waiting time of requests, and improving the throughput of the system. And by setting up a distributed proxy module, the decoupling of warehousing and online applications is achieved, thereby maintaining a high degree of independence and scalability between each application service, reducing the degree of coupling between applications, and improving the maintainability and scalability of the system.

[0088] Based on the same inventive concept, an embodiment of the present invention further provides a data warehousing method, which is applied to a warehousing service module in a data management system, wherein the data management system also includes a distributed agent module; Figure 5 As shown, the method includes:

[0089] S501. Receive a data entry request through a first data interface, wherein the data entry request includes the data to be entered and the logical table information of the business to which the data to be entered belongs, and the logical table information includes the target cluster and target table information where the data to be entered needs to be stored. The first data interface is used to receive the data entry request;

[0090] S502, calling a computing cluster to process the data to be stored, to obtain data to be stored;

[0091] S503, storing the data to be stored in the target table of the target cluster;

[0092] S504: Send the metadata of the data to be stored to the distributed proxy module, so that the distributed proxy module stores the metadata of the data to be stored and processes the data query request based on the metadata.

[0093] In a possible embodiment, the logic table information further includes: a merge quantity, where the merge quantity is the quantity of data of the same type to be merged; the computing cluster processes the data to be stored in the database through the following steps:

[0094] Determine the type of data to be stored, the type of data to be stored includes real-time data and batch data;

[0095] Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on the number of merged versions preset for the batch data, and converting the merged data into data in a preset format.

[0096] In a possible embodiment, a whitelist is pre-set for each cluster and each table, the whitelist includes a user identifier with storage query authority, and the data storage request includes the storage user identifier; the method further includes:

[0097] The authority verification is performed on the storage user ID based on the target cluster and the white list of the target table; if the storage user ID passes the authority verification, the computing cluster is called to process the data to be stored to obtain the data to be stored.

[0098] According to another aspect of the present invention, a data query method is provided, which is applied to a distributed agent module of a data management system, wherein the data management system also includes a storage service module; Figure 6 As shown, the method may include:

[0099] S601, receiving a data query request through a second data interface, wherein the data query request includes data to be queried;

[0100] S602. Based on the pre-stored metadata of the data to be queried, the data query request is routed to the storage cluster for processing, and the query result returned by the storage cluster is obtained; the metadata includes the storage cluster of the data to be queried; the metadata is: the warehousing service module receives a data warehousing request through a first data interface, the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored, and the first data interface is used to receive the data warehousing request; calling the computing cluster to process the data to be warehousing to obtain the data to be stored; storing the data to be stored in the target table of the target cluster and then sending it to the data of the distributed proxy module;

[0101] S603: Return the query result to the second data interface.

[0102] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0103] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0104] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0105] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0106] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0108] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0109] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned data warehousing method and data query method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the above-mentioned data warehousing method and data query method by any other appropriate means (e.g., by means of firmware).

[0110] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data management system, characterized in that: The system comprises: The warehousing service module is used to receive a data warehousing request through a first data interface, wherein the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, and the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored; call a computing cluster to process the data to be warehousing to obtain the data to be stored; store the data to be stored in the target table of the target cluster; and send the metadata of the data to be stored to the distributed proxy module; The distributed proxy module is used to receive a data query request through a second data interface, wherein the data query request includes the data to be queried; based on the pre-stored metadata of the data to be queried, route the data query request to a storage cluster for processing, obtain a query result returned by the storage cluster, wherein the metadata includes the storage cluster of the data to be queried; and return the query result to the second data interface; The storage module includes multiple KV storage clusters and is used to store the data to be stored and the data to be queried.

2. The system according to claim 1, characterized in that The logic table information also includes: the merge quantity, which is the number of data of the same type to be merged; the computing cluster processes the data to be stored through the following steps: Determine the type of data to be stored, the type of data to be stored includes real-time data and batch data; Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on the number of merged versions preset for the batch data, and converting the merged data into data in a preset format.

3. The system according to claim 1, characterized in that A whitelist is pre-set for each cluster and each table, wherein the whitelist includes user identifiers with storage and / or query permissions, the data storage request includes the storage user identifier, and the data query request includes the query user identifier; The storage service module is further used to perform permission verification on the storage user ID based on the target cluster and the white list of the target table; if the storage user ID passes the permission verification, the computing cluster is called to process the data to be stored to obtain the data to be stored; The distributed proxy module is further used to perform authority verification on the query user identifier based on the storage cluster and a white list of storage tables in the storage cluster; When the query user identifier passes the authority verification, the data query request is routed to the storage cluster for processing to obtain the query result returned by the storage cluster, wherein the storage table in the storage cluster is used to store the data to be queried.

4. The system according to claim 1, characterized in that The logical table information also includes the number of version tables that the business needs to save, and different version tables correspond to data generated at different times; the warehousing service module is used to disable the current version table when receiving data exception information, and use the previous version table as the new current version table. The new current version table is used to provide a basis for data query, and the previous version table is a table whose data generation time is before the current version table.

5. A data storage method, characterized in that: A storage service module is applied to a data management system, wherein the data management system also includes a distributed agent module; the method includes: Receive a data entry request through a first data interface, wherein the data entry request includes the data to be entered and the logical table information of the business to which the data to be entered belongs, and the logical table information includes the target cluster and target table information where the data to be entered needs to be stored, and the first data interface is used to receive the data entry request; Calling a computing cluster to process the data to be stored to obtain data to be stored; Storing the data to be stored in the target table of the target cluster; The metadata of the data to be stored is sent to the distributed proxy module, so that the distributed proxy module stores the metadata of the data to be stored and processes the data query request based on the metadata.

6. The method according to claim 5, characterized in that The logic table information also includes: the merge quantity, which is the number of data of the same type to be merged; the computing cluster processes the data to be stored through the following steps: Determine the type of data to be stored, the type of data to be stored includes real-time data and batch data; Based on the type of data to be stored, a preset algorithm is used to process the data to be stored to obtain the data to be stored, wherein the preset algorithm for the real-time data includes extracting the key-value data of the data to be stored, and converting the key-value data into data in a preset format; the preset algorithm for the batch data includes extracting the key-value data of the data to be stored, merging multiple versions of the key-value data based on the number of merged versions preset for the batch data, and converting the merged data into data in a preset format.

7. The method according to claim 5, characterized in that A whitelist is pre-set for each cluster and each table, the whitelist includes a user identifier with storage query authority, and the data storage request includes the storage user identifier; the method further includes: The authority verification is performed on the storage user ID based on the target cluster and the white list of the target table; if the storage user ID passes the authority verification, the computing cluster is called to process the data to be stored to obtain the data to be stored.

8. A data query method, characterized in that: A distributed proxy module applied to a data management system, wherein the data management system also includes a warehousing service module; the method includes: Receiving a data query request through a second data interface, wherein the data query request includes data to be queried; Based on the pre-stored metadata of the data to be queried, routing the data query request to the storage cluster for processing, and obtaining the query result returned by the storage cluster; The metadata includes the storage cluster of the data to be queried; the metadata is: the warehousing service module receives a data warehousing request through a first data interface, the data warehousing request includes the data to be warehousing and the logical table information of the business to which the data to be warehousing belongs, the logical table information includes the target cluster and target table information where the data to be warehousing needs to be stored, the first data interface is used to receive the data warehousing request; call the computing cluster to process the data to be warehousing to obtain the data to be stored; store the data to be stored in the target table of the target cluster and then send it to the data of the distributed proxy module; The query result is returned to the second data interface.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 5-7 or 8.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 5-7 or 8.