Data management method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses a data management method and device, equipment, a medium and a product, and the method comprises the steps: carrying out the authentication of a mapping relation between a first account and a first metadata directory in response to a data interaction request of the first account for the first metadata directory, and determining an authentication result; in response to authentication passing, determining data index information of data associated with the data interaction request in the first file system; the first file system is a file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system; performing data processing operation on corresponding data in the first file system according to the data index information; the data processing operation is an operation associated with the data interaction request. According to the technical scheme, the problem that library tables of different tenants cannot be managed independently is solved, and data storage isolation and management independence of the library tables among multiple tenants can be truly achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data management, and in particular to a data management method, device, equipment, medium and product. BACKGROUND

[0002] The existing multi-tenant implementation method in Hive (Hadoop Interactive Query Execution, interactive query execution based on Hadoop) mainly uses library table naming or data partitioning to distinguish the data of different tenants. However, in the process of implementing the present application, it is found that at least the following technical problems exist in the prior art:

[0003] The library tables of different tenants cannot be independently managed; all data processed by Hive is stored in HDFS (Hadoop Distributed File System, distributed file system), and is organized in the form of files (such as directories created according to tables and partitions), and the data of different tenants is either in the same HDFS cluster or needs to deploy multiple HDFS clusters or multiple HDFS NameServices, and the underlying storage cannot be independently implemented or cannot be dynamically implemented for multi-tenancy. SUMMARY

[0004] Embodiments of the present application provide a data management method, device, equipment, medium and product, which can independently store the database tables corresponding to the accounts of each database tenant, and independently implement resource isolation and data management of the database tables from the bottom.

[0005] In a first aspect, embodiments of the present application provide a data management method, which comprises:

[0006] In response to a data interaction request of a first account for a first metadata directory, authenticating a mapping relationship between the first account and the first metadata directory, and determining an authentication result;

[0007] In response to the authentication result being authentication passed, determining data index information of data associated with the data interaction request in a first file system; wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system;

[0008] Performing a data processing operation on the corresponding data in the first file system according to the data index information; wherein the data processing operation is an operation associated with the data interaction request.

[0009] In a second aspect, embodiments of the present application further provide a data management device, which comprises:

[0010] The request verification module is configured to, in response to the data interaction request of the first account for the first metadata directory, authenticate the mapping relationship between the first account and the first metadata directory, and determine an authentication result;

[0011] The request association data confirmation module is configured to, in response to the authentication result being authentication passed, determine data index information of data associated with the data interaction request in the first file system, wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system.

[0012] The request execution module is configured to perform a data processing operation on corresponding data in the first file system according to the data index information, wherein the data processing operation is an operation associated with the data interaction request.

[0013] In a third aspect, an embodiment of the present application further provides a computer device, which comprises:

[0014] one or more processors; a memory configured to store one or more programs;

[0015] when the one or more programs are executed by the one or more processors, the one or more processors implement the data management method provided by any embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the data management method provided by any embodiment of the present application.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the data management method provided by any embodiment of the present application.

[0018] The above-mentioned embodiments of the present application have the following advantages or beneficial effects:

[0019]

[0020] In the embodiment of the present application, in response to a data interaction request of a first account for a first metadata directory, a mapping relationship between the first account and the first metadata directory is authenticated, and an authentication result is determined; in response to the authentication result being authentication passed, data index information of data associated with the data interaction request in a first file system is determined; wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system; and a data processing operation is performed on corresponding data in the first file system according to the data index information; wherein the data processing operation is an operation associated with the data interaction request. The technical scheme of the embodiment of the present application solves the problem that the database tables of different tenants cannot be independently managed, so that the database tables corresponding to the accounts of each database tenant are independently stored, and resource isolation and independent data management of the database tables are realized from the bottom. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a data management method provided by the embodiment of the present application;

[0022] Figure 2 is a flowchart of another data management method provided by the embodiment of the present application;

[0023] Figure 3 is a logical relationship diagram of a Hive multi-tenant database table based on a preset distributed storage system provided by the embodiment of the present application;

[0024] Figure 4 is a creation flowchart of a Hive tenant provided by the embodiment of the present application;

[0025] Figure 5 is a use flowchart of a Hive tenant provided by the embodiment of the present application;

[0026] Figure 6 is a structural schematic diagram of a data management device provided by the embodiment of the present application;

[0027] Figure 7 is a structural schematic diagram of a computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0029] Hive is a data warehouse tool based on the Hadoop ecosystem. It transforms user-written SQL-like statements (HQL, Hive Query Language) into distributed computing tasks, thereby enabling the analysis and processing of massive amounts of data in HDFS (Hadoop Distributed File System). All data processed by Hive is stored in HDFS and organized in file format (e.g., by creating directories based on tables or partitions). Hive locates data through path information in metadata, and the computing engine reads and writes data directly from HDFS.

[0030] HDFS provides high-throughput distributed storage services for the Hadoop ecosystem, but its native architecture has significant limitations in supporting multi-tenancy in cloud-native scenarios. HDFS lacks metadata definitions for "tenants," and all isolation relies on manual configuration of directories and permissions. As the number of tenants increases (e.g., hundreds or thousands), management complexity rises exponentially. Although it supports space quotas, it cannot limit tenant I / O bandwidth, remote call request frequency, etc., and a large number of reads and writes by a single tenant can impact the performance of the entire cluster.

[0031] Figure 1 This is a flowchart illustrating a data management method provided in an embodiment of the present invention. This embodiment is applicable to multi-tenant data management scenarios, particularly multi-tenant data management in cloud-native scenarios. The method can be executed by a data management device, which can be implemented in software and / or hardware and integrated into a computer device with application development capabilities.

[0032] like Figure 1 As shown, the data management method in this embodiment includes the following steps:

[0033] S110. In response to the data interaction request from the first account to the first metadata directory, authenticate the mapping relationship between the first account and the first metadata directory, and determine the authentication result.

[0034] The first account can be information corresponding to a tenant. A tenant can refer to an independent user or organization in the fields of computer technology and cloud computing that shares the same set of infrastructure (such as servers, storage, software platforms, etc.), and the data and resources of tenants are isolated from each other to ensure security and independence.

[0035] The first account can be any account among several tenants in an infrastructure. Users of the first account can manage data by entering data interaction requests through the corresponding command-line interface of the data warehouse user interface (Client).

[0036] The data interaction request can be any request interacting with data, such as at least one of creation of a database table, data query, data addition, data deletion, and data modification. The form corresponding to the data interaction request can be an HQL query statement form.

[0037] The "metadata warehouse" of the Hive stores metadata of all libraries, tables, columns, and partitions, such as table names, databases to which the table names belong, field names, and types; storage paths of data in a distributed file system, file formats, partition rules, serialization modes, and the like. In the embodiment, the Hive can also store mapping relationships between the first account and the first metadata catalog, and other metadata information related to account information. For example, the account name and identifier of the first account, and the name and identifier information of the corresponding metadata catalog, and the like.

[0038] In response to the data interaction request of the first account for the first metadata catalog, account information of the first account issuing the data interaction request can be determined, and then metadata corresponding to the first account can be determined, the mapping relationship between the first account and the first metadata catalog is authenticated according to the metadata, and an authentication result is determined.

[0039] Authenticating the mapping relationship between the first account and the first metadata catalog can be determining whether the first account has the corresponding first metadata catalog. The metadata catalog can be a structured collection storing database metadata, used to describe objects in the database, such as tables, columns, indexes, views, user permissions, and the like. The first metadata catalog can include database information and data table information created by the first account. Different first accounts can correspond to different metadata catalogs.

[0040] In response to the authentication result being authentication passed, S120, data index information of data associated with the data interaction request in the first file system is determined.

[0041] The authentication result being authentication passed can mean that there is a first metadata catalog associated with the first account. The first file system is one file storage instance associated with the first metadata catalog in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system.

[0042] The preset distributed storage system can be a cloud-native distributed storage system specially designed for a cloud environment, supporting elasticity, containerized deployment, and dynamic scheduling, and can adapt to the storage requirements of cloud-native applications (such as containerized applications and microservice architectures) and fully utilize the elasticity, distribution, and automation characteristics of the cloud environment. The multi-tenant mechanism realizes isolation through tenant-level namespaces, resource quotas, and permission boundaries, and each tenant has an independent storage pool (logical resource division), and data directories are isolated through encryption and metadata in the physical storage layer to ensure that tenant data cannot be accessed by each other. This isolation allows each tenant's file system to be considered as an "independent instance", even if the underlying shared storage cluster, the data and resource boundaries can be maintained.

[0043] The preset distributed storage system can support the ability of different tenants to dynamically provide dedicated file systems. This mode not only meets the needs of tenants for "customized file system interfaces", but also improves resource efficiency through shared underlying storage, which is a typical advantage of cloud-native distributed storage in multi-scenario and multi-organization collaboration scenarios. That is, each first account can correspond to a metadata directory and a first file system.

[0044] According to the content of the data interaction request, the data index information of the data that needs to be processed is determined, which can be achieved by combining the associated metadata to perform semantic analysis on the data interaction request, verifying whether the table, column, and function exist, binding the data type and storage information (such as HDFS path and file format), and thus determining the data index information of the corresponding first file system.

[0045] S130, according to the data index information, performing a data processing operation on the corresponding data in the first file system; wherein the data processing operation is an operation associated with the data interaction request.

[0046] The data processing operation includes at least one of creating a database table, querying data, adding data, deleting data, and modifying data.

[0047] The technical scheme of the embodiment is characterized in that, in response to a data interaction request of a first account for a first metadata directory, the mapping relationship between the first account and the first metadata directory is authenticated, and an authentication result is determined; in response to the authentication result being authentication passed, data index information of data associated with the data interaction request in a first file system is determined, wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system; and a data processing operation is performed on corresponding data in the first file system according to the data index information, wherein the data processing operation is an operation associated with the data interaction request. The technical scheme of the embodiment solves the problem that the database tables of different tenants cannot be independently managed, enables the database tables corresponding to the accounts of each database tenant to be independently stored, and realizes resource isolation and independent data management of the database tables from the bottom. That is, by using the multi-tenant feature of the preset distributed storage system, the efficient sharing and management of “one cluster serving multiple users” are realized by multi-level isolation of resources, data and permissions on the basis of the preset distributed storage system.

[0048] Figure 2 The flowchart of another data management method provided by the embodiment is described further in the same inventive concept as the data management method in the above embodiment, and the process of creating a metadata directory is further described. The method can be executed by a data management device, which can be realized by software and / or hardware and integrated in a computer device with application development functions.

[0049] As shown in Figure 2 , the data management method of the embodiment includes the following steps:

[0050] S210, in response to a metadata directory creation operation of a first account, a first metadata directory associated with the first account is created, and metadata information corresponding to the first account and the first metadata directory is stored in a corresponding metadata database.

[0051] The data management device can be a data management device corresponding to a data warehouse, and a user with a data management requirement can create a user-level metadata directory, i.e., a corresponding metadata directory, in the data management device through a corresponding first account. The first metadata directory includes corresponding database information and data table information.

[0052] The metadata information corresponding to the first account and the first metadata directory can be identification information of the first account, identification information of the first metadata directory, and can also be a mapping relationship between the first account and the first metadata directory.

[0053] S220, an association relationship between the first metadata directory and a first file system is established, and metadata information corresponding to the first file system is stored in the metadata database.

[0054] The first file system is a file storage instance created in advance in the preset distributed storage system. The association relationship between the first metadata directory and the first file system can be to determine a corresponding data space for the first account.

[0055] In an example, a Hive multi-tenant (user) library table logical relationship based on a preset distributed storage system can refer to the relationship structure shown in Figure 3 The corresponding relationship between the Catalog, library, table and user of Hive, different users (tenants) correspond to different Catalogs, such as Ctlg-1 and Ctlg-2, and different Catalogs are bound to different file system instances; the business layer library table is exactly the same as using a dedicated Hive service, wherein, Figure 3 In the embodiment, prod and dev represent different business libraries, and t01 and t02 represent different business tables.

[0056] In an embodiment, only administrators can create accounts, that is, create Hive Catalogs, and return the Hive Catalog information of the created account and the corresponding file system information and authentication information to the account and its members, so as to use Hive later.

[0057] S230, generating authentication information of the first account based on at least one of the account information of the first account, the directory information of the first metadata directory, and the system information of the first file system.

[0058] The authentication information can be a string of codes generated based on at least one of the account information of the first account, the directory information of the first metadata directory, and the system information of the first file system, and returned to the account and its members, so as to use Hive later. The authentication information can also be placed in the Hive metadata database as metadata.

[0059] Figure 4 is a Hive tenant creation process schematic diagram provided by an embodiment of the application, which illustrates the creation process of the Hive Catalog in the account creation instance. The process occurs in the Hive Metastore service, first creates the Hive Catalog metadata corresponding to the tenant, then creates and binds the file system instance corresponding to the Catalog, and finally generates the access authentication information of the tenant.

[0060] S240, in response to a data interaction request of the first account to the first metadata directory, authenticating the mapping relationship between the first account and the first metadata directory according to the authentication information, and determining an authentication result.

[0061] S250, in response to the authentication result being authentication passed, determining data associated with the data interaction request in data index information of the first file system.

[0062] Wherein, the first file system is one file storage instance associated with the first metadata directory in the preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system.

[0063] S260, performing data processing operation on corresponding data in the first file system according to the data index information, wherein the data processing operation is an operation associated with the data interaction request.

[0064] Figure 5 The embodiment of the application provides a kind of use flow schematic diagram of Hive tenant;Tenant and its member must carry tenant access authentication information (such as account login password etc.) to access the tenant Hive Catalog information, and only after Catalog information authentication of tenant can all subsequent operations be carried out.Use Hive, first perform tenant Catalog information authentication, then obtain library table information and file system instance related information, connect with file system to carry out subsequent data related operation.

[0065] The technical solution of this embodiment, in response to the metadata directory creation operation of the first account, creates a first metadata directory associated with the first account and stores the metadata information corresponding to the first account and the first metadata directory in the corresponding metadata database; establishes an association between the first metadata directory and the first file system and stores the metadata information corresponding to the first file system in the metadata database; wherein, the first file system is a file storage instance pre-created in a preset distributed storage system; generates authentication information for the first account based on at least one of the account information of the first account, the directory information of the first metadata directory, and the system information of the first file system; in response to the data interaction request of the first account for the first metadata directory, authenticates the mapping relationship between the first account and the first metadata directory according to the authentication information and determines the authentication result; in response to the authentication result being successful, determines the data index information of the data associated with the data interaction request in the first file system; wherein, the first file system is a file storage instance associated with the first metadata directory in the preset distributed storage system, and the preset distributed storage system can create more than one file storage instance; performs data processing operations on the corresponding data in the first file system according to the data index information; wherein, the data processing operation is the operation associated with the data interaction request. The technical solution of this invention solves the problem of the inability to independently manage database tables for different tenants, enabling independent storage of the database tables corresponding to each database tenant's account, and achieving resource isolation and independent data management of database tables from the underlying level. Based on the multi-file system instance mechanism of the pre-defined distributed storage system, and through the extension of the HiveCatalog concept and some related modifications, the isolation of data and database tables for Hive multi-tenants is achieved, and the independent development of data warehouses among multi-tenants is fully realized.

[0066] Figure 6 This is a schematic diagram of a data management device provided in an embodiment of the present invention. This embodiment is applicable to multi-tenant data storage and management scenarios. The data management device can be implemented by software and / or hardware and integrated into a computer terminal device with application development capabilities.

[0067] like Figure 6 The data management device shown includes: a request verification module 310, a request associated data confirmation module 320, and a request execution module 330.

[0068] The request verification module 310 is configured to authenticate the mapping relationship between the first account and the first metadata directory in response to the data interaction request of the first account for the first metadata directory, and determine an authentication result; the request associated data confirmation module 320 is configured to determine data index information of data associated with the data interaction request in the first file system in response to the authentication result being authentication passed; the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system; and the request execution module 330 is configured to perform a data processing operation on corresponding data in the first file system according to the data index information; the data processing operation is an operation associated with the data interaction request.

[0069] The technical scheme of the embodiment is configured to authenticate the mapping relationship between the first account and the first metadata directory in response to the data interaction request of the first account for the first metadata directory, and determine an authentication result; determine data index information of data associated with the data interaction request in the first file system in response to the authentication result being authentication passed; the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system; and perform a data processing operation on corresponding data in the first file system according to the data index information; the data processing operation is an operation associated with the data interaction request. The technical scheme of the embodiment solves the problem that the database tables of different tenants cannot be independently managed, and enables the database tables corresponding to the accounts of each database tenant to be independently stored, thereby realizing resource isolation and independent data management of the database tables from the bottom.

[0070] In an optional embodiment, the first metadata directory includes database information and data table information created by the first account.

[0071] In an optional embodiment, the data processing operation includes at least one of creation, data query, data addition, data deletion, and data modification of the database table.

[0072] In an optional embodiment, the data management apparatus further includes an account creation module configured to:

[0073] Before the data interaction request is obtained, the first metadata directory associated with the first account is created in response to a metadata directory creation operation of the first account, and metadata information corresponding to the first account and the first metadata directory is stored in a corresponding meta-database.

[0074] Establish the association between the first metadata directory and the first file system, and store the metadata information corresponding to the first file system in the metadata database; wherein, the first file system is a file storage instance pre-created in the preset distributed storage system.

[0075] In one alternative implementation, the data management device further includes an account creation module, which can also be used for:

[0076] The authentication information of the first account is generated based on at least one of the account information of the first account, the directory information of the first metadata directory, and the system information of the first file system.

[0077] In an optional implementation, the request verification module 310 can also be used to:

[0078] The mapping relationship between the first account and the first metadata directory is authenticated based on the authentication information.

[0079] The data management device provided in the embodiments of the present invention can execute the data management method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0080] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 7 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 7 The computer device 12 shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as intelligent controllers and servers, mobile phones, and other terminal devices.

[0081] like Figure 7 As shown, the computer device 12 is represented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0082] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0083] Computer device 12 typically includes a variety of computer system readable media. Such media can be any available media that is located either internally or externally to computer device 12, such as volatile and non-volatile media, removable and non-removable media.

[0084] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Computer device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 7 not shown in FIG. 1, a magnetic hard disk drive, for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Although not specifically shown, such Figure 7 In alternative embodiments, a magnetic hard disk drive, a solid state drive (SSD) which is a non- volatile computer storage media, a floppy disk drive for reading from and / or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and / or an optical disk drive for reading from or writing to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, and / or other optical media) can be provided. In these instances, each drive can be connected to the system bus 18 by one or more data media interfaces. The system memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0085] Program / utility 40, having a set (at least one) of program modules 42, can be stored in system memory 28 by way of example, and can include an operating system, one or more application programs, other program modules, and program data, each of which

[0086] Computer device 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer device 12; and / or one or more devices that enable computer device 12 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interfaces 22. Further, computer device 12 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via network adapter 20. As depicted, network adapter 20 communicates with the other components of computer device 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software Figure 7Other hardware and / or software modules can be used in conjunction with computer device 12, as noted above, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0087] Processing unit 16 performs various function applications and data processing by running programs stored in system memory 28, such as implementing the data management method provided by embodiments of the present application, which includes:

[0088] In response to the authentication result being authentication passed, determining data index information of data associated with the data interaction request in a first file system; wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system;

[0089] Performing a data processing operation on corresponding data in the first file system according to the data index information; wherein the data processing operation is an operation associated with the data interaction request.

[0090] In particular, the processes described above with reference to the flowcharts can be implemented as computer software programs according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication unit, or installed from system memory 28, or installed from a ROM. When the computer program is executed by processing unit 16, the above-mentioned functions defined in the methods of embodiments of the present application are performed.

[0091] Embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data management method provided by any embodiment of the present application, which includes:

[0092] In response to the authentication result being authentication passed, determining data index information of data associated with the data interaction request in a first file system; wherein the first file system is one file storage instance associated with the first metadata directory in a preset distributed storage system, and more than one file storage instance can be created in the preset distributed storage system;

[0093] Performing a data processing operation on corresponding data in the first file system according to the data index information; wherein the data processing operation is an operation associated with the data interaction request.

[0094] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0095] The computer readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which computer readable program code is embodied. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer readable signal medium can also be any computer readable medium that is not a storage medium, that is capable of storing the program for use by or in connection with the instruction execution system, apparatus or device.

[0096] The program code contained on the computer readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0097] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry can execute the program code. In some embodiments, multiple processors or multiple cores can execute the program code.

[0098] Those skilled in the art should understand that each module or step of the present application described above can be implemented by a general computing device, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, each module or step can be implemented by computer-executable program code, which can be stored in a storage device and executed by a computing device, or each module or step can be implemented by an individual integrated circuit module, or a plurality of modules or steps can be implemented by a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0099] The present disclosure also provides a computer program product, comprising a computer program which, when executed by a processor, implements the data management method provided by any one of the embodiments of the present disclosure.

[0100] In implementing the computer program product, the computer program code for performing the operations of the present disclosure can be written in one or more programming languages or combinations of languages including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] Note that the above are only the preferred embodiments of the present application and the principles of technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data management method applied to a data warehouse client, characterized in that, include: In response to a data interaction request from a first account to a first metadata directory, the mapping relationship between the first account and the first metadata directory is authenticated, and the authentication result is determined. In response to the authentication result being successful, the data index information of the data associated with the data interaction request is determined in the first file system; wherein, the first file system is a file storage instance associated with the first metadata directory in a preset distributed storage system, and the preset distributed storage system can create more than one file storage instance; Data processing operations are performed on the corresponding data in the first file system based on the data index information; wherein, the data processing operation is an operation associated with the data interaction request.

2. The method according to claim 1, characterized in that, The first metadata directory includes database information and data table information created by the first account.

3. The method according to claim 1, characterized in that, The data processing operations include at least one of the following: database table creation, data query, data addition, data deletion, and data modification.

4. The method according to claim 1, characterized in that, Before receiving the data interaction request, the method includes: In response to the metadata directory creation operation of the first account, a first metadata directory associated with the first account is created, and the metadata information corresponding to the first account and the first metadata directory is stored in the corresponding metadata database; Establish the association between the first metadata directory and the first file system, and store the metadata information corresponding to the first file system in the metadata database; wherein, the first file system is a file storage instance pre-created in the preset distributed storage system.

5. The method according to claim 4, characterized in that, The method further includes: The authentication information of the first account is generated based on at least one of the account information of the first account, the directory information of the first metadata directory, and the system information of the first file system.

6. The method according to claim 5, characterized in that, The authentication of the mapping relationship between the first account and the first metadata directory includes: The mapping relationship between the first account and the first metadata directory is authenticated based on the authentication information.

7. A data management device, configured on a data warehouse client, characterized in that, include: The request verification module is used to respond to the data interaction request from the first account to the first metadata directory, authenticate the mapping relationship between the first account and the first metadata directory, and determine the authentication result. The request associated data confirmation module is used to determine the data index information of the data associated with the data interaction request in the first file system in response to the authentication result being successful; wherein, the first file system is a file storage instance associated with the first metadata directory in a preset distributed storage system, and the preset distributed storage system can create more than one file storage instance; The request execution module is used to perform data processing operations on the corresponding data in the first file system according to the data index information; wherein the data processing operation is an operation associated with the data interaction request.

8. A computer device, characterized in that, The computer device includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data management method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data management method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data management method as described in any one of claims 1-6.

Images