Separate storage and management method and device, equipment and storage medium

Abstract

The invention discloses a separate storage and management method, which comprises the following steps of: managing storage strategies of different models through a unified configuration center by using a distributed database as a back end of metadata storage; when the description type metadata are stored, each description type record corresponds to the metadata of one unstructured data item, and the metadata are stored in a distributed database in a database record form; in the process of storing the description type metadata, adding a storage strategy of unstructured data and obtaining parameters as a part of the metadata; the storage strategy is a specific storage mode of the unstructured data; the parameters are retrieval modes of the unstructured data. According to the scheme, separated storage of the unstructured data and the description metadata is realized, so that the storage medium can be flexibly selected, the storage medium can be iteratively replaced according to business requirements and development of a storage technology, migration of the unstructured data is also facilitated, and a more efficient and more intelligent data management mode is realized.

Description

Technical Field

[0001] The present invention relates to the field of Internet data storage technology, particularly to the field of application to unstructured data storage, and more particularly to a method, device and storage medium for separating storage and management. Background Art

[0002] With the rapid advancement of science and technology, storage technology is also constantly innovating and evolving. From the early NAS disks, to IBM's CM storage products, to the popular big data distributed file system HDFS, and the latest object storage, each technological innovation has brought us more efficient and flexible data storage solutions.

[0003] However, with the diversification of storage technologies, the interfaces and SDK calling methods of various storage technologies have also shown huge differences. This requires developers to invest a lot of time and energy to familiarize themselves with and master various technical details when integrating different storage systems. What is more difficult is that some storage technologies lack support for document description metadata, which makes it impossible to effectively retrieve and query document properties. This not only affects the efficiency of data utilization, but also increases the difficulty of data management.

[0004] Therefore, storage technology not only needs to continue to improve performance and capacity, but also needs to pay more attention to the support and management of metadata. Only in this way can we better utilize data and achieve more efficient and intelligent data management. Summary of the invention

[0005] This application proposes a strategy for storing unstructured data using multiple storage settings, which is used to shield the differences between different storage settings, provide unified interfaces for adding, modifying, appending, and deleting, and at the same time, use a distributed database to store and retrieve metadata for metadata of unstructured files.

[0006] To this end, the present invention provides a method, device and storage medium for separating storage and management to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a method for separate storage and management, which is applied to separate storage of unstructured data and metadata of unstructured files, comprising:

[0008] Use a distributed database as the backend for metadata storage, and manage storage strategies for different models through a unified configuration center.

[0009] When storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item and is stored in the distributed database in the form of a database record.

[0010] In the process of storing the descriptive metadata, a storage strategy and acquisition parameters of the unstructured data are added as part of the metadata; the storage strategy is a specific storage method of the unstructured data; and the parameters are a retrieval method of the unstructured data.

[0011] Furthermore, the descriptive metadata includes metadata information describing the unstructured data, such as the uploader of the unstructured data, the upload time, the file name, the file storage path, the file storage strategy, and the file acquisition parameters.

[0012] Furthermore, the specific storage method of the unstructured data is: managing the storage strategies of different models through a unified configuration center, and one model corresponds to multiple storage strategies.

[0013] Furthermore, the storage strategy is provided with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then according to the storage strategy in the record, the corresponding storage medium is matched to find the required target file.

[0014] In a second aspect, the present invention further provides a device for separating storage and management, the device comprising: a deployment unit, a configuration unit, and an expansion unit;

[0015] The deployment unit is used to use a distributed database as a backend for metadata storage and manage storage strategies of different models through a unified configuration center.

[0016] The configuration unit is used to store metadata of an unstructured data item corresponding to each descriptive record in the form of a database record in the distributed database when storing descriptive metadata.

[0017] The extension unit is used to add storage strategy and acquisition parameters of unstructured data as part of metadata during the process of storing the descriptive metadata; the storage strategy is a specific storage method of the unstructured data; and the parameters are a retrieval method of the unstructured data.

[0018] Furthermore, the descriptive metadata includes metadata information describing the unstructured data, such as the uploader of the unstructured data, the upload time, the file name, the file storage path, the file storage strategy, and the file acquisition parameters.

[0019] Furthermore, the specific storage method of the unstructured data is: managing the storage strategies of different models through a unified configuration center, and one model corresponds to multiple storage strategies.

[0020] Furthermore, the storage strategy is provided with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then according to the storage strategy in the record, the corresponding storage medium is matched to find the required target file.

[0021] In a third aspect, the present invention provides a computing device, comprising:

[0022] A memory for storing program instructions;

[0023] The processor is used to call the program instructions stored in the memory and execute the above-mentioned method of separating storage and management according to the obtained program instructions.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the above-mentioned method of separating storage and management is implemented. The readable medium may be multiple, and the multiple readable media can operate independently of each other.

[0025] Compared with the existing technology, this solution uses a distributed database as the backend for metadata storage, and manages the storage strategies of different models through a unified configuration center; when storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item, and is stored in the distributed database in the form of a database record; in the process of storing descriptive metadata, the storage strategy and acquisition parameters of unstructured data are added as part of the metadata; the storage strategy is the specific storage method of unstructured data; the parameters are the retrieval method of unstructured data. This solution realizes the separate storage of unstructured data and descriptive metadata, so that the storage medium can be flexibly selected, and the storage medium can be iteratively replaced according to business needs and the development of storage technology. It also facilitates the migration of unstructured data and realizes a more efficient and intelligent data management model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A flow chart of an embodiment of the method for separating storage and management provided by the present invention;

[0028] Figure 2A framework diagram of an embodiment of the method for separating storage and management provided by the present invention;

[0029] Figure 3 A structural block diagram of an embodiment of the device for separating storage and management provided by the present invention. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0031] Those skilled in the art will appreciate that, unless expressly stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0032] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0033] The core of the present invention is to provide a method for separating storage and management. Figure 1 FIG. 1 is a flow chart of the method for separating storage and management provided in this embodiment. Figure 2 The figure shows a framework diagram of the method for separating storage and management provided in this embodiment. In order to better understand the method, Figure 1 and Figure 2 The technical solution of this embodiment is described in detail:

[0034] A method for separate storage and management is applied to separate storage of unstructured data and metadata of unstructured files, comprising:

[0035] Step S1: Use a distributed database as the backend for metadata storage, and manage storage policies for different models through a unified configuration center; ensure that policy selection and application are consistent and flexible.

[0036] Step S2: When storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item and is stored in the distributed database in the form of a database record; this ensures structured storage and efficient retrieval of metadata.

[0037] Step S3: In the process of storing the descriptive metadata, add the storage strategy and acquisition parameters of the unstructured data as part of the metadata; the strategy is the specific storage method of the unstructured data; the parameters are the retrieval method of the unstructured data, which provides an important basis for subsequent data management.

[0038] The management and configuration of storage strategies are ensured, and then the structured storage of metadata is performed, and the storage strategy and acquisition parameters of unstructured data are integrated in the storage process. Such a process makes the entire storage and management process more orderly and efficient.

[0039] In a preferred embodiment, the descriptive metadata includes metadata information describing the unstructured data, such as the uploader of the unstructured data, upload time, file name, file storage path, file storage strategy, and file acquisition parameters.

[0040] In a preferred embodiment, the specific storage method of the unstructured data is: the storage strategies of different models are managed by a unified configuration center, and one model corresponds to multiple storage strategies. For example, the time slice configuration from 20201030 to 20221030 is stored in the big data cluster.

[0041] In a preferred embodiment, the storage strategy is provided with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then the corresponding storage medium is matched according to the storage strategy in the record to find the required target file.

[0042] For example, if policy 1 is object storage and policy 2 is big data, object storage is queried first during query. If the file does not exist, big data is queried.

[0043] In the scheme of this embodiment, by using a distributed database as the backend for metadata storage, the storage strategies of different models are managed through a unified configuration center; when storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item, and is stored in the distributed database in the form of a database record; in the process of storing descriptive metadata, the storage strategy and acquisition parameters of unstructured data are added as part of the metadata; the storage strategy is the specific storage method of unstructured data; the parameters are the retrieval method of unstructured data. This scheme realizes the separate storage of unstructured data and descriptive metadata, so that the storage medium can be flexibly selected, and the storage medium can be iteratively replaced according to business needs and the development of storage technology. It also facilitates the migration of unstructured data and realizes a more efficient and intelligent data management model.

[0044] In one embodiment, a separate storage and management device is provided, which is applied to separate storage of unstructured data and metadata of unstructured files, see Figure 3 The figure shows a structural block diagram of a device for separating storage and management provided by this embodiment, wherein the device comprises: a deployment unit 10, a configuration unit 20, and an expansion unit 30;

[0045] Deployment unit 10: Use a distributed database as the backend for metadata storage, and manage storage policies for different models through a unified configuration center; ensure consistency and flexibility in policy selection and application.

[0046] Configuration unit 20: When storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item and is stored in the distributed database in the form of a database record; this can ensure structured storage and efficient retrieval of metadata.

[0047] Extension unit 30: In the process of storing the descriptive metadata, the storage strategy and acquisition parameters of the unstructured data are added as part of the metadata; the strategy is the specific storage method of the unstructured data; the parameters are the retrieval method of the unstructured data, which provides an important basis for subsequent data management.

[0048] The management and configuration of storage strategies are ensured, and then the structured storage of metadata is performed, and the storage strategy and acquisition parameters of unstructured data are integrated in the storage process. Such a process makes the entire storage and management process more orderly and efficient.

[0049] In a preferred embodiment, the descriptive metadata includes metadata information describing the unstructured data, such as the uploader of the unstructured data, upload time, file name, file storage path, file storage strategy, and file acquisition parameters.

[0050] In a preferred embodiment, the specific storage method of the unstructured data is: the storage strategies of different models are managed by a unified configuration center, and one model corresponds to multiple storage strategies.

[0051] In a preferred embodiment, the storage strategy is provided with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then the corresponding storage medium is matched according to the storage strategy in the record to find the required target file.

[0052] In the scheme of this embodiment, by using a distributed database as the backend for metadata storage, the storage strategies of different models are managed through a unified configuration center; when storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item, and is stored in the distributed database in the form of a database record; in the process of storing descriptive metadata, the storage strategy and acquisition parameters of unstructured data are added as part of the metadata; the storage strategy is the specific storage method of unstructured data; the parameters are the retrieval method of unstructured data. This scheme realizes the separate storage of unstructured data and descriptive metadata, so that the storage medium can be flexibly selected, and the storage medium can be iteratively replaced according to business needs and the development of storage technology. It also facilitates the migration of unstructured data and realizes a more efficient and intelligent data management model.

[0053] The device for separation of storage and management of this embodiment is used to implement the aforementioned method for separation of storage and management. Therefore, the specific implementation methods of the device for separation of storage and management can be seen in the specific implementation methods of the method for separation of storage and management in the previous text. For example, the deployment unit 10 is used to implement step S1 in the aforementioned method for separation of storage and management, the configuration unit 20 is used to implement step S2 in the aforementioned method for separation of storage and management, and the expansion unit 30 is used to implement step S3 in the aforementioned method for separation of storage and management. Therefore, its specific implementation methods can refer to the descriptions of the corresponding embodiments of each part. The device provided by the present invention is attached to the method provided by the present invention. Its specific effects can refer to the aforementioned method and will not be repeated here.

[0054] In one embodiment, a computing device is provided, comprising:

[0055] A memory for storing program instructions;

[0056] A processor is used to implement the steps of the method of separating storage and management as described in any one of the above when executing the computer program. A method of separating storage and management of the present invention has obvious advantages and beneficial effects compared with the prior art. By using a distributed database as the backend of metadata storage, the storage strategies of different models are managed through a unified configuration center; when storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item, and is stored in the distributed database in the form of a database record; in the process of storing descriptive metadata, the storage strategy and acquisition parameters of the unstructured data are added as part of the metadata; the storage strategy is the specific storage method of the unstructured data; the parameter is the retrieval method of the unstructured data. This solution realizes the separate storage of unstructured data and descriptive metadata, so that the storage medium can be flexibly selected, and the storage medium can be iteratively replaced according to business needs and the development of storage technology. It is also convenient for the migration of unstructured data, and a more efficient and intelligent data management mode is realized.

[0057] In one embodiment, a computer-readable storage medium is provided, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the steps of the above-mentioned method of separating storage and management are implemented. Compared with the prior art, the method of separating storage and management of the present invention has obvious advantages and beneficial effects. By using a distributed database as the backend of metadata storage, the storage strategies of different models are managed through a unified configuration center; when storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item, and is stored in the distributed database in the form of a database record; in the process of storing descriptive metadata, the storage strategy and acquisition parameters of the unstructured data are added as part of the metadata; the storage strategy is the specific storage method of the unstructured data; the parameter is the retrieval method of the unstructured data. This solution realizes the storage of unstructured data and descriptive metadata separately, so that the storage medium can be flexibly selected, and the storage medium can be iteratively replaced according to business needs and the development of storage technology, which is also convenient for the migration of unstructured data, and realizes a more efficient and intelligent data management mode.

[0058] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0059] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0060] In several embodiments covered by the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0061] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0062] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0063] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for separate storage and management, applied to separate storage of unstructured data and metadata of unstructured files, characterized in that: include: Use a distributed database as the backend for metadata storage, and manage storage strategies for different models through a unified configuration center; When storing descriptive metadata, each descriptive record corresponds to the metadata of an unstructured data item and is stored in the distributed database in the form of a database record; In the process of storing the descriptive metadata, a storage strategy and acquisition parameters of the unstructured data are added as part of the metadata; the storage strategy is a specific storage method of the unstructured data; and the parameters are a retrieval method of the unstructured data.

2. The method for separation storage and management according to claim 1, characterized in that: The descriptive metadata includes metadata information describing the unstructured data, such as the uploader, upload time, file name, file storage path, file storage strategy, and file acquisition parameters.

3. The method for separation of storage and management according to claim 1, characterized in that: The specific storage method of the unstructured data is: managing the storage strategies of different models through a unified configuration center, and one model corresponds to multiple storage strategies.

4. The method for separation storage and management according to claim 3, characterized in that: The storage strategy is set with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then the corresponding storage medium is matched according to the storage strategy in the record to find the required target file.

5. A device for separate storage and management, applied to separate storage of unstructured data and metadata of unstructured files, characterized in that: include: Deployment unit, configuration unit, extension unit; Deployment unit: used to use the distributed database as the backend for metadata storage and manage storage strategies for different models through a unified configuration center; Configuration unit: used for storing metadata of a non-structured data item corresponding to each descriptive record in the form of a database record in a distributed database when storing descriptive metadata; Extension unit: used to add storage strategy and acquisition parameters of unstructured data as part of metadata during the process of storing the descriptive metadata; the storage strategy is the specific storage method of the unstructured data; the parameter is the retrieval method of the unstructured data.

6. The device for separating storage and management according to claim 5, characterized in that: The descriptive metadata includes metadata information describing the unstructured data, such as the uploader, upload time, file name, file storage path, file storage strategy, and file acquisition parameters.

7. The device for separating storage and management according to claim 5, characterized in that: The specific storage method of the unstructured data is: managing the storage strategies of different models through a unified configuration center, and one model corresponds to multiple storage strategies.

8. The device for separating storage and management according to claim 7, characterized in that: The storage strategy is set with a priority configuration, and when querying, the query is performed in order of priority; the storage strategy of the unstructured data is recorded in the metadata record; when querying, the distributed database is queried according to the unique identifier, a descriptive metadata query is first performed, and then the corresponding storage medium is matched according to the storage strategy in the record to find the required target file.

9. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 4 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: It includes computer-readable instructions. When a computer reads and executes the computer-readable instructions, it implements the method as claimed in any one of claims 1 to 4. There can be multiple readable media, and the multiple readable media can operate independently of each other.

Images