Data management method and device based on object storage and storage medium
By generating a directory structure and displaying it visually, the problem of complex object storage data management is solved, achieving the effect of simplifying data management and reducing costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2025-12-16
- Publication Date
- 2026-05-12
AI Technical Summary
Object storage has complex data management, requiring frequent data modification and deletion, which increases management difficulty and cost.
By receiving the object data to be stored and the object name, a directory structure is generated and displayed visually on the data management page. The storage address of the object data is determined based on the object name, and management operations are performed through the directory structure, reducing direct operations on the object storage end.
It reduces the difficulty and cost of data management, improves the efficiency of data management, conforms to user habits, and simplifies data migration and renaming operations.
Smart Images
Figure CN122018781A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data management method, device and storage medium based on object storage. Background Technology
[0002] Object storage, also known as object-based storage, places each object at the same level in a flat address space called a storage pool. It is suitable for storing unstructured data, such as video data and image data.
[0003] However, data management in object storage is very inconvenient. Generally, it is necessary to delete the object that needs to be modified and then rewrite it, which increases the difficulty and cost of data management. Summary of the Invention
[0004] To address the aforementioned technical problems, this application provides at least one data management method, device, and storage medium based on object storage.
[0005] The first aspect of this application provides a data management method based on object storage. The method includes: receiving object data to be stored and object names of the object data; sending the object data and object names to an object storage terminal for associated storage; determining the directory structure corresponding to the object data based on the object names; associating the object names as storage addresses under the directory structure; visually displaying the directory structure on a data management page; and managing the object data associated with the storage addresses under the directory structure in the object storage terminal based on the data management operations received on the data management page.
[0006] In one embodiment, the directory structure consists of multiple directory levels; determining the directory structure corresponding to the object data based on the object name, and associating the object name as a storage address to the directory structure, includes: determining the upper-level directory corresponding to the object data based on the object name; determining whether the upper-level directory exists; if it exists, associating the object name as a storage address to the upper-level directory; if it does not exist, creating the upper-level directory corresponding to the object data, and associating the object name as a storage address to the upper-level directory.
[0007] In one embodiment, after determining the directory structure corresponding to the object data based on the object name and associating the object name as a storage address to the directory structure, the method further includes: generating a file directory tree based on the parent directory corresponding to each directory level and the parent directory corresponding to the object data; and generating metadata corresponding to each directory level and metadata corresponding to the object data; and visually displaying the directory structure on the data management page, including: visually displaying the directory structure based on the file directory tree, the metadata corresponding to each directory level, and the metadata corresponding to the object data.
[0008] In one embodiment, the directory structure consists of multiple directory levels, and the metadata corresponding to the object data contains the upper-level directory of the object data. Based on the data management operation received by the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: in response to the directory move operation performed by the user on the object data, determining the directory level to be moved to obtain the target directory level; determining the new upper-level directory corresponding to the object data based on the target directory level; updating the upper-level directory in the metadata corresponding to the object data to the new upper-level directory, so as to move the storage address corresponding to the object data to the target directory level.
[0009] In one embodiment, the metadata corresponding to the object data contains the data name of the object data; based on the data management operation received by the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: in response to the user's renaming operation on the data name of the object data, obtaining the new name of the input object data; and updating the data name in the metadata corresponding to the object data to the new name.
[0010] In one embodiment, the management interface for object data is displayed on the data management page; based on the data management operations received on the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: responding to the user's query operation, deletion operation, or modification operation performed on the object data management interface, obtaining the storage address corresponding to the object data, and obtaining the target address; generating a data query instruction, data deletion instruction, or data modification instruction carrying the target address; and sending the data query instruction, data deletion instruction, or data modification instruction to the object storage terminal so that the object storage terminal can query, delete, or modify object data whose object name matches the target address.
[0011] In one embodiment, the application is applied to any node in a distributed cluster. The node is equipped with a service receiver and a service processor: the service receiver is used to receive object data to be stored and the object name of the object data; the service processor is used to determine the directory structure corresponding to the object data based on the object name, associate the object name as a storage address with the directory structure, calculate the metadata corresponding to the object data, send the directory structure and metadata to the metadata storage end for storage, and query the object storage end that matches the object data and send the object storage end to the service receiver; the service receiver sends the object data and object name to the object storage end for associated storage based on the object storage end sent by the service processor.
[0012] In one embodiment, the metadata storage terminal is implemented based on a distributed cluster, and the method further includes: when the number of nodes in the metadata storage terminal is a single node, creating multiple database instances on the node; wherein, the database instances are used to store directory structure and metadata; when the number of nodes in the metadata storage terminal is multiple, creating a primary database instance and a backup database instance on different nodes.
[0013] A second aspect of this application provides a data management device based on object storage. The device includes: a data receiving module, used to receive object data to be stored and object names of the object data, and send the object data and object names to an object storage terminal for associated storage; a directory determination module, used to determine the directory structure corresponding to the object data based on the object names, and associate the object names as storage addresses under the directory structure; and a data management module, used to visualize the directory structure on a data management page, and manage the object data associated with the storage addresses under the directory structure in the object storage terminal based on data management operations received on the data management page.
[0014] A third aspect of this application provides an electronic device including a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the above-described object-based data management method.
[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described object-based data management method.
[0016] The above solution receives the object data to be stored and its object name, then sends both to the object storage terminal for associated storage. Based on the object name, it determines the corresponding directory structure and associates the object name as a storage address within that directory structure. The directory structure is then visualized on the data management page. Based on data management operations received on the data management page, the solution manages the object data associated with the storage address in the directory structure within the object storage terminal. This directory structure presentation not only conforms to user habits and facilitates displaying the actual data to the user, but also allows users to manage any object data without actually manipulating the stored object data; they only need to modify the directory structure, reducing the difficulty and cost of data management. Furthermore, the storage address ensures that the corresponding associated object data can be correctly located from the object storage terminal, guaranteeing the normal use of the object data.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0019] Figure 1 This is a flowchart illustrating an exemplary embodiment of the data management method based on object storage, as shown in this application. Figure 2 This is a metadata diagram illustrating an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating directory information storage as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the storage of object data in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the business layer shown in an exemplary embodiment of this application; Figure 6 This is a schematic diagram illustrating a deployed database instance, as shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating a deployed database instance, as shown in another exemplary embodiment of this application; Figure 8 This is a block diagram illustrating an object storage-based data management apparatus in an exemplary embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 10 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation
[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0023] The data management method based on object storage provided in the embodiments of this application is described below.
[0024] The data management method based on object storage provided in this application can be executed by a server in each step. For example, it can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Of course, the executing entity of each step can also be other electronic devices with computing capabilities, and this application does not limit this.
[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the data management method based on object storage, as shown in the following example. Figure 1 As shown, the data management method based on object storage includes at least steps S110 to S130, which are described in detail below: Step S110: Receive the object data to be stored and the object name of the object data, and send the object data and object name to the object storage terminal for associated storage.
[0026] The object data includes, but is not limited to, one or more of the following: video data, image data, text data, and audio data.
[0027] The object name is a unique identifier for an object in a storage bucket. The object name can be generated according to preset rules.
[0028] For example, if the object data is view data captured by a camera, the object name of the view data can be composed of the camera identifier (cameraId), the channel identifier (channelId), the date the view data was generated (date), and the view data name (fileName). The object name would then be represented as " / cameraId / channelId / date / fileName".
[0029] For example, the object name of any view data is represented as " / LX_001 / 1 / 2025 / 01 / 04 / 202501011430.jpg", which means that the view data with the name 202501011430.jpg was collected by the first channel of camera identifier LX_001 on January 4, 2025.
[0030] The object data and object name are sent to the object storage terminal so that the object storage terminal can associate and store the received object data and object name. The object data can then be retrieved by querying the object name.
[0031] The object storage side uses object storage to store data.
[0032] Step S120: Determine the directory structure corresponding to the object data based on the object name, and associate the object name as the storage address with the directory structure.
[0033] The object name is parsed to determine the directory structure of the object data.
[0034] For example, the hierarchical structure of an object name can be broken down to obtain a directory structure. For instance, if the object name is " / cameraId / channelId / date / fileName", it can be broken down into directory 1: cameraId, directory 2: channelId, directory 3: date, and the final file fileName is placed in directory 3: date. This means that the object name is used as a storage address, and that storage address is associated with directory 3: date.
[0035] For example, the directory structure corresponding to the object data can be determined by combining the object name and pre-defined directory determination rules.
[0036] For example, the system receives a directory editing operation performed by the user, obtains the directory determination rules input by the user based on the directory editing operation, and the directory determination rules define the association between each directory and object name. If it is detected that the object name of the object data satisfies the association relationship of any directory, the system generates the directory structure corresponding to the object data based on the directories that satisfy the association relationship.
[0037] For example, the directory determination rules stipulate that: Specify the camera directory for the specified area: the cameraId in the object name belongs to the listed IDs in the preset identifier list; Date Directory 1: The date in the object name is January; Date Index 2: The "date" in the object name is for February; ... (and so on). Date Index 12: The date in the object name is December; Image directory: The fileName in the object name has the suffix .jpg; Video directory: The fileName in the object name has the suffix .dav.
[0038] For example, if the object name of the data to be stored is " / LX_001 / 1 / 2025 / 01 / 04 / 202501011430.jpg", then by matching the directory rules mentioned above, if cameraId=LX_001 belongs to the listed IDs in the preset identifier list, then a directory for the specified area camera is created; if date=January 4, 2025, then a date directory 1 is created; and if fileName has the .jpg suffix, then an image directory is created. Then, based on the hierarchical relationship between the directories, the final directory structure is: specified area camera directory / date directory 1 / image directory. The object name is used as the storage address, and this storage address is associated with the image directory.
[0039] In the above embodiments, the method of generating the directory structure can be flexibly determined according to user needs, thereby improving the flexibility of the directory structure and ensuring that the generated directory structure meets user habits.
[0040] Alternatively, the object name can be directly stored as the storage address in the directory structure; or an identifier for the object data can be generated, stored in the directory structure, and the identifier, storage address, etc., can be stored as metadata in the database. The storage address corresponding to the object data can be retrieved by the object data identifier. This application does not limit the association method between the storage address and the directory structure.
[0041] Step S130: Visualize the directory structure on the data management page, and manage the object data associated with the storage address under the directory structure in the object storage terminal based on the data management operation received on the data management page.
[0042] A data management page is provided for users, which displays the directory structure visually in a way that aligns with user habits. Users can perform data management operations on the data management page to manage object data associated with storage addresses under the directory structure in the object storage terminal.
[0043] The data management operations include, but are not limited to, querying, modifying, and deleting object data, or querying, modifying, and deleting the metadata corresponding to object data under the directory structure, or querying, modifying, and deleting the directory structure, etc. This application does not limit these operations.
[0044] It should be noted that the final directory structure obtained in this application differs from the object names stored in the object storage. When storing data using object storage, although the object names themselves have a hierarchy, the underlying object storage uses a flat key-value pair format, where the key is the object name and the value is the data content of the object itself (such as specific view data). In the object storage, the object name merely serves as an index for retrieving object data, and the specific data content is stored under the index.
[0045] In object storage, the flat storage method makes data management difficult and costly. For example, if object data in object storage needs to be moved or renamed, the object data to be modified is usually deleted first, and then the new object data is rewritten. It is also not convenient to show the actual data to the user.
[0046] The directory structure generated by this application based on the object name is a file directory system independent of the object storage system. Its directory structure is not used to store specific data content, but is associated with the object name (i.e., storage address). The presentation of the directory structure not only conforms to user habits and makes it easy to display the actual data situation to the user, but also allows the user to perform operations such as moving or renaming any object data without operating on the actual object data stored on the object storage terminal. Only the directory structure or the metadata of the object data in the directory structure needs to be modified, which reduces the difficulty and cost of data management.
[0047] The following describes some embodiments of this application in detail.
[0048] In some implementations, the directory structure consists of multiple directory levels; step S120 determines the directory structure corresponding to the object data based on the object name, and stores the object name as a storage address in the directory structure, including the following steps S121 to S124.
[0049] Step S121: Determine the parent directory corresponding to the object data based on the object name.
[0050] Determine the parent directory corresponding to the object data based on the hierarchical relationship reflected in the object name or the hierarchical relationship between predefined directories.
[0051] For example, the object name is represented as " / cameraId / channelId / date / fileName", and its corresponding parent directories include directory 1: cameraId, directory 2: channelId, and directory 3: date.
[0052] Step S122: Determine if the parent directory exists. If it does not exist, proceed to step S123; if it exists, proceed to step S124.
[0053] Step S123: Create the parent directory corresponding to the object data, and then execute step S124; Step S124: Associate the object name as a storage address with the parent directory.
[0054] Associate the object name as a storage address with the parent directory.
[0055] In some implementations, after determining the directory structure corresponding to the object data based on the object name and storing the object name as the storage address in the directory structure, the method further includes: generating a file directory tree based on the parent directory corresponding to each directory level and the parent directory corresponding to the object data; and generating metadata corresponding to each directory level and metadata corresponding to the object data; and visually displaying the directory structure on the data management page, including: visually displaying the directory structure based on the file directory tree, the metadata corresponding to each directory level, and the metadata corresponding to the object data.
[0056] For example, a unique identifier is generated for each directory level and for each object data. The file directory tree is generated by combining the hierarchical relationship between each directory level and each object data with the unique identifier.
[0057] For example, a global identifier generation algorithm can be used to generate globally unique identifiers.
[0058] Specifically, the data name `obj_name` of the object data is obtained. `obj_name` can be extracted from the object name or be user-defined. After `obj_name` is encoded according to a preset encoding rule, an identifier generator is used to generate the object data's identifier (OID). Taking hash encoding of `obj_name` and the snowflake algorithm for calculating the OID as an example, OID = snowflake(hash(obj_name)). In addition, directories are treated as special objects. For example, cameraId in the example above is an object without an actual database. The directory identifier can be calculated using the same method as the identifier used to generate object data, such as OID=snowflake(hash(cameraId)).
[0059] Using the above method, the OID corresponding to each directory level and each object data can be calculated.
[0060] Then, based on the parent directory corresponding to each directory level and the parent directory corresponding to the object data, the directory hierarchy is obtained. Combining the directory hierarchy and the OID corresponding to each directory level and the object data, a file directory tree is generated.
[0061] In addition, metadata corresponding to each directory level and metadata corresponding to the object data are generated. The metadata corresponding to the directory level includes, but is not limited to, one or more of the following: identifier, identifier of the parent directory, creation time, update time, name, size, etc. The metadata corresponding to the object data includes, but is not limited to, one or more of the following: storage address, identifier, identifier of the parent directory, creation time, update time, data name, size, etc.
[0062] The specific content of the metadata can be flexibly set according to the actual application scenario, and this application does not limit it.
[0063] For examples, please refer to Figure 2 , Figure 2 This is a metadata diagram illustrating an exemplary embodiment of this application, as shown below. Figure 2 The metadata types shown generate metadata for each directory level and metadata for the object data. Of course, in practical applications, other metadata types can be selected to generate metadata for each directory level and metadata for the object data; this application does not limit this selection.
[0064] Please refer to the above embodiments. Figure 3 , Figure 3 This is a schematic diagram illustrating directory information storage as shown in an exemplary embodiment of this application, such as... Figure 3 As shown, it includes the following steps S301 to S305.
[0065] Step S301: Determine the parent directory corresponding to the object data based on the object name; Step S302: Determine if the parent directory exists. If it does not exist, proceed to step S303; if it exists, proceed to steps S304 to S305. Step S303: Create the parent directory corresponding to the object data; Step S304: Generate the metadata of the object data, which contains the identifier and storage address of the object data; Step S305: Store the metadata of the object data in the metadata database and place the identifier of the object data in the upper-level directory.
[0066] The data management page displays the directory structure visually based on the file directory tree, the metadata corresponding to each directory level, and the metadata corresponding to each object data.
[0067] For example, the system determines the directory level and object data to be displayed based on the file directory tree. Then, based on the metadata corresponding to the directory level to be displayed, it obtains the name of that directory level and visualizes it. Similarly, based on the metadata corresponding to the object data to be displayed, it obtains the name of that object data and visualizes it. Of course, other metadata can also be displayed, and this application does not limit this.
[0068] Based on the displayed data management page, it can receive data management operations performed by the user on the data management page.
[0069] The following section provides specific examples of data management operations.
[0070] In some implementations, the management interface for object data is displayed on the data management page; based on the data management operations received on the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: responding to the user's query operation, deletion operation, or modification operation performed on the object data management interface, obtaining the storage address corresponding to the object data, and obtaining the target address; generating a data query instruction, data deletion instruction, or data modification instruction carrying the target address; and sending the data query instruction, data deletion instruction, or data modification instruction to the object storage terminal so that the object storage terminal can query, delete, or modify object data whose object name matches the target address.
[0071] By using the storage address under the directory structure, the associated object data in the object storage terminal can be queried, deleted, or modified, thereby realizing the management of the actual object data stored in the object storage terminal.
[0072] In some implementations, the directory structure consists of multiple directory levels, and the metadata corresponding to the object data contains the upper-level directory of the object data. Based on the data management operations received from the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: in response to the user's directory move operation on the object data, determining the directory level to be moved to obtain the target directory level; determining the new upper-level directory corresponding to the object data based on the target directory level; and updating the upper-level directory in the metadata corresponding to the object data to the new upper-level directory, so as to move the storage address corresponding to the object data to the target directory level.
[0073] If you need to move object data in the directory structure, you only need to modify the parent directory corresponding to the object data.
[0074] For example, the parent directory identifier indicates the parent directory corresponding to the object data. In response to the user's directory move operation on the object data, the target directory level is determined to be moved to. Based on the target directory level, the new parent directory corresponding to the object data is determined, the identifier of the new parent directory is obtained, and the identifier of the parent directory in the metadata corresponding to the object data is updated to the identifier of the new parent directory.
[0075] If the directory structure of this application is not adopted, when moving object data in the object storage, it is necessary to delete the object data to be migrated from the original storage bucket and rewrite the object data to be migrated into a new storage bucket. This operation is complicated and inconvenient for user management. However, when data migration is required, this application only needs to modify the parent directory corresponding to the object data. The actual object data in the object storage does not need to be moved, and the data migration effect can be shown to the user side, which is convenient for user-side data management.
[0076] In some implementations, the metadata corresponding to the object data contains the data name of the object data; based on the data management operation received from the data management page, the object data associated with the storage address under the directory structure in the object storage terminal is managed, including: in response to the user's renaming operation on the data name of the object data, obtaining the new name of the input object data; and updating the data name in the metadata corresponding to the object data to the new name.
[0077] If you need to rename the data name of the object data in the directory structure, you only need to modify the data name in the metadata corresponding to the object data.
[0078] If the directory structure of this application is not adopted, when renaming object data in the object storage, since the object name is the unique identifier of the object in the storage bucket, it is also necessary to delete the object data to be migrated from the storage bucket and rewrite the renamed object data into the storage bucket. This operation is complicated and inconvenient for user management. However, when data migration is required, this application only needs to modify the metadata corresponding to the object data. It does not need to delete and rewrite the actual object data in the object storage, and the data renaming effect can be reflected to the user side, which is convenient for user-side data management.
[0079] It should be noted that the above data management operations are merely illustrative examples, and more data management operations may exist in actual application scenarios, which are not limited in this application.
[0080] In some implementations, the directory structure and related metadata are stored in a metadata store, which houses a metadata database. For example, see [link to relevant documentation]. Figure 4 , Figure 4This is a schematic diagram illustrating stored object data in an exemplary embodiment of this application, such as... Figure 4 As shown, the business layer receives the object data to be stored and the object name of the object data. Then, it calculates the directory structure and related metadata, sends the directory structure and related metadata to the metadata storage end for storage, and sends the object data and the object name of the object data to the object storage end for storage.
[0081] The business layer, metadata storage, and object storage can be implemented using a distributed cluster, or other methods can be used; this application does not limit this.
[0082] The business layer, metadata storage, and object storage can be deployed independently, or multiple of them can be deployed together. This application does not limit this.
[0083] For example, please see Figure 5 , Figure 5 This is a schematic diagram of the service layer shown in an exemplary embodiment of this application, such as... Figure 5 As shown, the object storage-based data management method of this application is applied to any node in a distributed cluster. The node is equipped with a service receiver and a service processor. The camera generates view data, which is used as object data to be stored. The object name corresponding to the object data is obtained based on the camera identifier, channel identifier, generation time, etc. corresponding to the view data.
[0084] The business receiver is used to receive the object data to be stored and the object name of the object data.
[0085] The business processor is used to determine the directory structure corresponding to object data based on the object name, store the object name as the storage address in the directory structure, calculate the metadata corresponding to the object data, send the directory structure and metadata to the metadata storage terminal for storage, and query the object storage terminal that matches the object data and send the object storage terminal information to the business receiver. Of course, the business processor can also be used to calculate the metadata of the directory structure and send it to the metadata storage terminal for storage.
[0086] The service receiver sends the object data and object name to the object storage terminal for associated storage based on the object sent by the service processor.
[0087] Optionally, the service receiver and service processor are stateless and can be expanded according to actual needs. Because they are stateless, there is no need to consider the situation of other service receivers and service processors when expanding, which facilitates expansion.
[0088] Optionally, the service receiver stores the address of the storage gateway. The storage gateway is used to send object data and the object name of the object data to the object storage end. It can determine the load of each storage gateway and perform load balancing. For example, when the storage gateway is on the same node as the service receiver and the service processor, this storage gateway is selected first to reduce data transmission latency.
[0089] In some implementation methods, the metadata storage is based on a distributed cluster. The method also includes: when the number of nodes in the metadata storage is a single node, multiple database instances are created on the node; wherein, the database instances are used to store the directory structure and the metadata corresponding to the directory structure; when the number of nodes in the metadata storage is multiple, a primary database instance and a backup database instance are created on different nodes. Specifically, in a single-node deployment, n database instances are created simultaneously on that node. The value of n can be calculated during deployment based on estimated business demand. The data to be stored is evenly distributed across the n database instances, so that each database instance shares 1 / n of the actual business load. Metadata is placed on these database instances using a consistent hashing algorithm.
[0090] In a multi-node deployment, each database instance is deployed in a primary-standby configuration. A high-availability Virtual Internet Protocol Address (VIP) is requested for each primary-standby database instance pair. The business layer accesses the database instance through the VIP. When the primary database instance fails, the standby database instance can take over the business operations within seconds. Therefore, if n databases need to be deployed, at least 2n database instances plus n VIPs are required.
[0091] The principle of deploying primary and backup database instances is that the primary database instance and the backup database instance are located on different nodes, and the database instances on each node are evenly distributed.
[0092] For example, the following rules can be used to determine the distribution nodes of the primary and backup database instances: The primary database instance deployment node number is calculated as follows: (Primary DB Serial Number - 1) % Number of Nodes. Here, the primary DB serial number identifies the primary database instance, the number of nodes refers to the total number of nodes in the distributed cluster, and % is the modulo operator, returning the remainder of the division of two numbers. This formula is used to evenly distribute the primary database instance across the nodes.
[0093] Calculate the deployment node sequence number of the backup database instance: Scenario 1: If the number of standby database instances is greater than or equal to the number of nodes, then the standby database instance deployment node sequence number = (standby DB sequence number) % number of nodes. Here, the standby DB sequence number is used to identify the standby database instance.
[0094] For example, please see Figure 6 , Figure 6 This is a schematic diagram illustrating a deployed database instance as shown in an exemplary embodiment of this application, such as... Figure 6 As shown, taking 4 sets of primary database instances and standby database instances, and 4 nodes as an example, based on the above deployment method, node 1 (node number 0) deploys primary database instance 1 and standby database instance 4; node 2 (node number 1) deploys primary database instance 2 and standby database instance 1; node 3 (node number 2) deploys primary database instance 3 and standby database instance 2; and node 4 (node number 3) deploys primary database instance 4 and standby database instance 3.
[0095] Scenario 2: If the number of standby database instances is less than the number of nodes: then the standby database instance deployment node sequence number = (standby DB sequence number + total number of DBs - 1) % number of nodes. Where, the total number of DBs is the total number of standby database instances.
[0096] For example, please see Figure 7 , Figure 7 This is a schematic diagram illustrating a deployed database instance, as shown in another exemplary embodiment of this application. Figure 7 As shown, taking two sets of primary database instances and standby database instances, and four nodes as an example, based on the above deployment method, node 1 (node number 0) deploys primary database instance 1; node 2 (node number 1) deploys primary database instance 2; node 3 (node number 2) deploys standby database instance 1; and node 4 (node number 3) deploys standby database instance 2.
[0097] When scaling up nodes on the metadata storage end, the decision on whether database instances need to be migrated is based on the following condition: the difference in the number of database instances between all nodes is not greater than a preset threshold (e.g., the preset threshold is 1). If it is greater than the preset threshold, the database instances to be migrated out and migrated in are recalculated according to the database distribution calculation rules mentioned above.
[0098] The data directory of a database instance is usually stored directly in the physical disk file system. When migrating a database instance, the data directory stored in the physical disk file is copied directly to other nodes. This approach can avoid the need for subsequent consistent hashing algorithms to re-migrate old data and can quickly complete the database migration.
[0099] When migrating a database instance, the physical resources that the database instance can actually use during the migration process are limited based on the load and / or actual business status of the node to be migrated to.
[0100] For example, if the load on the node to be migrated is high, the amount of resources that the database instance to be migrated can use during data migration is reduced; if the load on the node to be migrated is low, the amount of resources that the database instance to be migrated can use during data migration is increased.
[0101] Figure 8 This is a block diagram illustrating an object storage-based data management apparatus as shown in an exemplary embodiment of this application. Figure 8 As shown, the exemplary object storage-based data management device 800 includes: The data receiving module 810 is used to receive the object data to be stored and the object name of the object data, and send the object data and object name to the object storage terminal for associated storage. The directory determination module 820 is used to determine the directory structure corresponding to the object data based on the object name, and associate the object name as a storage address with the directory structure. The data management module 830 is used to visualize the directory structure on the data management page and manage the object data associated with the storage address under the directory structure in the object storage terminal based on the data management operations received on the data management page.
[0102] It should be noted that the object storage-based data management device and the object storage-based data management method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments and will not be repeated here. In practical applications, the object storage-based data management device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.
[0103] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 900 includes a memory 901 and a processor 902. The processor 902 executes program instructions stored in the memory 901 to implement the steps in any of the above-described embodiments of the object-based data management method. In a specific implementation scenario, the electronic device 900 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 900 may also include mobile devices such as laptops and tablets, without limitation.
[0104] Specifically, processor 902 controls itself and memory 901 to implement the steps in any of the above-described embodiments of the object-based data management method. Processor 902 can also be referred to as a Central Processing Unit (CPU). Processor 902 may be an integrated circuit chip with signal processing capabilities. Processor 902 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 902 can be implemented using integrated circuit chips.
[0105] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 1000 stores program instructions 1010 that can be executed by a processor. The program instructions 1010 are used to implement the steps in any of the above-described embodiments of the object-based data management method.
[0106] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0107] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0108] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A data management method based on object storage, characterized in that, The method includes: Receive the object data to be stored and the object name of the object data, and send the object data and the object name to the object storage terminal for associated storage; Based on the object name, determine the directory structure corresponding to the object data, and associate the object name as a storage address with the directory structure; The directory structure is visualized on the data management page, and the object data associated with the storage address under the directory structure in the object storage terminal is managed based on the data management operations received on the data management page.
2. The method according to claim 1, characterized in that, The directory structure consists of multiple directory levels; determining the directory structure corresponding to the object data based on the object name, and associating the object name as a storage address with the directory structure, includes: Determine the parent directory corresponding to the object data based on the object name; Determine if the parent directory exists. If it exists, associate the object name as the storage address with the parent directory. If it does not exist, create the parent directory corresponding to the object data and associate the object name as the storage address with the parent directory.
3. The method according to claim 2, characterized in that, After determining the directory structure corresponding to the object data based on the object name and associating the object name as a storage address to the directory structure, the method further includes: Based on the parent directory corresponding to each directory level and the parent directory corresponding to the object data, a file directory tree is generated; and metadata corresponding to each directory level and metadata corresponding to the object data are generated. The visualization of the directory structure on the data management page includes: Based on the file directory tree, the metadata corresponding to each directory level, and the metadata corresponding to the object data, the directory structure is visualized.
4. The method according to claim 1, characterized in that, The directory structure consists of multiple directory levels, and the metadata corresponding to the object data contains the upper-level directory of the object data; the management of object data associated with the storage address under the directory structure in the object storage terminal based on the data management operation received by the data management page includes: In response to a user's directory move operation on the object data, the target directory level is determined and the directory level to be moved to is obtained. Determine the new parent directory corresponding to the object data based on the target directory hierarchy; The upper-level directory in the metadata corresponding to the object data is updated to the new upper-level directory, so as to move the storage address corresponding to the object data to the target directory level.
5. The method according to claim 1, characterized in that, The metadata corresponding to the object data contains the data name of the object data; the management of object data associated with the storage address under the directory structure in the object storage terminal based on the data management operation received by the data management page includes: In response to a user's renaming operation on the data name of the object data, obtain the new name of the input object data; Update the data name in the metadata corresponding to the object data to the new name.
6. The method according to claim 1, characterized in that, The management interface for the object data is displayed on the data management page; the management of object data associated with the storage address under the directory structure in the object storage terminal based on the data management operations received on the data management page includes: In response to a user's query, deletion, or modification operation performed on the management interface of the object data, the storage address corresponding to the object data is obtained, and the target address is obtained; Generate a data query instruction, a data deletion instruction, or a data modification instruction carrying the target address; Send the data query instruction, the data deletion instruction, or the data modification instruction to the object storage terminal so that the object storage terminal can query, delete, or modify object data whose object name matches the target address.
7. The method according to claim 1, characterized in that, This can be applied to any node in a distributed cluster, where the node is equipped with a service receiver and a service processor. The service receiver is used to receive the object data to be stored and the object name of the object data; The service processor is configured to determine the directory structure corresponding to the object data based on the object name, associate the object name as a storage address with the directory structure, calculate the metadata corresponding to the object data, send the directory structure and the metadata to the metadata storage end for storage, and query the object storage end that matches the object data and send the object storage end to the service receiver. The service receiver sends the object data and the object name to the object storage terminal for associated storage based on the information sent by the service processor.
8. The method according to claim 7, characterized in that, The metadata storage terminal is implemented based on a distributed cluster, and the method further includes: When the number of nodes in the metadata storage terminal is a single node, multiple database instances are created on that node; wherein, the database instances are used to store the directory structure and the metadata; When there are multiple nodes in the metadata storage terminal, a primary database instance and a backup database instance are created on different nodes.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.