Metadata processing method and electronic equipment
By deploying proxy service pairs and first metadata service pairs in containers on the host device, the problem of scaling up the metadata service architecture is solved, achieving efficient and low-cost expansion to adapt to dynamic business needs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-04
- Publication Date
- 2026-04-03
AI Technical Summary
The existing metadata service architecture is difficult to scale due to physical server limitations, and cannot adapt to the dynamic expansion of business, resulting in high expansion costs and low efficiency.
Deploy proxy service pairs and multiple first metadata service pairs in the container of the host device. Obtain metadata from the second metadata service pair through the proxy service pair and allocate it to the first metadata service pair for storage and management, forming a new metadata cluster architecture.
It enables rapid scaling of metadata services, reduces scaling costs, improves scaling efficiency, and adapts to dynamic business needs.
Smart Images
Figure CN121785707A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage technology, and in particular to a metadata processing method and an electronic device. Background Technology
[0002] In storage / computing systems, asymmetric architecture refers to the physical or logical separation of metadata services and data storage nodes, with each scaling independently. Data nodes can be easily scaled linearly, while scaling metadata services is more complex, requiring adjustments to the mapping relationship between the metadata service and nodes, as well as load balancing.
[0003] A common metadata service architecture is a dual-machine hot standby mechanism, providing a unified service address to the outside world. When the primary machine fails, the standby machine takes over and becomes the new primary machine. Both the primary and standby machines are deployed on physical servers. As the data scales up, the metadata service architecture becomes difficult to expand and transform due to the limitations of physical servers, and cannot better adapt to the dynamic expansion of business. Summary of the Invention
[0004] The main technical problem addressed by this application is to provide a metadata processing method and electronic device that can adjust the metadata cluster architecture so that the adjusted cluster architecture can be better expanded.
[0005] To address the aforementioned technical problems, a first aspect of this application provides a metadata processing method, comprising: in response to receiving a data expansion instruction, adding a proxy service pair and a plurality of first metadata service pairs to a plurality of containers deployed on a first host device; wherein the proxy service pair and the plurality of first metadata service pairs are deployed in different containers, and the proxy service pair and the plurality of first metadata service pairs are used to store and manage metadata; controlling the proxy service pair to acquire metadata from a second metadata service pair, and allocating the acquired metadata as metadata to be allocated to each first metadata service pair, so that each first metadata service pair can store and manage the metadata to be allocated; wherein the second metadata service pair is deployed on a second host device.
[0006] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the method provided in the first aspect.
[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application, in response to a data expansion instruction, adds proxy service pairs and multiple first metadata service pairs to multiple containers deployed on a first host device. The proxy service pairs and multiple first metadata service pairs are deployed in different containers and are used to store and manage metadata. The proxy service pairs are controlled to acquire metadata from second metadata service pairs and distribute the acquired metadata as unallocated metadata to each first metadata service pair, enabling each first metadata service pair to store and manage the unallocated metadata. The second metadata service pairs are deployed on a second host device. Existing technologies directly deploy metadata service pairs on physical servers. When the storage of metadata service pairs is insufficient, a new physical server must be added to add metadata service pairs. Adding a new physical server requires high cost and time, hindering rapid expansion and failing to better adapt to dynamic business expansion. In contrast, this application deploys the multiple newly added first metadata service pairs in multiple containers on a physical server. Since adding containers is convenient, low-cost, and efficient, tools can be used directly to add them, thereby achieving the addition of first metadata service pairs and realizing metadata expansion. Multiple first metadata service pairs form a new metadata cluster architecture. In this cluster architecture, multiple metadata service pairs are managed in a containerized manner, which can better adapt to the dynamic expansion of business. Attached Figure Description
[0008] Figure 1 This is a schematic diagram of the framework structure of one implementation of the existing metadata service architecture; Figure 2 This is a flowchart illustrating one implementation of the metadata processing method provided in this application; Figure 3 This is a flowchart illustrating step S23 of one embodiment of the metadata processing method provided in this application. Figure 4 This is a flowchart illustrating another implementation of the metadata processing method provided in this application; Figure 5 This is a schematic diagram of the framework structure of one implementation of a metadata cluster cloud storage architecture provided in this application; Figure 6 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application; Figure 7 This is a schematic diagram of the framework structure of one embodiment of the computer-readable storage medium provided in this application. Detailed Implementation
[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0010] It should be noted that the embodiments of this application contain descriptions involving "first," "second," etc., which are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.
[0011] In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movement of the components in a specific posture (as shown in the figures). If the specific posture changes, the directional indication will also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0012] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0013] This application primarily relates to distributed object storage, particularly for self-built cloud storage clusters with asymmetric architectures. For example... Figure 1As shown, the existing metadata service architecture is a dual-machine hot standby mechanism, including a pair of secondary metadata services deployed on different physical servers. This pair includes a primary metadata service and a backup metadata service, providing a unified service IP address externally. These secondary metadata services act as mutual backups; when the primary metadata service fails, the backup metadata service takes over as the new host. However, as the cluster data grows, the secondary metadata service becomes a bottleneck due to primary / backup resource limitations, failing to adapt well to dynamic business expansion. Storage nodes are used to store target data, and each storage node can store different target data. The metadata service is used to store and manage the metadata of the target data.
[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the metadata processing method provided in this application. The method includes: S21: In response to receiving the data expansion instruction, add a proxy service pair and a number of first metadata service pairs to the multiple containers deployed on the first host device.
[0015] In one embodiment, to expand and upgrade the existing metadata service architecture, in response to a received data expansion command, proxy service pairs and multiple first metadata service pairs are added to multiple containers deployed on the first host device. Both the proxy service pairs and the multiple first metadata service pairs can be used to store and manage metadata. The metadata refers to the metadata of the target data stored on the storage nodes. In one implementation scenario, this application is applied to the video and image domain, where the video or image captured by each camera is the target data, and the target data is stored in storage nodes. Preferably, each camera corresponds to one storage node, and the target data obtained by multiple cameras is isolated and stored on different storage nodes. The metadata of the target data stored in each storage node is obtained, and the metadata of the target data can be stored in the storage space corresponding to each first metadata service pair.
[0016] In this implementation, the data expansion command can be issued by the user. Specifically, the user can perform relevant operations (such as double-clicking or single-clicking) on a terminal device. The terminal device generates a corresponding operation request based on the obtained operation and sends the operation request to a server or other non-operational devices. The server then responds to the operation request and executes the relevant operation. In other words, in this embodiment, the executing entity is the server or other non-operational devices, and their response to the user's relevant operation refers to responding to the relevant operation request issued by the terminal device connected to the non-operational device based on the user's operation. At this time, the server, in response to the user's data expansion command, adds proxy service pairs and multiple first metadata service pairs to multiple containers.
[0017] In one embodiment, the first host device can be a server or a virtual machine. Multiple containers can be pre-deployed on the first host device. Upon receiving a data scaling instruction, the proxy service pair and the first metadata service pair can be directly deployed in each container. In other embodiments, multiple containers can be deployed after receiving the data scaling instruction, and the proxy service pair and the first metadata service pair can be deployed after the container deployment is complete.
[0018] Both the proxy service pair and the first metadata service pair include corresponding primary and backup services, which form a dual-machine hot standby group. Specifically, the proxy service pair includes a primary proxy service and a backup proxy service, and the first metadata service pair includes a first primary metadata service and a first backup metadata service. The primary and backup services provide the same IP address and can both handle user-initiated requests for metadata processing. The backup service can synchronize data from the primary service in real time, and will handle requests when the primary service is unable to process them.
[0019] In one embodiment, the proxy service pair and the first metadata service pair can be deployed in different containers. Similarly, the primary service and the backup service in a service pair can also be deployed in different containers to isolate faults and prevent a container failure from causing both the primary and backup services to be unable to respond to requests. In other embodiments, the primary service and the backup service can be deployed in the same container. In this embodiment, the service pair includes a proxy service pair and a first metadata service pair.
[0020] In other words, the primary proxy service, backup proxy service, first primary metadata service, and first backup metadata service can be deployed in different containers, with only one service in each container; alternatively, two or more services can be deployed in the same container. Preferably, to isolate faults, only one service can be deployed in a single container.
[0021] S22: Control the proxy service to obtain metadata from the second metadata service pair.
[0022] In one embodiment, the second metadata service is deployed on a second host device, which is also a server or a virtual machine. The second host device may be the same as or different from the first host device. Preferably, the second host device and the first host device are different servers.
[0023] The proxy service can obtain metadata from the second metadata service pair in two ways: either by initiating a request to the second metadata service pair (i.e., the proxy service pair actively obtains metadata), or by having the second metadata service pair send metadata to the proxy service pair (i.e., the proxy service pair passively obtains metadata). Data synchronization can be used to enable the proxy service pair to obtain metadata from the second metadata service pair; the synchronization method can be either full synchronization or incremental synchronization, without specific limitations here.
[0024] In one implementation, after deploying the proxy service pair on the container, the proxy service pair can be pre-started. At this time, the proxy service pair does not handle any business; that is, it does not process requests, which are handled by the second metadata service pair. After the proxy service pair is pre-started, it is controlled to obtain metadata from the second metadata service pair. Specifically, a connection change request is sent to the second metadata service pair, causing the second metadata service pair to respond to the connection change request by disconnecting the second backup metadata service from the second primary metadata service and connecting the backup proxy service to the second primary metadata service to form a new dual-machine hot standby group. One of the two machines in the hot standby group automatically obtains data from the other, thus enabling the backup proxy service to obtain data from the second primary metadata service. In this case, metadata requests are handled by the second primary metadata service. After the backup proxy service completes the data acquisition, a stop request is sent to the second primary metadata service to stop processing metadata requests. The backup proxy service then takes over, and the connection between the primary and backup proxy services is broken. The backup proxy service is then reconnected to the primary proxy service, allowing the primary proxy service to acquire data from the backup proxy service. Once the primary proxy service has acquired the data, it switches to handling metadata requests. Metadata from the second primary metadata service is transferred to the new server where the proxy service resides for subsequent stages, avoiding processing on the original server and preventing system memory overflow caused by duplicate memory.
[0025] S23: The acquired metadata is allocated as metadata to be allocated to each first metadata service pair, so that each first metadata service pair can store and manage the metadata to be allocated.
[0026] In one implementation, this step can be performed by a proxy server, where the metadata in the second metadata service pair includes metadata stored in multiple buckets. Distributing the synchronized metadata to each of the first metadata service pairs includes: using a consistent hashing algorithm to distribute the metadata in each bucket to each of the first metadata service pairs.
[0027] In another embodiment, the allocation weight of each first metadata service pair can be combined during the allocation process of the metadata to be allocated. Specifically, the allocation weight of each first metadata service pair is obtained. The allocation weight is used to determine the number of virtual nodes corresponding to the first metadata service pair on the ring hash space. The consistent hashing algorithm is used to determine the virtual nodes to which the metadata in the storage bucket should be allocated. The metadata in the storage bucket is allocated as the metadata to be allocated to the first metadata service pair corresponding to the virtual node.
[0028] In this process, the metadata is allocated to the first metadata service pair. This can be done by allocating the metadata to the first primary metadata service pair and the first backup metadata service pair within the first metadata service pair, or by allocating the metadata to the first primary metadata service pair within the first metadata service pair, and then synchronizing the metadata from the first primary metadata service pair to the backup metadata service pair.
[0029] The above method, in response to receiving a data expansion command, adds proxy service pairs and multiple first metadata service pairs to multiple containers deployed on the first host device. The proxy service pairs and the multiple first metadata service pairs are deployed in different containers and are used to store and manage metadata. The control proxy service pairs acquire metadata from the second metadata service pairs and distribute the acquired metadata as unallocated metadata to each first metadata service pair, enabling each first metadata service pair to store and manage the unallocated metadata. The second metadata service pairs are deployed on the second host device. The newly added multiple first metadata service pairs migrate the metadata from the second metadata service pairs originally deployed on the physical server to each first metadata service pair. These multiple first metadata service pairs form a new metadata cluster architecture. In this cluster architecture, multiple metadata service pairs are managed using containers, which better adapts to the dynamic expansion of business operations. In short, the above method expands and transforms the primary / standby metadata architecture (i.e., the existing metadata service architecture) into a cloud-native container-based metadata cluster scenario, achieving online transformation across cluster architectures.
[0030] Existing technologies directly deploy metadata service pairs on physical servers. When the storage of metadata service pairs is insufficient, a new physical server must be added to accommodate the new metadata service pairs. However, adding a new physical server is costly and time-consuming, hindering rapid scaling and failing to adapt to dynamic business expansion. This application, on the other hand, deploys multiple new first metadata service pairs in multiple containers on physical servers. Since adding containers is convenient, low-cost, and efficient, it can be done directly using tools, thereby enabling the addition of first metadata service pairs and achieving metadata scaling.
[0031] In one embodiment, the metadata processing method provided in this application can be executed by a first host device. That is, the first host device adds a proxy service pair and a number of first metadata service pairs in multiple containers deployed on itself, and controls the proxy service pairs to obtain metadata from the second metadata service pairs, and allocates the obtained metadata as metadata to be allocated to each first metadata service pair.
[0032] In other embodiments, the method provided in this application can also be executed by other devices, which are different from the first host device. In this case, in response to receiving the data expansion instruction, the other device sends an addition instruction to the first host device to add proxy service pairs and multiple first metadata service pairs in the multiple containers deployed on the first host device; and sends an acquisition instruction to the proxy service pairs to control the proxy service pairs to acquire the metadata in the second metadata service pairs, and allocates the acquired metadata as metadata to be allocated to each first metadata service pair.
[0033] Please see Figure 3 , Figure 3 This is a flowchart illustrating step S23 of an embodiment of the metadata processing method provided in this application, which involves allocating the acquired metadata as metadata to be allocated to each first metadata service pair, including: S331: Obtain the allocation weights of each first metadata service pair.
[0034] The weight allocation is used to determine the number of virtual nodes corresponding to the first metadata service pair on the ring hash space.
[0035] In one implementation, the metadata of the target data captured by each camera is written to a separate storage bucket. In reality, there may be uneven distribution of metadata across some cameras. To address this issue, weights are adjusted based on the amount of original data in the storage bucket to prevent data skew after migration due to uneven data distribution, which could cause excessive load on some first metadata services.
[0036] Specifically, the consistent hashing algorithm is used to pre-determine the first metadata service pair to which the metadata in the bucket should be assigned; a first ring hash space is defined, with all possible hash values distributed on the ring, connected end to end; the first hash value of each first metadata service pair is calculated, and the first hash value is mapped on the ring to obtain a target position in the ring space; the second hash value of the metadata in the bucket is calculated, and the second hash value is mapped on the ring; starting from the position of the second hash value on the ring, the first target position encountered is searched clockwise; the first metadata service pair corresponding to this target position is the first metadata service pair to which the metadata in the bucket should be assigned.
[0037] Based on the relationship between the amount of metadata in the bucket and a preset quantity, a first weight of the bucket is determined. The relationship between the amount of metadata in the bucket and the preset quantity is then assessed. If the amount of metadata is not greater than the preset quantity, a first value is used as the first weight. This first value can be set as needed; in one specific implementation, the first value is 1. If the amount of metadata is greater than the preset quantity, the first weight is determined based on the first value and the ratio of the amount of metadata to the preset quantity. In one implementation, the logarithm of the ratio of the amount of metadata to the preset quantity can be taken to obtain a second value. The sum of the first and second values is used as the first weight. The second value can also be the ratio of the amount of metadata to the preset quantity. In another specific implementation, the difference between the amount of metadata and the preset quantity can be obtained, and the ratio of this difference to the preset quantity can be used as a third value. The sum of the third value and the second value is used as the first weight.
[0038] The central tendency value of the first weights of each bucket assigned to the same first metadata service pair is statistically analyzed to obtain the allocation weight of the first metadata service pair. The first weights of each bucket to be assigned to the same first metadata service pair are obtained, and the central tendency value of each first weight is calculated to obtain the allocation weight of the first metadata service pair. The central tendency value is either the mean or the median.
[0039] S332: Use a consistent hashing algorithm to determine the virtual nodes to which the metadata in the bucket should be assigned.
[0040] In one embodiment, after obtaining the allocation weight of each first metadata service pair, the number of virtual nodes corresponding to each first metadata service pair can be determined. In one embodiment, the number of virtual nodes for each first metadata service pair is a reference number multiplied by the allocation weight. The reference number can be set as needed, for example, 150.
[0041] The overall process of the consistent hashing algorithm is the same as described above. Specifically, first, the ring hash space range (0~2^32-1) is initialized, the hash values of all virtual nodes are calculated, and they are mapped onto the ring. The second hash value of the metadata in the bucket is calculated and modulo 2^32 to obtain a value. This value is mapped onto the ring, and starting from the position of this value, the first virtual node encountered in a clockwise direction is found. This is the virtual node to which the metadata in the bucket should be assigned.
[0042] S333: Assign the metadata in the bucket to the first metadata service pair corresponding to the virtual node.
[0043] Based on the aforementioned known correspondence between the first metadata service and the virtual node, the metadata in the storage bucket (i.e., the metadata to be allocated) will be allocated to the first metadata service pair corresponding to the virtual node.
[0044] This implementation utilizes an improved consistent hashing algorithm, which considers the amount of metadata in each bucket during the metadata allocation process, thus achieving a better balanced distribution of metadata.
[0045] In one embodiment, the proxy service pair distributes the acquired metadata as metadata to be allocated to each of the first metadata service pairs, or the metadata to be allocated can be sent to each of the first metadata service pairs respectively. In other embodiments, each of the first metadata service pairs can also obtain the metadata to be allocated from the proxy service pair.
[0046] Please see Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of the metadata processing method provided in this application, the method comprising: S41: In response to receiving a data expansion instruction, add a proxy service pair and a number of first metadata service pairs to multiple containers deployed on the first host device. S42: Control the proxy service to obtain metadata from the second metadata service pair.
[0047] S43: Assign the acquired metadata as metadata to be allocated to each first metadata service pair.
[0048] For details on the specific implementation of steps S41-S43, please refer to the aforementioned descriptions, which will not be repeated here.
[0049] S44: Control each first metadata service pair to obtain the metadata to be allocated from the proxy service pair.
[0050] In one implementation, for each first metadata service pair, it can directly subscribe to the allocated metadata from the proxy service pair. Alternatively, it can first synchronize the allocated metadata to temporary memory, and then subscribe to the allocated metadata from the temporary memory. It is understood that each first metadata service pair may obtain the allocated metadata from the proxy service pair only for the first primary metadata service within the first metadata service pair.
[0051] When there are many first metadata service pairs, if they all simultaneously obtain the metadata to be allocated from the proxy service pairs, it may affect the operation of the proxy service pairs. Therefore, each first metadata service pair can be divided into different batches, and each first metadata service pair can be controlled to obtain the metadata to be allocated from the proxy service pairs in batches. The number of first metadata service pairs in each batch can be set and adjusted as needed. Specifically, the first metadata service pairs in the current batch are taken as target metadata service pairs; the metadata to be allocated to the target metadata service pairs is obtained and taken as target metadata; the target metadata is synchronized to the temporary memory of the target metadata service pairs, and the target metadata service pairs are controlled to subscribe to the target metadata from the temporary memory; after the target metadata service pairs complete the subscription, the temporary memory is cleared, and the next batch of first metadata service pairs continues to obtain the metadata to be allocated from the proxy service pairs, until all first metadata service pairs have obtained the metadata to be allocated. In other embodiments, before clearing the temporary memory, it can be determined whether the difference between the subscribed metadata and the target metadata is less than a threshold. If so, the temporary memory is cleared; otherwise, it can be resubscribed.
[0052] In one specific implementation, only one first metadata service pair in each batch retrieves data from the proxy service, and the proxy service starts a separate thread to process each metadata pair. The proxy service can extract only the metadata to be allocated to the target metadata service pair.
[0053] In one embodiment, after synchronizing the metadata to be allocated to the temporary memory of the first metadata service pair, synchronized metadata is obtained, and a first verification can be performed on the synchronized metadata. Specifically, a target digest generated before synchronization and a verification digest generated after synchronization are obtained; wherein, the target digest is generated based on the metadata to be allocated before synchronization and the identifier of the first metadata service pair to which the metadata is allocated; the verification digest is generated based on the synchronized metadata and the identifier of the synchronized first metadata service pair; in response to the target digest and the verification digest being consistent, the first verification is determined to have passed; otherwise, if the first verification fails, the step of synchronizing the metadata to be allocated to the temporary memory of the first metadata service pair can be re-executed.
[0054] In one embodiment, after controlling the first metadata service to subscribe to synchronization metadata from temporary memory and obtaining the subscribed metadata, a second verification can be performed on the subscribed metadata. Specifically, target verification information generated based on the synchronization metadata before subscription and information to be verified generated based on the subscribed metadata after subscription are obtained; in response to the target verification information and the information to be verified being consistent, the second verification is determined to have passed.
[0055] In one specific implementation, target verification information can be generated based solely on a portion of the pre-subscription synchronization metadata, and similarly, verification information can be generated based solely on a portion of the post-subscription subscription metadata. For example, several pieces of synchronization metadata and subscription metadata can be sampled at intervals as partial synchronization metadata and partial subscription metadata to calculate the verification value; alternatively, some core fields can be extracted from each piece of synchronization metadata and each piece of subscription metadata to calculate the verification value. By comparing the verification values before and after subscription, it can be determined whether the second verification passes. Understandably, if the verification values before and after subscription are consistent, the second verification passes; otherwise, resubscription is possible. The core fields include, but are not limited to, bucket identifier, metadata file identifier, and shard identifier.
[0056] In the above embodiments, the working modes of the proxy service pair include takeover mode and gateway mode; the step of controlling the proxy service pair to synchronize the metadata in the second metadata service pair and distributing the synchronized metadata to be allocated to each first metadata service pair is performed when the proxy service is in takeover mode.
[0057] After the synchronized metadata to be allocated is distributed to each first metadata service pair, and each first metadata service pair has subscribed to and completed the allocation of its assigned metadata, the working mode of the proxy service pair is adjusted to gateway mode. In gateway mode, the proxy service pair receives multiple requests and distributes each request to the respective first metadata service pair for processing. The proxy service pair can use a consistent hashing algorithm to distribute each request to the respective first metadata service pair.
[0058] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework structure of one implementation of a metadata cluster cloud storage architecture provided in this application.
[0059] The metadata cluster cloud storage architecture includes proxy service pairs and multiple primary metadata service pairs. Each proxy service pair includes a primary proxy service and a backup proxy service. Each primary metadata service pair includes a primary metadata service and a secondary metadata service. Both the primary metadata service pairs and the proxy service pairs run in containers. The proxy service pairs provide a service IP address externally, which is identical to the IP address of the secondary metadata service pair and uses the same port. For clients, the service entry point of the cloud storage cluster remains unchanged, providing a theoretical basis for online modifications. The proxy service pairs perform load balancing based on request information, providing services jointly through multiple primary and backup primary metadata service pairs. These primary metadata service pairs form a unified metadata view. The two services within a metadata service pair are typically distributed across different physical machines to provide server-level disaster recovery and availability. In special scenarios, they can be deployed on a single server, but the disaster recovery level will be downgraded to service level. Each distributed storage node communicates with each primary metadata service pair. The cluster shares the underlying storage and uses partitioning rules to distinguish which primary metadata service pair a data belongs to.
[0060] In the process of expanding and transforming the cloud data storage architecture from the original primary and backup metadata architecture consisting of a second metadata service pair to the cloud-native container-based metadata cluster architecture of this application, three main steps are involved: metadata splitting, metadata migration, and metadata verification.
[0061] Metadata splitting involves splitting the metadata stored in the second metadata service pair. During the splitting process, the first weight of each storage bucket is determined based on the relationship between the amount of metadata in each storage bucket and a preset number. The allocation weight of each first metadata service pair is then determined based on the first weight of the storage bucket. A consistent hashing algorithm is used to determine the virtual nodes to which the metadata in each storage bucket should be allocated. The metadata in the storage bucket is then allocated to the first metadata service pair corresponding to the virtual nodes. The number of virtual nodes corresponding to the first metadata service pair is determined based on the allocation weight.
[0062] Metadata migration involves moving metadata stored in the second metadata service pair to the first metadata service pair. To ensure smooth operation of client read / write services, the migration can be performed in stages. The proxy server's operating mode is set to takeover mode. In takeover mode, the proxy service pair functions identically to the second metadata service pair and can interact normally with it. Metadata from the second metadata service pair is synchronized to the proxy service pair. After synchronization, the second metadata service pair stops processing metadata, and the proxy server pair takes over. Each first metadata service pair then retrieves data from the proxy server pair. Once data retrieval is complete, each first metadata service pair becomes ready. Finally, the proxy server's operating mode is switched to gateway mode. The proxy service pair no longer processes metadata but performs consistent hash calculations on requests and routes them to the corresponding first metadata service pair for metadata processing. The first metadata service pairs no longer retrieve metadata from the proxy server.
[0063] Metadata verification is performed by the first metadata service to verify the data before and after it is retrieved from the proxy server, in order to avoid inaccurate data after splitting.
[0064] Please see Figure 6 , Figure 6 This is a schematic diagram of the framework structure of one embodiment of the electronic device provided in this application.
[0065] The electronic device 60 includes a memory 61 and a processor 62 coupled to each other. The memory 61 stores program instructions, and the processor 62 executes the program instructions stored in the memory 61 to implement the steps of any of the above-described method embodiments. In a specific implementation scenario, the electronic device 60 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 60 may also include mobile devices such as laptops and tablets, which are not limited here.
[0066] Specifically, processor 62 controls itself and memory 61 to implement the steps of any of the above-described method embodiments. Processor 62 may also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 62 may be implemented using integrated circuit chips.
[0067] Please see Figure 7 , Figure 7 This is a schematic diagram of the framework structure of one embodiment of the computer-readable storage medium provided in this application.
[0068] The computer-readable storage medium 70 stores program instructions 71, which, when executed by a processor, are used to implement the steps of any of the above-described method implementations.
[0069] The computer-readable storage medium 70 can specifically be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store computer programs. Alternatively, it can be a server that stores the computer program, which can send the stored computer program to other devices for execution or can also run the stored computer program itself.
[0070] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.
[0071] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.
Claims
1. A metadata processing method, characterized in that, include: In response to receiving a data expansion instruction, a proxy service pair and a plurality of first metadata service pairs are added to multiple containers deployed on the first host device; wherein, the proxy service pair and the plurality of first metadata service pairs are deployed in different containers, and the proxy service pair and the plurality of first metadata service pairs are used to store and manage metadata; The proxy service is controlled to obtain metadata from the second metadata service pair and allocate the obtained metadata as metadata to be allocated to each of the first metadata service pairs, so that each of the first metadata service pairs can store and manage the metadata to be allocated; wherein, the second metadata service pair is deployed on the second host device.
2. The method according to claim 1, characterized in that, The metadata in the second metadata service pair includes metadata stored in multiple buckets; The step of allocating the acquired metadata as metadata to be allocated to each of the first metadata service pairs includes: Obtain the allocation weight of each of the first metadata service pairs; wherein, the allocation weight is used to determine the number of virtual nodes corresponding to the first metadata service pairs on the ring hash space; The consistent hashing algorithm is used to determine which virtual node the metadata in the storage bucket should be assigned to; The metadata in the storage bucket is assigned to the first metadata service pair corresponding to the virtual node.
3. The method according to claim 2, characterized in that, The step of obtaining the allocation weights for each of the first metadata service pairs includes: The consistent hashing algorithm is used to pre-determine the first metadata service pair to which the metadata in the storage bucket should be assigned; Based on the relationship between the amount of metadata in the storage bucket and a preset amount, a first weight of the storage bucket is determined; The central tendency value is statistically analyzed for the first weights of each bucket assigned to the same first metadata service pair to obtain the allocation weight of the first metadata service pair.
4. The method according to claim 1, characterized in that, After allocating the acquired metadata as metadata to be allocated to each of the first metadata service pairs, the method further includes: Control each of the first metadata service pairs to obtain the metadata to be allocated from the proxy service pair.
5. The method according to claim 4, characterized in that, The control of each of the first metadata service pairs to obtain the metadata to be allocated from the proxy service pair includes: Each of the first metadata service pairs is divided into different batches, and the first metadata service pairs in the current batch are used as target metadata service pairs; Obtain the metadata to be allocated that should be assigned to the target metadata service pair, as the target metadata; The target metadata is synchronized to the temporary memory of the target metadata service pair, and the target metadata service pair is controlled to subscribe to the target metadata from the temporary memory; After the target metadata service pair completes its subscription, the temporary memory is cleared, and the process continues to complete the step of the next batch of the first metadata service pairs obtaining the metadata to be allocated from the proxy service pair, until all the first metadata service pairs have obtained the metadata to be allocated.
6. The method according to claim 4, characterized in that, The control of each of the first metadata service pairs to obtain the metadata to be allocated from the proxy service pair includes: The metadata to be allocated is synchronized to the temporary memory of the first metadata service pair to obtain synchronized metadata; The first metadata service is controlled to subscribe to the synchronization metadata from the temporary memory; wherein, after synchronizing the metadata to be allocated to the temporary memory of the first metadata service pair, a first verification is performed on the synchronization metadata, and / or, after subscribing to the synchronization metadata from the temporary memory, a second verification is performed on the subscribed metadata.
7. The method according to claim 6, characterized in that, After synchronizing the metadata to be allocated to the temporary memory of the first metadata service pair, the first verification of the synchronized metadata includes: Obtain the target digest generated before synchronization and the verification digest generated after synchronization; wherein, the target digest is generated based on the metadata to be allocated before synchronization and the identifier of the first metadata service pair to which the metadata is allocated; the verification digest is generated based on the synchronized metadata and the identifier of the first metadata service pair to which it is synchronized; If the target digest and the digest to be verified are consistent, the first verification is determined to be successful. And / or, the step of performing a second verification on the subscribed metadata after subscribing to the synchronization metadata from the temporary memory includes: Obtain the target verification information generated based on the synchronization metadata before subscription and the verification information generated based on the subscription metadata after subscription; In response to the target verification information and the information to be verified being consistent, it is determined that the second verification has passed.
8. The method according to claim 1, characterized in that, The proxy service pair, the first metadata service pair, and the second metadata service pair all include a corresponding primary service and a backup service. The primary service and the backup service form a dual-machine hot standby group. The backup service synchronizes data from the primary service in real time, and when the primary service is unable to process the request, the backup service processes the request. The control of the proxy service to obtain metadata from the second metadata service pair includes: Send a connection change request to the second metadata service pair, so that in response to the connection change request, the second metadata service pair disconnects the connection between the second backup metadata service and the second primary metadata service in the second metadata service pair, and connects the backup proxy service in the proxy service pair to the second primary metadata service to form a new dual-machine hot standby group, so that the backup proxy service can obtain data from the second primary metadata service; After the backup proxy service completes the acquisition, a stop request is sent to the second metadata service to cause the second primary metadata service to stop processing the metadata request. The backup proxy service then processes the metadata request, and the connection between the second primary metadata service and the backup proxy service is disconnected. The backup proxy service and the primary proxy service are connected so that the primary proxy service obtains data from the backup proxy service. After the primary proxy service has obtained the data, the process switches to the primary proxy service to handle the metadata request.
9. The method according to claim 1, characterized in that, The proxy service pair operates in two modes: takeover mode and gateway mode. The step of controlling the proxy service to obtain metadata from the second metadata service pair and allocating the obtained metadata as metadata to be allocated to each of the first metadata service pairs is performed when the proxy service pair is in the takeover mode. And / or, after allocating the acquired metadata as metadata to be allocated to each of the first metadata service pairs, the method further includes: The working mode of the proxy service pair is adjusted to gateway mode. In gateway mode, the proxy service pair receives multiple requests and assigns each request to each of the first metadata service pairs for processing.
10. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the method according to any one of claims 1-9.