Metadata processing method and device, storage medium and electronic device

Abstract

The present application discloses a metadata processing method and device, storage medium and electronic device. The method includes: performing a sampling mark operation on a container set, determining container set information, and determining a target container based on the container set information, wherein the container set information includes the migration mark, allowed migration period and container load of each container in the container set, the target container represents a container in the container set whose container load meets the preset load condition, and the allowed migration period corresponding to the target container is the target period; adding the target container to the queue to be migrated; in response to the system time reaching the target period, migrating the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated, wherein the target server is different from the server deployed by the target host. The present application solves the technical problem that the storage and access overhead of metadata is too large, resulting in high expansion performance loss and low expansion reliability.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a metadata processing method and device, a storage medium, and an electronic device. Background Art

[0002] At present, the video acquisition business faces the storage demand for a large amount of video data. Each camera generates a large amount of video stream data. Cloud storage has become a mainstream storage solution due to its efficient metadata management and retrieval technology. However, a single cloud storage has problems such as insufficient performance scalability and load imbalance, and metadata cluster technology has emerged. As the amount of data stored increases, metadata information is also becoming more and more massive, which may cause system performance degradation or failure, becoming a bottleneck in the cloud storage system. When the cluster business volume reaches a certain scale, there will be technical problems such as high expansion performance loss and low expansion reliability of the metadata storage system.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a metadata processing method and device, a storage medium, and an electronic device to at least solve the technical problem of high expansion performance loss and low expansion reliability due to excessive metadata storage and access overhead.

[0005] According to one aspect of an embodiment of the present application, a metadata processing method is provided, comprising: performing a sampling mark operation on a container set, determining container set information, and determining a target container according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host machine load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is a target period; adding the target container to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated in the target period; in response to the system time reaching the target period, migrating the target metadata stored in the target container to a target server in the queue to be migrated in the form of blocks, wherein the target server is different from the server deployed by the target host machine.

[0006] According to another aspect of an embodiment of the present application, a metadata processing device is further provided, comprising: a sampling module, configured to perform a sampling mark operation on a container set, determine container set information, and determine a target container according to the container set information, wherein the container set information comprises a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host machine load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is a target period; an adding module, configured to add the target container to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated in the target period; and a migration module, configured to migrate the target metadata stored in the target container to a target server in the form of blocks according to the queue to be migrated in response to the system time reaching the target period, wherein the target server is different from the server deployed by the target host machine.

[0007] Optionally, the device is used to perform a sampling marking operation on a container set in the following manner to determine container set information, and determine a target container based on the container set information: whenever the system time reaches a sampling time point, traverse and sample the container load corresponding to each container in the container set; whenever a traversal sampling ends, generate multiple marking sub-files, wherein one marking sub-file among the multiple marking sub-files is used to represent the container load corresponding to a container during a traversal sampling process; in response to the end of sampling in the current sampling cycle, obtain multiple groups of marking sub-files, merge and mark the multiple groups of marking sub-files according to container identifiers to obtain the container set information, and determine the target container based on the container set information, wherein the current sampling cycle includes multiple sampling time points, and the time period with the lowest container load corresponding to each container in the current sampling cycle is set as the allowed migration period.

[0008] Optionally, the device is used to traverse and sample the container load corresponding to each container in the container set whenever the system time reaches a sampling time point in the following manner: whenever the system time reaches a sampling time point, traverse and sample each container in the container set in the following manner, wherein the container sampled each time is regarded as the current container, and the current container is deployed on the current host machine: obtain the container load index of the current container and the host load index corresponding to the current host machine, wherein the number of indicators of the container load index and the host load index is the same; when the number of indicators is multiple, determine the target proportional relationship based on the value of the host load index; determine the container load corresponding to the current container based on the container load index and the target proportional relationship.

[0009] Optionally, the device is used to obtain multiple groups of marked subfiles in response to the end of sampling in the current sampling period, merge and mark the multiple groups of marked subfiles according to container identifiers, obtain the container set information, and determine the target container according to the container set information: in response to the end of sampling in the current sampling period, obtain the multiple groups of marked subfiles and the load level of the expansion server allowed to migrate; determine the number of containers to be migrated based on the load level; arrange the container set in descending order according to the container load according to the container set information to obtain a target list; filter out containers corresponding to the number of containers to be migrated from the target list, add a first migration mark, and add a second migration mark to the containers in the container set that do not have the first migration mark added, to generate the container set information, wherein the first migration mark indicates that the corresponding container is allowed to migrate, and the second migration mark indicates that the corresponding container is prohibited from migrating; determine the container in the container set information that has the first migration mark and the allowed migration period is the target period as the target container.

[0010] Optionally, the device is used to perform a sampling marking operation on a container set, determine container set information, and determine a target container based on the container set information in the following manner: perform a sampling marking operation on a container set, determine container set information, and determine a target container based on the container set information, wherein the container set information also includes a number of migration blocks set for each container in the container set; the device is used to migrate target metadata stored in the target container to a target server in the form of blocks according to the queue to be migrated in response to the system time reaching the target time period in the following manner: perform a block operation on the target metadata according to the number of migration blocks in response to the system time reaching the target time period to obtain a group of data blocks; set a corresponding migration time label for each data block in the group of data blocks according to the length of the target time period and the number of migration blocks; and migrate the group of data blocks to the target server in sequence according to the migration time labels.

[0011] Optionally, the device is used to migrate the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated in response to the system time reaching the target time period in the following manner: in response to the system time reaching the target time period, taking out the first target container from the queue to be migrated, and setting a first deletion mark for the first target container, wherein the first deletion mark is used to indicate that the first target container is a container to be deleted in the first host machine, the target container includes the first target container, and the target host machine includes the first host machine; in response to the first deletion mark being set for the first target container, starting a migration process of migrating the metadata stored in the first target container to the target server; in the migration process After completion, determining a first service status of a first service corresponding to the first target container; if the first service status indicates that the first service is running normally, determining that the metadata stored in the first target container has been migrated to the target server; if the first service status indicates that the first service is running abnormally, performing a verification operation on the metadata stored in the first target container in the target server to obtain a target verification result; if the target verification result indicates an abnormality, restarting the migration process of migrating the metadata stored in the first target container to the target server, and if the number of times the target verification result indicates an abnormality meets a preset number condition, deleting the first deletion mark and restoring the status of the first target container.

[0012] Optionally, the device is further used to: when it is determined that the metadata stored in the first target container has been migrated to the target server, or when the number of abnormalities indicated by the target verification result satisfies a preset number condition, determine a second target container from the queue to be migrated, and set a second deletion mark for the second target container, wherein the second deletion mark is used to indicate that the second target container is a container to be deleted in the second host machine, the target container includes the second target container, and the target host machine includes the second host machine; in response to the second deletion mark being set for the second target container, start a migration process of migrating the metadata stored in the second target container to the target server.

[0013] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned metadata processing method when running.

[0014] According to another aspect of the embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above metadata processing method.

[0015] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the metadata processing method through the computer program.

[0016] In an embodiment of the present application, a sampling mark operation is performed on a container set to determine container set information, and a target container is determined based on the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host machine load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, the allowed migration period corresponding to the target container is the target period, the target container is added to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated in the target period, and in response to the system time reaching the target period, the target metadata stored in the target container is migrated in the form of blocks according to the queue to be migrated. Migrate to the target server, where the target server is deployed on a different server from the target host. By designing a metadata dynamic expansion technology, a containerized method is adopted to adaptively adjust the queue of containers to be migrated and the migration plan according to system load information to realize online dynamic expansion of metadata. In addition, by using a dynamic sampling marking algorithm and a time-sharing and block migration scheduling strategy, the metadata container to be migrated is accurately screened and migrated, so that the cloud storage metadata cluster can dynamically and evenly expand the metadata without business perception, and at the same time has automatic fault detection and rapid rollback capabilities, further improving the fault tolerance of the expansion process, achieving the purpose of further improving the fault tolerance of the expansion process, thereby achieving the technical effect of reducing expansion overhead and improving expansion efficiency, and further solving the technical problem of high expansion performance loss and low expansion reliability due to excessive metadata storage and access overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application environment of an optional metadata processing method according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of an optional metadata processing method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional metadata processing method according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0027] Fig.10 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application;

[0028] Fig.11 is a schematic structural diagram of an optional metadata processing device according to an embodiment of the present application;

[0029] Fig.12 is a schematic diagram of the structure of an optional metadata processing product according to an embodiment of the present application;

[0030] Fig.13 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretation:

[0034] Metadata is data that describes data and is usually used to provide information such as the context, structure, content, and source of the data.

[0035] The present application is described below in conjunction with embodiments:

[0036] According to one aspect of an embodiment of the present application, a metadata processing method is provided. Optionally, in this embodiment, the metadata processing method can be applied to Figure 1 In the hardware environment composed of the server 101 and the terminal device 103 shown in FIG. Figure 1As shown, the server 101 is connected to the terminal 103 via a network, and can be used to provide services for the terminal device or an application 107 installed on the terminal device. The application can be a video application, an instant messaging application, a browser application, an educational application, a game application, etc. A database 105 may be set up on the server or independently of the server to provide data storage services for the server 101, for example, a game data storage server. The above-mentioned network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication. The terminal device 103 may be a terminal configured with an application, and may include but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart TV, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a virtual reality (Virtual Reality, VR for short) terminal, an augmented reality (Augmented Reality, AR for short) terminal, a mixed reality (Mixed Reality, MR for short) terminal and other computer devices. The above-mentioned server may be a single server, or a server cluster consisting of multiple servers, or a cloud server.

[0037] Combination Figure 1 As shown, the above-mentioned metadata processing method can be executed by an electronic device, which can be a terminal device or a server. The above-mentioned metadata processing method can be implemented by the terminal device or the server respectively, or by the terminal device and the server together.

[0038] The above is only an example and is not specifically limited in this embodiment.

[0039] Optionally, as an optional implementation, as Figure 2 As shown, the method for processing the above metadata includes:

[0040] S202, performing a sampling mark operation on the container set, determining the container set information, and determining the target container according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is the target period;

[0041] Optionally, in this embodiment, the above-mentioned container set includes but is not limited to a set of independent environments for storing and managing metadata, which can be deployed on different host machines. The above-mentioned sampling and marking operation includes but is not limited to the process of collecting and marking information on the container set in order to effectively manage and migrate containers. The above-mentioned migration mark includes but is not limited to an identifier indicating whether the container needs to be migrated. If it is marked as "need to migrate", it means that the container should be migrated to another host machine. The above-mentioned allowed migration period includes but is not limited to a time period within a specific time range during which the container can be migrated. The above-mentioned container load includes but is not limited to the workload or resource usage carried by the container. The above-mentioned host includes but is not limited to a physical server or virtual machine on which the container runs. The above-mentioned target container includes but is not limited to a container that meets the preset load conditions, and its load is associated with the load of the target host. The above-mentioned preset load conditions include but are not limited to container load standards set in the migration operation to optimize resource allocation. The above-mentioned target period includes but is not limited to the expected time period for the migration operation.

[0042] In an exemplary embodiment, it may be implemented including but not limited to the following process:

[0043] Data collection: Information sampling is performed on the container set, and the migration mark, allowed migration period and container load of each container are recorded to ensure the accuracy and timeliness of data collection and facilitate subsequent processing.

[0044] Migration tag setting: Set the migration tag of each container according to the current state of the container and the system load. The migration tag of the container determines whether it can be migrated within the allowed migration period.

[0045] Target container determination: Based on the container load and the host load, select the container that meets the preset load conditions as the target container, and confirm the match between the target container's allowed migration period and the target period.

[0046] S204, adding the target container to a queue to be migrated, where the queue to be migrated represents a queue of containers that need to be migrated during the target period;

[0047] Optionally, in this embodiment, metadata can be but is not limited to being stored in a containerized environment. As business needs change, container migration becomes a common operation. The target container refers to a container that needs to be migrated within a specific time period. The reasons for migration may include resource reallocation, load balancing, fault recovery, etc. The above-mentioned queue to be migrated is a data structure for storing containers that are planned to be migrated in a certain period of time in the future. The queue ensures that the migration tasks can be effectively managed and processed within the specified time.

[0048] In an exemplary embodiment, the step of adding the target container to the queue to be migrated includes but is not limited to the following:

[0049] Identify the target container: First, identify the target container that needs to be migrated. This can be done by the container's ID, name, or other identifier.

[0050] Verify migration conditions: Ensure that the target container meets the migration conditions, such as resource usage and dependencies.

[0051] Update the queue to be migrated: insert the target container information into the queue to be migrated. You can use the first-in-first-out (FIFO) or other suitable queue management strategies.

[0052] Confirm that the addition was successful: Finally, check the queue to be migrated to confirm that the target container has been successfully added.

[0053] S206 , in response to the system time reaching the target time period, the target metadata stored in the target container is migrated to the target server in the form of blocks according to the queue to be migrated, wherein the target server is different from the server deployed by the target host.

[0054] Optionally, in this embodiment, the above “system time reaches the target time period” means that the system starts to execute a specific task at a preset time node. This mechanism is usually implemented through a timer or a scheduler. When the system detects that the current time matches the set target time, it triggers the subsequent migration operation.

[0055] Specifically, the system time may include but is not limited to the time when the computer system runs internally. The response may include but is not limited to the reaction of the system to a specific event. The target period may include but is not limited to a preset time interval during which the system needs to perform certain operations.

[0056] In an exemplary embodiment, the data migration process usually has a "queue to be migrated", which contains the data items that need to be migrated. In the target metadata migration, these data items are organized in blocks for efficient transmission and storage. The queue to be migrated may include but is not limited to the metadata stored in the container to be transferred, usually represented as a data structure that can store the containers to be processed in sequence. The target metadata is stored in a structure called a "target container". The target container is designed to facilitate data access and management. During the migration process, the data in the target container will be divided into multiple small blocks to facilitate gradual migration to the target server. Block migration may include but is not limited to transferring the data in the target container in small blocks, rather than migrating all the data at once. This method can reduce the burden on the system and improve the efficiency and stability of the migration process. The target server refers to the server that receives the migration data, which is a different entity from the target host. The target host refers to the physical or virtual machine that runs the target container, which is responsible for managing and storing data.

[0057] For example, suppose a large e-commerce platform performs system maintenance between 12:00 and 1:00 at night and decides to migrate metadata to a new database server. When the system time reaches 12:00, the migration operation is triggered.

[0058] S1, responding to system time: The system detects that the current time is 12 o'clock and starts the migration process.

[0059] S2, queue to be migrated: the platform puts the corresponding metadata into the queue to be migrated.

[0060] S3, target container: Metadata is stored in the target container, ready for migration.

[0061] S4, block migration: The system divides the metadata into 100 small blocks and sends them to the target server one by one.

[0062] S5, target server: The new database server receives the data and completes the storage.

[0063] For example, Figure 3 is a schematic diagram of an optional metadata processing method according to an embodiment of the present application, such as Figure 3 As shown, the present application may include but is not limited to a cloud storage metadata cluster system, which is implemented based on containerization, wherein: Figure 1On the left are 6 pairs of metadata management nodes, deployed in active-standby mode, based on an asymmetric cloud storage architecture. efs is the general name for all metadata services in the management node. Through containerization technology, the services in the 6 pairs of metadata management nodes are encapsulated in 12 containers and deployed on a pair of nodes. A new active-standby proxy service container (proxy) is added as the external unified entrance of the entire cloud storage system to achieve the management of all metadata service containers and improve server resource utilization and system scalability. Subsequent technologies and solutions are all implemented based on this metadata cluster architecture.

[0064] For example, Figure 4 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, such as Figure 4 As shown in the figure, based on the above architecture and metadata expansion requirements, an online metadata dynamic expansion technology is designed, and a dynamic sampling and marking algorithm is added to accurately screen out all container queues to be migrated; the time-sharing and block migration scheduling strategy is used to achieve seamless migration and load balancing of metadata; and the automatic fault detection algorithm is used to ensure system stability and abnormal rollback during the migration process.

[0065] Through this embodiment, a sampling mark operation is performed on a container set to determine container set information, and a target container is determined according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host machine load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, the allowed migration period corresponding to the target container is the target period, the target container is added to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated in the target period, and in response to the system time reaching the target period, the target metadata stored in the target container is migrated in the form of blocks according to the queue to be migrated. The target server is different from the server deployed by the target host. By designing a metadata dynamic expansion technology, a containerized method is adopted to adaptively adjust the queue of containers to be migrated and the migration plan according to system load information to realize online dynamic expansion of metadata. In addition, by using a dynamic sampling marking algorithm and a time-sharing and block migration scheduling strategy, the metadata container to be migrated is accurately screened and migrated, so that the cloud storage metadata cluster can dynamically and evenly expand the metadata without business perception, and at the same time has automatic fault detection and rapid rollback capabilities, further improving the fault tolerance of the expansion process, achieving the purpose of further improving the fault tolerance of the expansion process, thereby achieving the technical effect of reducing expansion overhead and improving expansion efficiency, and then solving the technical problem of high performance loss and low expansion reliability in the expansion process due to excessive metadata storage and access overhead.

[0066] As an optional solution, performing a sampling marking operation on the container set, determining the container set information, and determining the target container according to the container set information includes:

[0067] Whenever the system time reaches the sampling time point, the container load corresponding to each container in the sampling container set is traversed;

[0068] Whenever a traversal sampling is completed, a plurality of marking sub-files are generated, wherein one marking sub-file among the plurality of marking sub-files is used to represent a container load corresponding to a container in a traversal sampling process;

[0069] In response to the end of sampling in the current sampling cycle, multiple groups of marked sub-files are obtained, the multiple groups of marked sub-files are merged and marked according to the container identifier, the container set information is obtained, and the target container is determined according to the container set information, wherein the current sampling cycle includes multiple sampling time points, and the time period with the lowest container load corresponding to each container in the current sampling cycle is set as the migration allowed period.

[0070] Optionally, in this embodiment, the above sampling time point can be pre-set as a fixed time point, or can be set to an adaptive setting based on the traffic of the business. During the operation of the system, in order to regularly monitor the load status of each container, the system will set specific sampling time points to ensure that the load information of each container can be accurately obtained at these time points. When the system time reaches these sampling time points, the system will automatically trigger a load sampling.

[0071] It should be noted that at each sampling, the system will traverse the preset container set and obtain the current load of each container. Container load usually includes indicators such as CPU usage, memory usage, network traffic, etc. Through this information, you can fully understand the operating status of the container. For example, the system may record the following data at each sampling:

[0072] Container A: CPU usage 70%;

[0073] Container B: Memory usage 50%;

[0074] Container C: network traffic 300 KB / s.

[0075] Furthermore, each time a load sampling is completed, the system will generate multiple marker sub-files. Each marker sub-file corresponds to the load information of a container during a traversal sampling process. These sub-files store the container's load data and its timestamp to facilitate subsequent data analysis and processing. For example, the marker sub-file may contain the following information:

[0076] Load data of container A and its sampling time.

[0077] Load data of container B and its sampling time.

[0078] Then, every time the current sampling period ends, the system will obtain multiple sets of marked sub-files and merge them according to the container identifier. By merging these marked sub-files, a complete set of container information can be generated. This process ensures the integrity and consistency of the data and provides a reliable data basis for subsequent decision-making.

[0079] Finally, after obtaining the container set information, the system will analyze the load of each container in the current sampling period, especially the period with the lowest load. This period is defined as the "allowed migration period" to provide a basis for the dynamic migration of containers. By migrating containers with lower loads, the system can optimize resource usage and improve overall performance.

[0080] By implementing the above sampling and migration strategies, the system can monitor container load changes in real time and make dynamic adjustments based on actual conditions. The technical effects of this process are mainly reflected in the following aspects:

[0081] Resource optimization: By migrating containers with lower loads, the system can effectively utilize resources, reduce redundancy, and improve resource utilization.

[0082] Load balancing: Load balancing is achieved among multiple containers, avoiding overloading of a single container and improving system stability.

[0083] Data-driven decision-making: Through regular sampling and data analysis, the system can make reasonable decisions based on facts, improving management efficiency.

[0084] For example, Figure 5 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, such as Figure 5 As shown in the figure, it is performed through a scheduled task. The default trigger period is half an hour and the number of times is 48 times (covering all time periods of the day). It can also be set manually. When sampling, the load, time and other information of each container will be recorded in the tag subfile, and each sampling point will record a copy. After the sampling number is reached, the final tag file of the container is obtained by merging the information of each subfile.

[0085] As an optional solution, whenever the system time reaches the sampling time point, the container load corresponding to each container in the sampling container set is traversed, including:

[0086] Whenever the system time reaches the sampling time point, each container in the sampling container set is traversed in the following way, where the container sampled each time is regarded as the current container, and the current container is deployed on the current host:

[0087] Obtain the container load index of the current container and the host load index corresponding to the current host, wherein the container load index and the host load index have the same number of indicators;

[0088] When there are multiple indicators, the target ratio is determined based on the values ​​of the host load indicators.

[0089] The container load corresponding to the current container is determined based on the container load index and the target ratio relationship.

[0090] Optionally, in this embodiment, whenever the system time reaches a predetermined sampling time point, the system will start to traverse the container set. These containers are deployed on their respective host machines, with the purpose of collecting the load indicators of the current container and the load indicators of its host machine. The core of this process is to obtain relevant indicators for performance analysis and resource optimization.

[0091] It should be noted that when traversing each container, you first need to obtain the load indicators of the current container. These indicators usually include CPU usage, memory usage, network traffic, etc. At the same time, you also need to obtain the load indicators of the current host, including the CPU, memory, and IO usage of the host. It is worth noting that the number of container load indicators and host load indicators must be the same to ensure the effectiveness of subsequent analysis.

[0092] Furthermore, when there are multiple indicators, a target ratio can be determined based on the value of the host load indicator. The purpose of setting the target ratio is to dynamically adjust the container load indicator when the host load is high so that it can better adapt to the current resource conditions. For example, if the CPU usage of the host reaches 80%, and the CPU usage of a container is only 30%, the target ratio can be set to gradually increase the CPU usage of the container to achieve a more balanced load distribution.

[0093] Finally, based on the relationship between the container load index and the target ratio, the container load corresponding to the current container can be calculated. This calculation process involves multiple steps, including analysis of the host load, application of the target ratio, and final container load adjustment. Through this series of calculations, dynamic adjustment of container load can be achieved, making it more reasonable in resource usage.

[0094] Through the above method, the load balancing between the container and the host can be effectively achieved, and the overall performance and resource utilization of the system can be improved.

[0095] By way of example, the above algorithm can be used to accurately screen out all metadata containers with high migration priority in the environment, and automatically add them to the migration queue according to the information in the tag file. Figure 6 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application. The container load calculation formula is as follows: Figure 6 As shown. To balance the original host load to the greatest extent, the three indicators of CPU usage, memory usage, and network IO usage can be used as the main measurement parameters, but are not limited to. According to the load of different hosts, the weight of each indicator is dynamically set. All container loads in the same host use the same calculation formula, and the host performance bottleneck is effectively migrated to improve the overall performance of the system.

[0096] As an optional solution, in response to the end of sampling in the current sampling period, multiple groups of marked subfiles are obtained, the multiple groups of marked subfiles are merged and marked according to the container identifiers to obtain container set information, and the target container is determined according to the container set information, including:

[0097] In response to the end of sampling in the current sampling period, obtaining multiple groups of marked sub-files and load levels of expansion servers that are allowed to be migrated;

[0098] Determine the number of containers to migrate based on the load level;

[0099] According to the container set information, the container set is sorted in descending order according to the container load to obtain a target list;

[0100] Filter out containers corresponding to the number of containers to be migrated from the target list, add a first migration mark, and add a second migration mark to the containers in the container set that are not added with the first migration mark, to generate container set information, wherein the first migration mark indicates that the corresponding container is allowed to migrate, and the second migration mark indicates that the corresponding container is prohibited from migrating;

[0101] A container having a first migration flag in the container set information and whose allowed migration period is the target period is determined as a target container.

[0102] Optionally, in this embodiment, at the end of the current sampling period, the system will obtain multiple groups of marked sub-files, which contain load level information of the expansion server. The acquisition of the load level can be achieved through detection tools, which collect the resource usage of the server in real time, including indicators such as CPU, memory, and I / O. By analyzing these data, the load status of the server can be determined, thereby providing a basis for subsequent migration decisions.

[0103] It should be noted that based on the load level data obtained, the system needs to determine the number of containers to be migrated. This process usually involves setting a threshold for the load level. When the server load exceeds the set threshold, the system will identify the number of containers that need to be migrated, thereby reducing the burden on the server and optimizing resource allocation.

[0104] Next, the current container set information will be processed. First, the containers are sorted in descending order according to their load to obtain a target list. Containers with high loads will be given priority for migration because they consume more resources on the current server. Therefore, this sorting method can effectively identify the containers that need to be migrated the most.

[0105] Then, after obtaining the target list, the system will filter out containers corresponding to the number of containers to be migrated, and add the first migration mark to these containers, indicating that they are allowed to migrate. At the same time, for unmarked containers, the system will add the second migration mark to indicate that these containers are prohibited from migration. Through this marking mechanism, the system can clearly manage the migration status of the container and provide guidance for subsequent migration operations.

[0106] Finally, the system will filter out the target containers that meet the conditions from the marked container information. Specifically, only when the container has the first migration mark and its allowed migration period is consistent with the target period, it will be determined as the target container. This process ensures the timeliness and effectiveness of the migration operation, allowing the system to maintain stable operation under high load conditions.

[0107] Through the implementation of the above steps, the container migration strategy can not only effectively reduce the server load and improve resource utilization, but also ensure the high availability and performance stability of the system to a certain extent. In short, through scientific load analysis and intelligent decision-making, the container migration strategy not only improves the efficiency of resource management in the cloud computing environment, but also provides solid technical support for enterprises in digital transformation.

[0108] For example, Figure 7 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, wherein the content of the container migration mark file is as follows: Figure 7 As shown in the figure, Migrate.tag is the merged total tag file, and each container saves a copy; the is_migrate field is the migration flag, which is calculated based on the number of metadata servers to be expanded and the load level, 0 and 1 respectively indicate no migration and migration; the migrate_period field is the migration time, and the time with lower load in each sub-tag file is selected as the migration time, 0-5 corresponds to 6 equal time periods in 24 hours; fragment_num is the number of blocks during data migration, and the default division unit is 100MB; load_level is the load of the container, and the value is obtained according to the dynamic load calculation formula. This migration tag file serves as important basis information for subsequent container migration.

[0109] As an optional solution,

[0110] Performing a sampling and marking operation on the container set, determining container set information, and determining a target container according to the container set information, including: performing a sampling and marking operation on the container set, determining container set information, and determining the target container according to the container set information, wherein the container set information also includes a number of migration blocks set for each container in the container set;

[0111] In response to the system time reaching the target time period, the target metadata stored in the target container is migrated to the target server in the form of blocks according to the queue to be migrated, including: in response to the system time reaching the target time period, a block operation is performed on the target metadata according to the number of migration blocks to obtain a group of data blocks; a corresponding migration time label is set for each data block in the group of data blocks according to the length of the target time period and the number of migration blocks; and a group of data blocks are migrated to the target server in sequence according to the migration time labels.

[0112] Optionally, in this embodiment, the above-mentioned “container set information” includes but is not limited to:

[0113] Container ID: A string that uniquely identifies a container.

[0114] Resource usage: such as CPU, memory, and storage usage.

[0115] Running state: Whether the container is currently running, stopped, or crashed.

[0116] Migration chunks: The number of chunks set for each container. This is usually used to split the data in the container into multiple parts when migration is required to facilitate data transmission and management.

[0117] Exemplarily, in the container set, sampling is first performed, that is, resource usage, operating status, and number of migration blocks of these containers are evaluated, so as to obtain the performance characteristics of the overall container.

[0118] Next, after sampling is complete, the selected containers are marked. The marking process involves adding label information to each container, such as "high load", "low load" or "to be migrated". In this way, it is possible to quickly identify which containers need special attention or optimization.

[0119] Finally, after the marking is completed, the target container can be determined based on the information of the container set. Containers with high resource usage may need to be optimized or expanded.

[0120] By performing sampling and marking operations on container collections, not only can containers be effectively identified and managed, but also significant technical effects can be achieved in terms of resource usage, migration process, and fault recovery. This process is of great significance in the management and optimization of containerized applications, and provides solid support for the cloud computing and microservice architecture of enterprises. With the continuous development of container technology, this method will become more mature and efficient, providing greater flexibility and scalability for future application scenarios.

[0121] Optionally, in this embodiment, the block operation refers to dividing the target metadata according to a preset number of migration blocks to generate a set of data blocks. For example, suppose a database containing 1,000 records needs to be migrated to a new server, and the number of migration blocks is set to 10. Through the block operation, the data will be divided into 10 parts, each containing 100 records. This approach can not only reduce the amount of data for a single migration, but also reduce network load and server pressure, and improve migration efficiency.

[0122] Exemplarily, after completing the data segmentation, it is necessary to set a migration time label for each data segment. The setting of the time label is calculated based on the length of the target period and the number of migration segments. For example, if the target period is 60 minutes and the number of migration segments is 10, the migration time label of each data segment will be set to 6 minutes. Specifically, the first group of data segments will be migrated within the first 6 minutes of the target period, the second group of data segments will be migrated between the 6th and 12th minutes, and so on. This setting of time labels ensures that the data segments are carried out in an orderly manner during the migration process, avoiding system crashes or data loss caused by migrating multiple large data volumes at the same time.

[0123] According to the above-mentioned block and time label settings, the migration implementation process can be carried out in the following steps:

[0124] Block generation: Divide the target metadata into blocks according to the number of blocks to form data blocks.

[0125] Time label allocation: Set a corresponding migration time label for each data block based on the target period and number of blocks.

[0126] Data migration execution: Migrate each data block to the target server in sequence according to the order of time labels.

[0127] For example, the first group of 100 records will be migrated between the 1st and 6th minutes, the second group between the 6th and 12th minutes, and so on, until all data chunks have been migrated.

[0128] Through the analysis of the block operation and time label setting in the data migration process, we can see the significant advantages of this method in terms of improving efficiency, reducing risks and facilitating management. As the amount of data continues to increase, a reasonable and effective migration strategy will provide strong support for the information management of enterprises. In the future, with the further development of technology, the methods and tools of data migration will continue to improve, providing more intelligent solutions for various businesses.

[0129] In an exemplary embodiment, the present application inserts the tasks to be migrated into the migration queue according to the migration mark file of each container, that is, the container with the is_migrate value of 1. Figure 8 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, such as Figure 8 As shown in the figure, the migration queue is implemented based on the Map structure, where the Key is the migration time migrate_period and the Value is the queue of the container migration task. The container migration tasks with the same migration time will be added to the same queue and sorted according to the load. The high-load container will be traversed and migrated first. The migration scheduling algorithm will match the queue in the corresponding Key value according to the current time.

[0130] In another exemplary embodiment, the time-sharing and block-sharing migration scheduling strategy involved in the present application can be, but is not limited to, using data blocks as the minimum migration granularity, and performing migration in sequence within a specified time period while ensuring physical fault tolerance. Fig. 9 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, such as Fig. 9 As shown in the figure, the left side shows the containers to be migrated within the same migration time, and the right side shows the metadata server with expanded capacity. Different migration time labels are automatically assigned to each data block in the same container to avoid excessive transmission pressure affecting the normal business of the system. The status of the primary and backup services will be detected and switched in advance before the container is migrated to ensure that the metadata service in the container is the backup machine during migration. The containers where the same pair of primary and backup metadata services are located will ensure that they are not on the same server after migration to ensure physical fault tolerance. Different containers in the same migration queue will also be preferentially allocated to different servers to avoid excessive pressure on server resources in some time periods due to similar business models, ensure load balancing of the system after migration, and improve system stability and reliability.

[0131] As an optional solution, in response to the system time reaching the target period, the target metadata stored in the target container is migrated to the target server in the form of blocks according to the queue to be migrated, including:

[0132] In response to the system time reaching the target time period, taking out a first target container from the queue to be migrated, and setting a first deletion mark for the first target container, wherein the first deletion mark is used to indicate that the first target container is a container to be deleted in the first host machine, the target container includes the first target container, and the target host machine includes the first host machine;

[0133] In response to the first target container being marked with a first deletion flag, starting a migration process of migrating the metadata stored in the first target container to the target server;

[0134] After the migration process is completed, determining a first service state of a first service corresponding to the first target container;

[0135] When the first service status indicates that the first service is operating normally, determining that the metadata stored in the first target container has been migrated to the target server;

[0136] When the first service status indicates that the first service is running abnormally, performing a verification operation on metadata stored in the first target container in the target server to obtain a target verification result;

[0137] When the target verification result indicates an abnormality, the migration process of migrating the metadata stored in the first target container to the target server is restarted, and when the number of times the target verification result indicates an abnormality meets a preset number of conditions, the first deletion mark is deleted and the state of the first target container is restored.

[0138] Optionally, in this embodiment, it is assumed that the container to be migrated is called the first target container, which is currently running on the first host. The system decides to perform the migration operation within the target period according to the set time policy. To this end, it is first necessary to take the first target container out of the queue to be migrated and set a first deletion mark for it. The main function of this mark is to indicate that the container is about to be deleted, so as to avoid unnecessary operations on it during the migration process.

[0139] When the first target container is marked for deletion, the system will begin to migrate its stored metadata to the target server. This step is an important step to ensure data consistency, because the normal operation of the service is crucial during the migration. The setting of the mark provides clear guidance for subsequent operations, ensuring that other operations will not affect the status of the target container during the migration process.

[0140] After the metadata migration process is completed, the system needs to confirm the status of the first service corresponding to the first target container. If the service status is normal, the system can confirm that the metadata has been successfully migrated to the target server. Conversely, if the service status is abnormal, the system will verify the metadata stored in the target server to ensure the integrity and consistency of the data.

[0141] If the target verification result is found to be abnormal during the verification process, the system will restart the metadata migration process. This mechanism ensures that even if problems occur during the migration process, the data can still be effectively restored. In addition, the system will record the number of abnormal verifications; once the number of abnormalities reaches the preset condition, the system will delete the first deletion marker and restore the state of the first target container. This strategy not only reduces the risk of data loss, but also improves the robustness of the system.

[0142] Through the above mechanism, data security and service availability during container migration are effectively guaranteed. Setting deletion markers makes operations during migration more targeted and secure. At the same time, metadata verification and exception handling mechanisms provide multiple safeguards for the system, effectively avoiding data loss or service interruption caused by migration.

[0143] As an optional solution, the above method also includes:

[0144] In a case where it is determined that the metadata stored in the first target container has been migrated to the target server, or in a case where the number of times the target verification result indicates an abnormality satisfies a preset number condition, a second target container is determined from the queue to be migrated, and a second deletion mark is set for the second target container, wherein the second deletion mark is used to indicate that the second target container is a container to be deleted in the second host machine, the target container includes the second target container, and the target host machine includes the second host machine;

[0145] In response to the second target container being set with the second deletion mark, a migration process of migrating the metadata stored in the second target container to the target server is started.

[0146] Optionally, in this embodiment, by utilizing the deletion marking and metadata verification mechanism, efficient and secure data management can be achieved in container migration. This not only improves the reliability of the service, but also provides strong support for container management in a cloud computing environment. With the development of technology, this mechanism will continue to be optimized to adapt to more complex and dynamic cloud computing needs. It should be noted that during the migration and expansion process, there may be scenarios such as network anomalies or data packet transmission damage, which often require manual intervention to restore the system environment, which is time-consuming and labor-intensive, and the system metadata service no longer has redundancy during this period. In response to the above-mentioned abnormal scenarios, the present application designs an automatic fault detection and processing algorithm during the migration process to achieve automatic detection of anomalies, retries, and rapid rollback.

[0147] In an exemplary embodiment, Fig.10 is a schematic diagram of another optional metadata processing method according to an embodiment of the present application, such as Fig.10 As shown, when the migration starts, a deletion mark will be set on the current migration container, and the services in the container will be stopped. The success of the migration will be determined based on the service status in the container after the migration. If the migration is successful, the next container migration task will be executed. If the migration fails, the MD5 value of the server's current network and the peer data packet will be checked first. When the check is abnormal, the migration will be automatically retried twice by default. To improve the recovery efficiency, the migration retry process does not need to be divided into time and blocks. When the retry fails, the deletion mark on the original container will be removed, the container status will be restored, and the container will be skipped for subsequent container migration. This application utilizes an automatic detection and retry mechanism to handle occasional network transmission anomalies, thereby improving the robustness of the migration. For exceptions that cannot be handled automatically, this application adopts a mode of adding deletion marks to achieve rapid rollback recovery after the exception, thereby improving the reliability of the system.

[0148] In cloud computing and microservice architecture, containers, as a lightweight virtualization technology, can achieve rapid deployment and elastic expansion of applications. In the life cycle management of containers, metadata migration and management are key links to ensure system stability and high availability. When the metadata of the first target container is successfully migrated to the target server, or the number of abnormal target verification results reaches the preset conditions, the second target container needs to be processed.

[0149] After confirming that the metadata of the first target container has been successfully migrated to the target server, the system will identify the second target container from the queue to be migrated. At this time, in order to identify the migration status of the container, the system will set a "second deletion mark" for the second target container. The significance of this mark is that it clearly indicates that the container is a container to be deleted in the second host, which is convenient for subsequent management and operation.

[0150] For example, suppose in a containerized application, the metadata of the first target container is successfully migrated, but the second target container fails to migrate in time for some reason. The system will automatically mark it as pending deletion to prevent data conflicts or resource waste.

[0151] Once the "second deletion marker" is set for the second target container, the system will start the process of migrating its stored metadata to the target server. This process usually includes the following steps:

[0152] Before the migration, the system will perform an integrity check on the metadata in the second target container to ensure the accuracy and consistency of the data. The metadata is transferred from the second target container to the target server through a secure network channel. In this process, encryption technology is usually used to ensure the security of the data. After receiving the metadata, the target server will verify that the data is complete and correct. Once the migration is confirmed to be successful, the system will clear the second target container from the second host to free up resources.

[0153] In a containerized environment, the migration process for the second target container storage metadata is not only an effective management of resources, but also a guarantee of data security and system stability. Through a reasonable tagging and migration mechanism, enterprises can ensure the efficient and secure operation of their applications in a rapidly developing technology environment.

[0154] The present application is further described below with reference to specific examples:

[0155] The cloud storage metadata cluster system involved in this application is implemented based on containerization. When the cloud storage system is expanded to a certain scale, the metadata upper limit becomes the bottleneck of the system. Under the premise of not reducing system performance and affecting business, the migration mark of the container can be dynamically set according to the load situation in the current cluster, combined with the time-sharing and block-by-block migration scheduling strategy, and the metadata upper limit of the system can be effectively increased through the online expansion of the metadata cluster.

[0156] This application realizes the dynamic expansion of metadata clusters without user and business perception, automatic detection and rapid recovery of migration anomalies, and automatic load balancing after expansion, which greatly improves the reliability of the expansion process and the performance of the system after expansion.

[0157] Before the metadata cluster is expanded, the system will set the migration tag information of each container according to the dynamic sampling tag algorithm, which contains important information such as container migration status, priority, load status, etc., to reduce the cost and time of migration. When migrating metadata containers, the system can automatically divide the data to be migrated into multiple data blocks according to the tag information, and migrate these blocks in time, reducing the impact on the business and helping to improve the availability, stability and performance of the system. Based on physical fault tolerance, the migration scheduling algorithm can automatically perform load balancing, including business model balancing and pressure balancing. The metadata cluster dynamic expansion method designed in this paper combines dynamic tagging and migration scheduling algorithms to linearly increase the metadata upper limit, effectively solving the metadata bottleneck problem in large-scale cloud storage cluster scenarios, while avoiding the reduction of cluster performance and improving the reliability and scalability of the system.

[0158] By adding an automatic fault detection and processing strategy, this application can detect, process and roll back anomalies in the migration process in a timely manner, greatly improving the reliability of the system during the expansion process. Existing container migration methods do not have the ability to detect and roll back anomalies during the migration process. When problems occur, manual investigation and processing are often required, and the environment and business cannot be restored in time. The automatic detection and processing strategy designed in this application solves this problem. During the container migration process, the running status of the container and service after migration will be verified. In the event of an abnormality, an environmental verification and retransmission mechanism will be added, and by setting a deletion mark, a rapid rollback of the abnormality can be achieved, further improving the reliability and fault tolerance of the migration process.

[0159] Based on the above algorithms and strategies, this application effectively solves the system performance bottleneck problem caused by the metadata upper limit in large-scale cloud storage cluster scenarios, ensuring the reliability of online migration while further improving system performance.

[0160] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0162] According to another aspect of the embodiment of the present application, a metadata processing device for implementing the above metadata processing method is also provided. Fig.11 As shown, the device comprises:

[0163] The sampling module 1102 is used to perform a sampling mark operation on the container set, determine the container set information, and determine the target container according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is a target period;

[0164] An adding module 1104 is configured to add the target container to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated during the target period;

[0165] The migration module 1106 is configured to migrate the target metadata stored in the target container to a target server in blocks according to the queue to be migrated in response to the system time reaching the target time period, wherein the target server is different from the server deployed by the target host.

[0166] As an optional solution, the device is used to perform a sampling marking operation on a container set, determine container set information, and determine a target container according to the container set information in the following manner:

[0167] Whenever the system time reaches a sampling time point, the container load corresponding to each container in the container set is traversed and sampled;

[0168] Whenever a traversal sampling is completed, a plurality of marking sub-files are generated, wherein one marking sub-file among the plurality of marking sub-files is used to represent a container load corresponding to a container in a traversal sampling process;

[0169] In response to the end of sampling in the current sampling cycle, multiple groups of marked sub-files are obtained, the multiple groups of marked sub-files are merged and marked according to container identifiers to obtain the container set information, and the target container is determined according to the container set information, wherein the current sampling cycle includes multiple sampling time points, and the time period with the lowest container load corresponding to each container in the current sampling cycle is set as the allowed migration period.

[0170] As an optional solution, the device is used to traverse and sample the container load corresponding to each container in the container set every time the system time reaches a sampling time point in the following manner:

[0171] Whenever the system time reaches a sampling time point, each container in the container set is sampled in the following manner, wherein the container sampled each time is regarded as the current container, and the current container is deployed on the current host:

[0172] Obtaining a container load indicator of the current container and a host load indicator corresponding to the current host, wherein the container load indicator and the host load indicator have the same indicator quantity;

[0173] In the case where there are multiple indicators, determining a target proportional relationship based on the values ​​of the host machine load indicators;

[0174] The container load corresponding to the current container is determined based on the container load indicator and the target proportional relationship.

[0175] As an optional solution, the device is used to obtain multiple groups of marked subfiles in response to the end of sampling in the current sampling period in the following manner, merge and mark the multiple groups of marked subfiles according to container identifiers to obtain the container set information, and determine the target container according to the container set information:

[0176] In response to the end of sampling in the current sampling period, acquiring the multiple groups of marked sub-files and the load levels of the expansion servers allowed to be migrated;

[0177] determining a number of containers to be migrated based on the load level;

[0178] Arrange the container sets in descending order according to the container loads according to the container set information to obtain a target list;

[0179] Filtering containers corresponding to the number of containers to be migrated from the target list, adding a first migration mark, and adding a second migration mark to the containers in the container set that are not added with the first migration mark, to generate the container set information, wherein the first migration mark indicates that the corresponding container is allowed to migrate, and the second migration mark indicates that the corresponding container is prohibited from migrating;

[0180] A container in the container set information that has the first migration mark and whose allowed migration period is the target period is determined as the target container.

[0181] As an optional solution,

[0182] The device is used to perform a sampling and marking operation on a container set, determine container set information, and determine a target container according to the container set information in the following manner: perform a sampling and marking operation on a container set, determine container set information, and determine a target container according to the container set information, wherein the container set information also includes a number of migration blocks set for each container in the container set;

[0183] The device is used to migrate the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated in response to the system time reaching the target time period in the following manner: in response to the system time reaching the target time period, perform a block operation on the target metadata according to the number of migration blocks to obtain a group of data blocks; set a corresponding migration time label for each data block in the group of data blocks according to the length of the target time period and the number of migration blocks; and migrate the group of data blocks to the target server in sequence according to the migration time labels.

[0184] As an optional solution, the device is used to migrate the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated in response to the system time reaching the target time period in the following manner:

[0185] In response to the system time reaching the target time period, taking out a first target container from the queue to be migrated, and setting a first deletion mark for the first target container, wherein the first deletion mark is used to indicate that the first target container is a container to be deleted in the first host machine, the target container includes the first target container, and the target host machine includes the first host machine;

[0186] In response to the first target container setting the first deletion mark, starting a migration process of migrating the metadata stored in the first target container to the target server;

[0187] After the migration process is completed, determining a first service state of a first service corresponding to the first target container;

[0188] When the first service status indicates that the first service is operating normally, determining that the metadata stored in the first target container has been migrated to the target server;

[0189] When the first service status indicates that the first service is running abnormally, performing a verification operation on the metadata stored in the first target container in the target server to obtain a target verification result;

[0190] When the target verification result indicates an abnormality, the migration process of migrating the metadata stored in the first target container to the target server is restarted, and when the number of times the target verification result indicates an abnormality meets a preset number of conditions, the first deletion mark is deleted and the state of the first target container is restored.

[0191] As an optional solution, the device is also used for:

[0192] In a case where it is determined that the metadata stored in the first target container has been migrated to the target server, or in a case where the number of times the target verification result indicates an abnormality satisfies a preset number condition, a second target container is determined from the queue to be migrated, and a second deletion mark is set for the second target container, wherein the second deletion mark is used to indicate that the second target container is a container to be deleted in the second host machine, the target container includes the second target container, and the target host machine includes the second host machine;

[0193] In response to the second target container setting the second deletion mark, a migration process of migrating the metadata stored in the second target container to the target server is started.

[0194] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0195] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0196] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program.

[0197] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0198] Fig.12 The structure block diagram of a computer system for implementing an electronic device according to an embodiment of the present application is schematically shown.

[0199] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0200] like Fig.12 As shown, the computer system 1200 includes a central processing unit 1201 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1202 (ROM) or the program loaded from the storage part 1208 to the random access memory 1203 (RAM). Various programs and data required for system operation are also stored in the random access memory 1203. The central processing unit 1201, the read-only memory 1202 and the random access memory 1203 are connected to each other through a bus 1204. The input / output interface 1205 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1204.

[0201] The following components are connected to the input / output interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0202] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processor 1201, various functions defined in the system of the present application are executed.

[0203] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processor 1201, various functions provided by the embodiments of the present application are performed.

[0204] According to another aspect of the embodiment of the present application, an electronic device for implementing the above metadata processing method is also provided. The electronic device may be Figure 1 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.13 As shown, the electronic device includes a memory 1302 and a processor 1304. The memory 1302 stores a computer program, and the processor 1304 is configured to execute the steps in any of the above method embodiments through the computer program.

[0205] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0206] Optionally, in this embodiment, the above-mentioned processor can be configured to execute the methods in each embodiment of the present application through a computer program.

[0207] Alternatively, a person skilled in the art may understand that: Fig.13 The structure shown is for illustration only. Fig.13 The structure of the electronic device is not limited. Fig.13 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.13 Different configurations are shown.

[0208] Among them, the memory 1302 can be used to store software programs and modules, such as the program instructions / modules corresponding to the metadata processing method and device in the embodiments of the present application. The processor 1304 executes various functional applications and data processing by running the software programs and modules stored in the memory 1302, that is, realizing the above-mentioned metadata processing method. The memory 1302 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1302 may further include a memory remotely located relative to the processor 1304, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1302 can be specifically, but not limited to, used to store information such as metadata. As an example, such as Fig.13 As shown, the memory 1302 may include but is not limited to the processing device for the metadata. In addition, it may also include but is not limited to other module units in the processing device for the metadata, which will not be described in detail in this example.

[0209] Optionally, the transmission device 1306 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 1306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1306 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0210] In addition, the electronic device further comprises: a display 1308 for displaying the container; and a connection bus 1310 for connecting various module components in the electronic device.

[0211] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. The nodes may form a peer-to-peer network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0212] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the metadata processing method provided in various optional implementations of the above-mentioned metadata processing aspects.

[0213] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be configured to store data for executing the methods in various embodiments of the present application.

[0214] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0215] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0216] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for one or more electronic devices to execute all or part of the steps of the methods described in each embodiment of the present application.

[0217] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0218] In the several embodiments provided in the present application, it should be understood that the disclosed application can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0219] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0220] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0221] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing metadata, characterized in that: include: Performing a sampling mark operation on a container set to determine container set information, and determining a target container according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is a target period; Adding the target container to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated during the target period; In response to the system time reaching the target time period, migrating the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated, wherein the target server is different from the server deployed by the target host; The method further includes: obtaining a container load index of a current container and a host load index corresponding to a current host, wherein a container sampled each time is regarded as the current container, the current container is deployed on the current host, and the number of indicators of the container load index and the host load index is the same; when the number of indicators is multiple, determining a target proportional relationship based on the value of the host load index; and determining the container load corresponding to the current container based on the container load index and the target proportional relationship.

2. The method according to claim 1, characterized in that The performing a sampling marking operation on the container set, determining container set information, and determining a target container according to the container set information includes: Whenever the system time reaches a sampling time point, the container load corresponding to each container in the container set is traversed and sampled; Whenever a traversal sampling is completed, a plurality of marking sub-files are generated, wherein one marking sub-file among the plurality of marking sub-files is used to represent a container load corresponding to a container in a traversal sampling process; In response to the end of sampling in the current sampling cycle, multiple groups of marked sub-files are obtained, the multiple groups of marked sub-files are merged and marked according to container identifiers to obtain the container set information, and the target container is determined according to the container set information, wherein the current sampling cycle includes multiple sampling time points, and the time period with the lowest container load corresponding to each container in the current sampling cycle is set as the allowed migration period.

3. The method according to claim 2, characterized in that In response to the end of sampling in the current sampling period, acquiring multiple groups of marked subfiles, merging and marking the multiple groups of marked subfiles according to container identifiers to obtain the container set information, and determining the target container according to the container set information, including: In response to the end of sampling in the current sampling period, acquiring the multiple groups of marked sub-files and the load levels of the expansion servers allowed to be migrated; determining a number of containers to be migrated based on the load level; Arrange the container sets in descending order according to the container loads according to the container set information to obtain a target list; Filtering containers corresponding to the number of containers to be migrated from the target list, adding a first migration mark, and adding a second migration mark to the containers in the container set that are not added with the first migration mark, to generate the container set information, wherein the first migration mark indicates that the corresponding container is allowed to migrate, and the second migration mark indicates that the corresponding container is prohibited from migrating; A container in the container set information that has the first migration mark and whose allowed migration period is the target period is determined as the target container.

4. The method according to claim 1, characterized in that: The performing a sampling and marking operation on the container set, determining the container set information, and determining the target container according to the container set information, includes: performing a sampling and marking operation on the container set, determining the container set information, and determining the target container according to the container set information, wherein the container set information also includes the number of migration blocks set for each container in the container set; In response to the system time reaching the target time period, the target metadata stored in the target container is migrated to the target server in the form of blocks according to the queue to be migrated, including: in response to the system time reaching the target time period, a block operation is performed on the target metadata according to the number of migration blocks to obtain a group of data blocks; a corresponding migration time label is set for each data block in the group of data blocks according to the length of the target time period and the number of migration blocks; and the group of data blocks are migrated to the target server in sequence according to the migration time labels.

5. The method according to claim 1, characterized in that: In response to the system time reaching the target time period, migrating the target metadata stored in the target container to the target server in the form of blocks according to the queue to be migrated, includes: In response to the system time reaching the target time period, taking out a first target container from the queue to be migrated, and setting a first deletion mark for the first target container, wherein the first deletion mark is used to indicate that the first target container is a container to be deleted in the first host machine, the target container includes the first target container, and the target host machine includes the first host machine; In response to the first target container setting the first deletion mark, starting a migration process of migrating the metadata stored in the first target container to the target server; After the migration process is completed, determining a first service state of a first service corresponding to the first target container; When the first service status indicates that the first service is operating normally, determining that the metadata stored in the first target container has been migrated to the target server; When the first service status indicates that the first service is running abnormally, performing a verification operation on the metadata stored in the first target container in the target server to obtain a target verification result; When the target verification result indicates an abnormality, the migration process of migrating the metadata stored in the first target container to the target server is restarted, and when the number of times the target verification result indicates an abnormality meets a preset number of conditions, the first deletion mark is deleted and the state of the first target container is restored.

6. The method according to claim 5, characterized in that The method further comprises: In a case where it is determined that the metadata stored in the first target container has been migrated to the target server, or in a case where the number of times the target verification result indicates an abnormality satisfies a preset number condition, a second target container is determined from the queue to be migrated, and a second deletion mark is set for the second target container, wherein the second deletion mark is used to indicate that the second target container is a container to be deleted in the second host machine, the target container includes the second target container, and the target host machine includes the second host machine; In response to the second target container setting the second deletion mark, a migration process of migrating the metadata stored in the second target container to the target server is started.

7. A metadata processing device, characterized in that: include: A sampling module, configured to perform a sampling mark operation on a container set, determine container set information, and determine a target container according to the container set information, wherein the container set information includes a migration mark, an allowed migration period, and a container load of each container in the container set, each container in the container set is used to store metadata, different containers in the container set are allowed to be deployed on different host machines, the migration mark is used to indicate whether the corresponding container needs to be migrated, the container load of the target container is related to the host load of the target host machine where the target container is deployed, the target container represents a container in the container set whose container load meets a preset load condition, and the allowed migration period corresponding to the target container is a target period; An adding module, configured to add the target container to a queue to be migrated, wherein the queue to be migrated represents a queue of containers that need to be migrated during the target period; a migration module, configured to migrate the target metadata stored in the target container to a target server in a block form according to the queue to be migrated in response to the system time reaching the target time period, wherein the target server is different from a server deployed by the target host; The device is also used to: obtain a container load index of a current container and a host load index corresponding to a current host, wherein a container sampled each time is regarded as the current container, the current container is deployed on the current host, and the number of indicators of the container load index and the host load index is the same; when the number of indicators is multiple, determine a target proportional relationship based on the value of the host load index; and determine the container load corresponding to the current container based on the container load index and the target proportional relationship.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program can be executed by an electronic device to perform the method described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images