Capacity determination method, device and electronic equipment for object storage service

Abstract

The application discloses a capacity determination method and device of an object storage service and electronic equipment. It relates to the technical field of data storage. The method comprises the following steps: acquiring log streams of at least one first service partition to obtain a log stream set, wherein the first service partition is a service partition corresponding to a data index service in the object storage service; if it is detected that the first service partition is changed, the log stream of the service partition obtained through the change is merged into the log stream set; and based on data change information in log entries in the log stream set, used capacity information of a data container in the object storage service is determined. The application solves the technical problem that, in related technologies, once a data index service partition used for performing a data change operation on a data container is changed, additional capacity measurement work is generated, resulting in high work cost.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and more specifically, to a method, apparatus, and electronic device for determining the capacity of an object storage service. Background Technology

[0002] In object storage services, capacity metering of data containers (e.g., buckets) is a crucial function. Capacity metering refers to the statistical analysis of storage resource usage, which not only affects the accuracy of billing but also directly relates to the management and optimization of storage resources. Traditional capacity metering schemes rely on data indexing services. However, when partition changes occur in the data indexing service partitions, such as partition splitting or merging, additional capacity metering work is generated, leading to high operating costs.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for determining the capacity of an object storage service, in order to at least solve the technical problem that when measuring the capacity of data containers in an object storage service, additional capacity measurement work is required and the work cost is high once the data index service partition used to perform data change operations on the data container is changed.

[0005] According to one aspect of the embodiments of this application, a method for determining the capacity of an object storage service is provided, comprising: acquiring log streams of at least one first service partition to obtain a log stream set, wherein the first service partition is a service partition corresponding to a data index service in the object storage service, and the data index service is used to perform data modification operations on data containers in the object storage service; if a partition change is detected in the first service partition, the log streams of the service partition obtained from the partition change are merged into the log stream set; and based on the data change information in the log entries in the log stream set, determining the used capacity information of the data containers in the object storage service.

[0006] Furthermore, before incorporating the log stream of the service partition obtained from the partition change into the log stream set, the method further includes: detecting whether there is a target log entry in the log stream, wherein the target log entry is a log entry that represents the partition change of the first service partition; if the target log entry is detected in the log stream, determining the log stream corresponding to the service partition obtained from the partition change based on the target log entry.

[0007] Furthermore, a log stream set is obtained through the second service partition, which is the service partition corresponding to the metering service in the object storage service. The partition change includes: partition merging. If there are multiple first service partitions, and a partition change is detected in the first service partition, the log stream of the service partition obtained by the partition change is merged into the log stream set. This includes: if a partition merging of the first service partition is detected, and there are multiple second service partitions, the target first service partition is determined from the merged first service partitions based on the partition information of the merged first service partition; if the log stream of the target first service partition belongs to the log stream set, the log stream of the service partition obtained by the partition merging is merged into the log stream set.

[0008] Furthermore, based on the data change information in the log entries of the log stream set, determining the used capacity information of the data container in the object storage service includes: determining the incremental capacity information of the data container in the object storage service based on the data change information in the log entries of the log stream set; and determining the used capacity information of the data container based on the incremental capacity information of the data container.

[0009] Furthermore, based on the data change information in the log entries in the log stream set, and according to the first statistical time interval, the incremental capacity information of the data container in the object storage service within the first time interval is determined based on the data change information in the log entries generated within the first time interval in the log stream set. Here, the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0010] Furthermore, based on the data change information in the log entries generated within the first time interval in the log stream set, the incremental capacity information of the data container in the object storage service within the first time interval is determined as follows: based on the data change information in the log entries generated within the first time interval in the log stream set, the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval is determined, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained by partition changes; based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, the incremental capacity information of the data container within the first time interval is determined.

[0011] Furthermore, the first time interval is determined in the following way: when based on the first statistical incremental capacity information of the log stream set, the generation time of the first log entry in the log stream set is determined, and the first time interval is determined based on the generation time and the first statistical time interval; when based on the Nth statistical incremental capacity information of the log stream set, the first time interval corresponding to the Nth statistical is determined based on the time end point of the first time interval of the (N-1)th statistical, and the first statistical time interval, where N is a positive integer greater than 1.

[0012] Furthermore, determining the used capacity information of the data container based on the incremental capacity information of the data container includes: determining the used capacity information at the end point of the second time interval based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, according to the second statistical time interval. Here, the second statistical time interval refers to the statistical time interval between the used capacity information of the data container, and the time length of the second time interval is the same as the second statistical time interval.

[0013] Furthermore, after determining the incremental capacity information of the data container in the object storage service, the method also includes: recording the log consumption progress corresponding to the log stream set into the consumption progress information table; in the case of statistical anomalies in the incremental capacity information, if the statistical anomalies are detected to have been resolved, then based on the log consumption progress in the consumption progress information table, the incremental capacity information is further statistically analyzed based on the log stream set.

[0014] According to another aspect of the embodiments of this application, a capacity determination apparatus for an object storage service is also provided, comprising: an acquisition unit, configured to acquire log streams of at least one first service partition to obtain a log stream set, wherein the first service partition is a service partition corresponding to a data index service in the object storage service, and the data index service is used to perform data modification operations on data containers in the object storage service; a first processing unit, configured to, if a partition change is detected in the first service partition, merge the log streams of the service partition obtained from the partition change into the log stream set; and a first determination unit, configured to determine the used capacity information of the data containers in the object storage service based on the data change information in the log entries in the log stream set.

[0015] Furthermore, the capacity determination device for the object storage service includes: a detection unit, used to detect whether a target log entry exists in the log stream, wherein the target log entry is a log entry representing a partition change in the first service partition; and a second determination unit, used to determine the log stream corresponding to the service partition obtained from the partition change based on the target log entry when the target log entry is detected in the log stream.

[0016] Furthermore, a log stream set is obtained through the second service partition, which is the service partition corresponding to the metering service in the object storage service. The partition change includes partition merging. In the case of multiple first service partitions, the first processing unit includes: a first determining subunit, used to determine the target first service partition from the merged first service partitions based on the partition information of the merged first service partition when a partition merging of the first service partition is detected and there are multiple second service partitions; and a processing subunit, used to merge the log stream of the service partition obtained by partition merging into the log stream set when the log stream of the target first service partition belongs to the log stream set.

[0017] Furthermore, the first determining unit includes: a second determining subunit, used to determine the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set; and a third determining subunit, used to determine the used capacity information of the data container based on the incremental capacity information of the data container.

[0018] Furthermore, the second determining subunit includes: a first determining module, used to determine the incremental capacity information of the data container in the object storage service within the first time interval based on the data change information in the log entries generated in the log stream set within the first time interval according to the first statistical time interval, wherein the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0019] Furthermore, the second determining module includes: a first determining submodule, used to determine the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval based on the data change information in the log entries generated within the first time interval in the log stream set, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained by partition changes; and a second determining submodule, used to determine the incremental capacity information of the data container within the first time interval based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval.

[0020] Furthermore, the capacity determination device for the object storage service includes: a third determining unit, configured to determine the generation time of the first log entry in the log stream set based on the first statistical incremental capacity information of the log stream set, and determine a first time interval based on the generation time and the first statistical time interval; and a fourth determining unit, configured to determine the first time interval corresponding to the Nth statistical event based on the time end point of the first time interval of the N-1th statistical event and the first statistical time interval, based on the Nth statistical incremental capacity information of the log stream set, where N is a positive integer greater than 1.

[0021] Furthermore, the third determining subunit includes: a second determining module, used to determine the used capacity information at the end point of the second time interval based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, according to the second statistical time interval, wherein the second statistical time interval refers to the statistical time interval between the used capacity information of the data container, and the time length of the second time interval is the same as the second statistical time interval.

[0022] Furthermore, the capacity determination device for the object storage service includes: a second processing unit, used to record the log consumption progress corresponding to the log stream set into a consumption progress information table; and a third processing unit, used to continue to calculate incremental capacity information based on the log stream set according to the log consumption progress in the consumption progress information table if the statistical anomaly in the incremental capacity information is detected to have been resolved.

[0023] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the capacity determination method for the object storage service described above during runtime.

[0024] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein the storage medium stores a program, wherein the program controls the device where the storage medium is located to execute the object storage service capacity determination method described above during program execution.

[0025] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the capacity determination method for the object storage service described above.

[0026] In this embodiment, a log stream set is obtained by acquiring the log streams of at least one first service partition, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data change operations on the data container in the object storage service. If a partition change is detected in the first service partition, the log stream of the service partition resulting from the partition change is merged into the log stream set. Based on the data change information in the log entries in the log stream set, the used capacity information of the data container in the object storage service is determined. By acquiring the log streams of at least one first service partition corresponding to the data index service in the object storage service, the capacity change information of the data container is effectively acquired. By merging the log stream of the service partition resulting from the partition change into the log stream set when a partition change is detected in the first service partition, the capacity change information of the data container after the partition change is acquired, and the re-statistical analysis of the capacity information before the partition change is avoided. This allows capacity measurement to avoid being affected by the partition change. When determining the used capacity information of the data container based on the data change information in the log entries in the log stream set, the working cost can be effectively reduced. This approach achieves the goal of capacity metering based on consumption log streams, thereby reducing operational costs. It also solves the technical problem that when related technologies perform capacity metering on data containers in object storage services, additional capacity metering work is required and leads to high operational costs once the data index service partition used for data change operations on the data container changes. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 This is a schematic diagram of a computer terminal provided according to Embodiment 1 of this application;

[0029] Figure 2 This is a flowchart of the object storage service capacity determination method provided in Embodiment 1 of this application;

[0030] Figure 3 This is a flowchart illustrating the consumer log stream set provided in Embodiment 1 of this application. Figure 1 ;

[0031] Figure 4 This is a flowchart illustrating the consumer log stream set provided in Embodiment 1 of this application. Figure 2 ;

[0032] Figure 5This is a flowchart for determining incremental capacity information according to Embodiment 1 of this application;

[0033] Figure 6 This is a schematic diagram of an object storage service capacity determination device according to Embodiment 2 of this application;

[0034] Figure 7 This is a structural block diagram of an electronic device provided according to Embodiment 3 of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0038] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0039] Data indexing service: responsible for data persistence and indexing.

[0040] Partition: Also known as service partition, it is a key concept in distributed services and usually represents a subset of a service.

[0041] Capacity data refers to statistical data stored in the object storage service, such as the total number of objects, the total size of objects, and statistics on object types.

[0042] Capacity metering refers to the statistical analysis of storage resource usage.

[0043] Consumption progress: refers to the progress during the consumption log stream process.

[0044] Splitting: refers to the process of splitting a service partition into two partitions.

[0045] Merging: refers to the process of merging two service partitions into one partition.

[0046] Parent / Child partition: refers to the new (child) and old (parent) partitions during the partition change process.

[0047] Example 1

[0048] According to an embodiment of this application, a method for determining the capacity of an object storage service is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0049] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a capacity determination method for object storage services is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include, but is not limited to, processing devices such as microprocessors (MCUs) or field-programmable gate arrays (FPGAs), and the processor set 102 may include a processor set, Figure 1 (Illustrated using 102a, 102b, ..., 102n), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0050] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be wholly or partially embodied in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the computer terminal 10 (or mobile device).

[0051] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the object storage service capacity determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned object storage service capacity determination method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0053] The display may be a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0054] In object storage services, capacity metering of data containers (e.g., buckets) is a crucial function. Capacity metering refers to the statistical analysis of storage resource usage, which not only affects the accuracy of billing but also directly relates to the management and optimization of storage resources. Traditional capacity metering schemes rely on data indexing services, which are responsible not only for storing and indexing objects but also for recording information on changes in the capacity of data containers. However, this dependency exposes problems when partition changes occur in the data indexing service, such as partition splitting or merging.

[0055] In related technologies, capacity metering is typically tightly coupled with the control plane of data indexing services. This means that any partition changes to the data indexing service directly impact the accuracy and efficiency of capacity metering. For example, in some solutions, operations on data indexing service partitions are directly written to their corresponding metering partitions. While this one-to-one binding is intuitive, when data indexing service partitions split or merge, the newly created partitions need to recalculate the capacity data for data containers within that partition. This is undoubtedly a huge burden for large-scale data storage scenarios. Furthermore, this approach also suffers from coupling with metering logic, making the metering service less flexible in responding to the elastic scaling of the data indexing service.

[0056] Therefore, traditional capacity metering schemes often require additional computing resources and complex algorithms to ensure the accuracy and consistency of capacity metering data when handling partition changes in data indexing services, resulting in high operating costs.

[0057] Against the above-mentioned technical background, this application provides as follows Figure 2 The method for determining the capacity of the object storage service is shown. Figure 2 This is a flowchart of a method for determining the capacity of an object storage service according to Embodiment 1 of this application. The method includes:

[0058] Step S201: Obtain log streams from at least one first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data modification operations on the data container in the object storage service.

[0059] Optionally, electronic devices, servers, processing systems, and other similar devices can be used as the executing entity of this application. In this embodiment, object storage services are used as the executing entity of this application. Optionally, the object storage service can be a cloud-based object storage service.

[0060] Optionally, modern cloud storage architectures, taking object storage as an example, all possess partitioning capabilities, thereby providing horizontally scalable elastic service capabilities. In object storage services, there is at least one service partition corresponding to a data indexing service, i.e., at least one first service partition. The data indexing service is responsible for data persistence and indexing, and it is used to perform data modification operations on data containers (e.g., buckets) in the object storage service. Data modification operations include, but are not limited to, object creation, object modification, and object deletion. Data containers are used to store objects. Multiple data containers can exist in an object storage service. Different first service partitions can perform data modifications on different data within the same data container, or on data within different data containers.

[0061] In some embodiments, the objects described above may be images, videos, audio, documents, structured data, etc., and are not specifically limited in this embodiment.

[0062] Optionally, the first service partition generates its own corresponding log stream. The log stream contains multiple log entries, and the log entries provided by a single log stream are strictly ordered, referred to as log order. Log entries are used to record information about data change operations. For example, a log entry may include a log identifier, operation type, operation time, object identifier, the data container to which the object belongs, object size, and object type. Optionally, a log entry may contain both the new and old values ​​of the object being written. For example, if the data change operation type is an overwrite, the new value should at least include the object size and object type of the newly written object, and the old value should at least include the object size and object type of the overwritten object.

[0063] Alternatively, although the first service partition may not have continuous write requests to generate log entries, it can ensure continuous updates to the log stream based on a mechanism of periodically writing no-op operations.

[0064] In an optional embodiment, the object storage service obtains log streams from multiple first service partitions to obtain a log stream set.

[0065] In an optional embodiment, the log stream set can be obtained through a second service partition in the object storage service. The second service partition is the service partition corresponding to the metering service in the object storage service, which is used to perform capacity metering jobs.

[0066] For example, you can obtain the log stream of a first service partition in an object storage service through a single second service partition, thus obtaining the corresponding log stream set for that second service partition. For instance, if there are 10 first service partitions and 1 second service partition in the object storage service, you can obtain the log streams of all 10 first service partitions through the second service partition, resulting in a unique log stream set.

[0067] For example, log streams from the first service partition in the object storage service can be obtained through multiple second service partitions, resulting in multiple log stream sets corresponding to each second service partition. The log streams obtained from different second service partitions are different. For instance, if there are 10 first service partitions and 3 second service partitions in the object storage service, the log streams from the first three first service partitions can be obtained through the first second service partition, resulting in one log stream set. The log streams from the fourth to sixth first service partitions can be obtained through the second second service partition, resulting in another log stream set. Finally, the log streams from the last four first service partitions can be obtained through the third second service partition, resulting in another log stream set. Thus, three log stream sets are obtained.

[0068] Optionally, for a single set of log streams, log entries within different streams are also ordered (according to log timestamps), and this ordering does not depend on the clocks of different log streams being consistent; that is, clock inconsistencies within the cluster are allowed. Even so, the reason it is still considered ordered is that once a log entry is generated, its fundamental properties (including its log timestamp) do not change. Therefore, the order determined based on this timestamp is fixed and unchanging. Based on this invariant, it is called ordered.

[0069] Step S202: If a partition change is detected in the first service partition, the log stream of the service partition resulting from the partition change is merged into the log stream set.

[0070] In an optional embodiment, the aforementioned partition changes include, but are not limited to, partition splitting and partition merging. Partition splitting refers to the process of splitting a service partition into two partitions, and partition merging refers to the process of merging two service partitions into one partition.

[0071] Optionally, the presence of a partition change in the first service partition can be detected based on log entries in the log stream. For example, when a partition change occurs in the first service partition, the first service partition generates a partition change log to indicate that a partition change has occurred. The presence of this partition change log in the log stream can be used to determine whether a partition change has occurred.

[0072] Optionally, the detection of a partition change in the first service partition can be based on message passing. For example, when a partition change occurs in the first service partition, the first service partition generates a partition change message and sends it to the second service partition in the object storage service.

[0073] If a partition change is detected in the first service partition, the object storage service can merge the log stream of the service partition resulting from the partition change into the log stream collection. For example, if the partition change is a partition merge, the service partition resulting from the partition change refers to the new partition obtained after the partition merge; if the partition change is a partition split, the service partition resulting from the partition change refers to the new partition obtained after the partition split.

[0074] After determining the service partition resulting from the partition change, the log stream of the newly generated service partition is added to the original log stream set to achieve dynamic adjustment of the log stream set. This facilitates timely capacity statistics based on relevant data change operations, even if these operations occur after the partition change.

[0075] Step S203: Based on the data change information in the log entries in the log stream set, determine the used capacity information of the data container in the object storage service.

[0076] In an alternative embodiment, the used capacity information of a data container in an object storage service can be determined based on updates to log entries. For example, when a new log entry is generated in the log stream, the used capacity information of the data container is determined based on the newly generated log entry.

[0077] In an optional embodiment, the used capacity information of a data container in the object storage service can be determined periodically. For example, the used capacity information of the data container can be determined according to a preset statistical time interval.

[0078] Optionally, in determining the used capacity information of a data container, the data container targeted by the data change operation corresponding to the log entry and the used capacity change information of that data container can be determined based on the data change information in the log entry, thereby calculating the used capacity information of the data container.

[0079] Optionally, the object storage service can record the statistically obtained used capacity information in a first capacity table. For example, a row in the first capacity table corresponds to the used capacity information of a certain data container, and the records in the first capacity table can be sorted by time.

[0080] In an optional embodiment, object storage service is a storage solution provided by a cloud service provider to users, allowing users to store and access data in the cloud, and the statistically obtained used capacity information can be used for capacity billing of users.

[0081] In an optional embodiment, the statistically obtained used capacity information can also help administrators or users understand the usage of storage resources in the object storage service.

[0082] In an optional embodiment, the statistically obtained used capacity information can be used to determine whether the object storage service needs to be expanded.

[0083] In this solution, log streams from at least one first service partition corresponding to the data index service in the object storage service are obtained, enabling effective acquisition of capacity change information for the data container. By merging the log streams of the service partition resulting from the partition change into a log stream set when a partition change is detected in the first service partition, capacity change information of the data container after the partition change is obtained, avoiding the need to re-calculate capacity information before the partition change. This allows capacity measurement to avoid being affected by partition changes. When determining the used capacity information of the data container based on data change information in the log entries in the log stream set, the operating cost can be effectively reduced. This achieves the goal of capacity measurement based on consuming log streams, realizing the technical effect of reducing operating costs. Furthermore, it solves the technical problem of high operating costs caused by additional capacity measurement work when the data index service partition used for data change operations in the object storage service changes.

[0084] Detecting whether a partition change has occurred in the first service partition is crucial. Therefore, in the capacity determination method of the object storage service provided in Embodiment 1 of this application, before merging the log stream of the service partition obtained from the partition change into the log stream set, it is detected whether there is a target log entry in the log stream, wherein the target log entry is a log entry that represents the partition change of the first service partition; if the target log entry is detected in the log stream, the log stream corresponding to the service partition obtained from the partition change is determined based on the target log entry.

[0085] In an optional embodiment, log entries in the log stream are sorted according to log timestamps, that is, according to the generation time of the log entries. When a partition change occurs in the first service partition, the first service partition generates a target log entry, and the target log entry is the last log entry in the first service partition stream.

[0086] Optionally, the target log entry is a log entry that represents a partition change in the first service partition. The target log entry contains detailed information about the partition change, such as the partition identifier before and after the change, the change type (e.g., split, merge, etc.), and the change timestamp.

[0087] The first log entry in the log stream of the service partition resulting from the partition change is later than the target log entry in the log stream of the first service partition where the partition change occurred. Therefore, the partition change of the first service partition can be detected by detecting the target log entry.

[0088] Optionally, if a target log entry is detected in the log stream, it is determined that the first service partition to which the log stream belongs has undergone a partition change. In this case, the service partition resulting from the partition change can be determined based on the partition identifier in the target log entry, thereby locating the corresponding log stream and merging the log stream into the log stream set.

[0089] Optionally, if no target log entry is detected in the log stream, it is determined that the first service partition to which the log stream belongs has not yet undergone partition changes.

[0090] In an optional embodiment, after merging the log streams of the service partition resulting from the partition change into the log stream set, since no new log entries are generated in the log stream of the first service partition where the partition change occurred, a cleanup operation can be performed on that log stream. For example, the log stream of the first service partition can be deleted.

[0091] It should be noted that by detecting target log entries to determine whether a partition change has occurred, the partition change event can be accurately identified. Furthermore, a new log stream can be quickly located and initialized based on the target log entries, thereby improving detection efficiency and avoiding the need to add a new mechanism on top of the log stream consumption mechanism, thus reducing operating costs.

[0092] To more reliably incorporate log streams into the log stream set, the capacity determination method for object storage services provided in Embodiment 1 of this application obtains the log stream set through a second service partition. The second service partition is the service partition corresponding to the metering service in the object storage service. The partition change includes: partition merging. If a partition change is detected in the first service partition when there are multiple first service partitions, incorporating the log streams of the service partitions obtained from the partition change into the log stream set includes: when a partition merging of the first service partitions is detected and there are multiple second service partitions, determining the target first service partition from the merged first service partitions based on the partition information of the merged first service partitions; and if the log streams of the target first service partition belong to the log stream set, incorporating the log streams of the service partitions obtained from the partition merging into the log stream set.

[0093] Optionally, the second service partition is the service partition corresponding to the metering service in the object storage service, which is used to perform capacity metering operations.

[0094] Log streams from the first service partition in the object storage service can be obtained through multiple second service partitions, resulting in multiple log stream sets corresponding to each second service partition. Different second service partitions will yield different log stream sets. For example, if there are 10 first service partitions and 3 second service partitions in the object storage service, the log streams from the first three first service partitions can be obtained through the first second service partition, resulting in one log stream set. The log streams from the fourth to sixth first service partitions can be obtained through the second second service partition, resulting in another log stream set. Finally, the log streams from the last four first service partitions can be obtained through the third second service partition, resulting in another log stream set. This leads to three log stream sets.

[0095] In an optional embodiment, in layman's terms, the service partitions before and after the partition change can be referred to as parent and child partitions. That is, the service partition that has undergone the partition change (which can also be understood as the old partition) is called the parent partition, and the service partition obtained from the partition change (which can also be understood as the new partition) is called the child partition.

[0096] Since in the case of partition change to partition merging, two parent partitions will correspond to one child partition, and if there are multiple second service partitions, different second service partitions may be responsible for consuming the log streams of the two parent partitions, it is necessary to determine which parent partition's log stream set the newly generated child partition belongs to, in order to avoid two second service partitions consuming the log stream of the child partition at the same time, which would lead to capacity measurement errors.

[0097] Optionally, if a partition merge is detected in the first service partition and there are multiple second service partitions, the second service partition can determine the target first service partition from the merged first service partitions based on the partition information of the merged first service partition. For example, if the partition information is a partition identifier, the second service partition can determine the first service partition on the right (or left) of the merged first service partitions according to the order of the partition identifiers (e.g., partition 1, partition 2, partition 3, etc.).

[0098] Optionally, after the target first service partition is determined, the second service partition may merge the log stream of the service partition obtained by partition merging into the log stream set if the log stream of the target first service partition belongs to the log stream set under the responsibility of the second service partition. Conversely, if the log stream of the target first service partition does not belong to the log stream set under the responsibility of the second service partition, the merging of the log stream of the service partition obtained by partition merging into the log stream set is prohibited.

[0099] In an alternative embodiment, Figure 3 This is a flowchart illustrating the consumer log stream set provided in Embodiment 1 of this application. Figure 1 ,according to Figure 3 The process of consuming an optional set of log streams in this embodiment will be described. For example... Figure 3 As shown, when a partition merge is detected in a first service partition, the partition identifier (i.e., sub-partition information) of the merged sub-partition is extracted from the target log entry of the log stream of that first service partition. Then, based on the sub-partition identifier, it is determined whether the right-hand parent partition belongs to the log stream set currently managed by the second service partition. If it does, the log stream of that sub-partition is initialized, for example, by communicating with the sub-partition to confirm the start of reading the log stream. After that, the parent partition's log stream is removed from the log stream set, and the consumption of the log stream set continues. Optionally, if the right-hand parent partition does not belong to the log stream set currently managed by the second service partition, the parent partition's log stream is directly removed from the log stream set, and the consumption of the log stream set continues.

[0100] In one alternative embodiment, the partition change also includes partition splitting. In the case of partition splitting, one parent partition corresponds to two child partitions, so the log streams of the two child partitions can be directly merged into the log stream set to which the parent partition belongs.

[0101] In an alternative embodiment, Figure 4 This is a flowchart illustrating the consumer log stream set provided in Embodiment 1 of this application. Figure 2 ,according to Figure 4 The process of consuming an optional set of log streams in this embodiment will be described. For example... Figure 4 As shown, when a partition split is detected in a first service partition, the partition identifier (i.e., sub-partition information) of the split sub-partition is extracted from the target log entry of the log stream of the first service partition. Then, the log stream of the sub-partition is initialized based on the partition identifier of the sub-partition. For example, the sub-partition is communicated to confirm the start of reading the log stream. After that, the parent partition log stream is removed from the log stream set, and the consumption of the log stream set continues.

[0102] In one alternative embodiment, the second service partition also has independent splitting and merging capabilities, can be horizontally scaled according to performance requirements, can quickly adapt to the elastic scaling of data indexing services, and has independent migration capabilities, so that services can continue to run in the event of abnormal scenarios.

[0103] In an alternative embodiment, the log stream collection itself may also be split and merged.

[0104] It's important to note that by employing a second service partition to consume the log stream set, capacity metering is achieved based on relatively independent services. As long as the data stream format remains unchanged, services can be isolated from each other. This means that at the management level, both deployment and service elastic scaling are completely decoupled from the front-end services of the data indexing service and object storage service. This results in a smaller blast radius, more flexible iteration and upgrades, reduced data processing complexity, and improved reliability of capacity statistics. Changes to the data indexing service will not affect it, demonstrating good scalability and maintainability, and adaptability to different scenarios. By determining the log stream merging method based on the log stream set to which the target first service partition belongs, the situation where two second service partitions simultaneously consume the log streams of the same sub-partition is effectively avoided, further improving the reliability of log stream merging.

[0105] To more accurately determine the used capacity information of a data container, in the object storage service capacity determination method provided in Embodiment 1 of this application, determining the used capacity information of a data container in the object storage service based on the data change information in the log entries in the log stream set includes: determining the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set, and determining the used capacity information of the data container based on the incremental capacity information of the data container.

[0106] In an alternative embodiment, incremental capacity information for a data container in an object storage service can be determined based on updates to log entries. For example, when a new log entry is generated in the log stream, the incremental capacity information for the data container is determined based on the newly generated log entry.

[0107] In an optional embodiment, the incremental capacity information of data containers in the object storage service can be determined periodically. For example, the incremental capacity information of data containers can be determined according to a preset statistical time interval.

[0108] Optionally, in determining the incremental capacity information, for log entries in the log stream set, if the data change information in the log entry includes both a new value and an old value, the incremental capacity information of the data container corresponding to the log entry is determined based on the difference between the new and old values. Here, the data container corresponding to the log entry refers to the data container targeted by the data change operation for that log entry, and the incremental capacity information includes, but is not limited to, incremental information on the number of objects, incremental information on the size of objects, and object types. If the data change information in the log entry only includes a new value, the incremental capacity information of the data container corresponding to the log entry is determined based on the new value.

[0109] Optionally, after obtaining the incremental capacity information of the data container, this information can be recorded in a second capacity table to determine the used capacity information of the data container. For example, one row in the second capacity table corresponds to one incremental capacity information of a certain data container. Alternatively, the incremental capacity information can be broken down into multiple incremental capacity sub-information, with different sub-information corresponding to different first service partitions and service partitions obtained from partition changes. Thus, one row in the second capacity table corresponds to one incremental capacity sub-information of a certain data container, and multiple rows of records constitute one incremental capacity information.

[0110] Optionally, after obtaining the incremental capacity information of the data container, the used capacity information of the data container can be determined based on the incremental capacity information. The used capacity information includes, but is not limited to, the total number of objects, the total size of objects, and object type statistics. For example, the initial used capacity information of this statistic (e.g., the used capacity information obtained in the previous statistic) can be added to the incremental capacity information obtained in this statistic to obtain the used capacity information of this statistic.

[0111] It should be noted that by first calculating incremental capacity information and then calculating used capacity information based on incremental capacity information, computational complexity can be effectively reduced, capacity metering efficiency can be improved, and resource consumption can be reduced. Furthermore, by decoupling the collection of data change information and the calculation of used capacity information, the system can minimize the impact on stability when dealing with large amounts of data changes.

[0112] To reduce the workload of statistical incremental capacity information, in the object storage service capacity determination method provided in Embodiment 1 of this application, determining the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set includes: determining the incremental capacity information of the data container in the object storage service within the first time interval based on the data change information in the log entries generated within the first time interval in the log stream set according to the first statistical time interval, wherein the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0113] Optionally, the first statistical time interval can be a user-preset time interval. Incremental capacity information can be statistically analyzed periodically according to the first statistical time interval to obtain incremental capacity information corresponding to different time intervals (i.e., the first time interval).

[0114] Optionally, the length of the first time interval is the same as the first statistical time interval, and the start point of the first time interval can be determined based on the end point of the first time interval in the previous statistical analysis. For example, assuming the first statistical time interval is 5 minutes and the end point of the first time interval in the previous statistical analysis was 12:05:00, then the start point of the current first time interval is 12:05:00, and the current first time interval is 12:05:00-12:10:00. Optionally, the start point of the first time interval can be determined based on the generation time of the first log entry in the log stream set.

[0115] Since multiple log streams are ordered and comparable, the second service partition can treat the multiple log streams it acquires as a logical whole for consumption, i.e., a log stream set, based on log order and following the multi-stream merging approach, when determining incremental capacity information. The second service partition can continuously consume log entries generated within the first time interval from the log stream set according to their timestamps, accumulating the incremental capacity corresponding to each log entry in memory until it consumes a log entry whose timestamp falls within the next first time interval. At this point, the accumulated incremental capacity information in memory is determined as the incremental capacity information for the currently statistical first time interval. Note that log timestamps may drift, but this does not affect the accuracy of the statistical data.

[0116] In an optional embodiment, after obtaining the incremental capacity information of the data container within the first time interval, the obtained incremental capacity information can be recorded in the second capacity table mentioned above, and then the incremental capacity of the memory statistics can be cleared to zero, and the next round of incremental capacity information can be started.

[0117] In an alternative embodiment, Figure 5 This is a flowchart for determining incremental capacity information according to Embodiment 1 of this application. Figure 5 The process of periodically determining incremental capacity information in this embodiment is described below. Figure 5As shown, suppose a second service partition is responsible for acquiring log streams from three first service partitions. This second service partition can merge the three log streams into a single stream ordered by log timestamps. Then, it consumes log entries from the single stream, extracts data change information from the log entries, and determines whether the next first time interval has been counted based on the log timestamps in the log entries. If it is still the current first time interval, it performs incremental capacity accumulation calculation in memory based on the data change information. If the next first time interval has been counted, it determines the incremental capacity information accumulated in memory as the incremental capacity information within the current first time interval and records it in the second capacity table. Then, it clears the incremental capacity information counted in memory.

[0118] It should be noted that by statistically analyzing incremental capacity information based on the first statistical time interval, periodic statistics of incremental capacity information are achieved, thereby avoiding the phenomenon of excessive system resource consumption during real-time statistics and effectively reducing system resource consumption.

[0119] To improve the reliability of calculating the used capacity information, in the object storage service capacity determination method provided in Embodiment 1 of this application, determining the incremental capacity information of the data container in the object storage service within the first time interval based on the data change information in the log entries generated in the log stream set within the first time interval includes: determining the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval based on the data change information in the log entries generated in the log stream set within the first time interval, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained by partition changes; and determining the incremental capacity information of the data container within the first time interval based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval.

[0120] In some embodiments, different first service partitions can perform data modification operations on different data in the same data container; that is, changes in the used capacity of the same data container may originate from operations performed by different first service partitions. Therefore, in the process of statistically analyzing incremental capacity information, the incremental capacity information can be subdivided into multiple incremental capacity sub-information, and different incremental capacity sub-information corresponds to different target service partitions.

[0121] For example, suppose a log stream set includes log streams from three first service partitions (partitions A, B, and C), and suppose that between 13:00 and 13:05 (i.e., the first time interval), these partitions performed data modification operations, and one partition underwent a partition change, namely, first service partition C split into two new service partitions C1 and C2. Then, the data modification information in the log entries generated within the first time interval might be as follows:

[0122] Log entry 1: Partition A performed an overwrite operation on object 1 in data container a, increasing the size of object 1 from 100MB to 120MB.

[0123] Log entry 2: Partition B performed a deletion operation on object number 2 in data container a. The object size was 50MB.

[0124] Log entry 3: Partition C performed an overwrite operation on object 3 in data container a, increasing the size of object 3 from 30MB to 60MB.

[0125] Log entry 4: Partition C1 created a new object in data container a: object number 4, with a size of 40MB.

[0126] Log entry 5: Partition C2 performed an overwrite operation on object 5 in data container a, reducing the size of object 5 from 20MB to 10MB.

[0127] Based on the log change information in these log entries, the second partition service determines the incremental capacity sub-information of the data container under the target service partitions (i.e., the original partitions A, B, and C, and the new service partitions C1 and C2 obtained due to partition changes) within the first time interval as follows:

[0128] Incremental Capacity Sub-Information 1: Data Container: Data Container a; First Time Interval: 13:00-13:05; Service Partition: Partition A; Incremental Capacity: +20MB;

[0129] Incremental capacity sub-information 2: Data container: Data container a; First time interval: 13:00-13:05; Service partition: Partition B; Incremental capacity: -50MB;

[0130] Incremental capacity sub-information 3: Data container: Data container a; First time interval: 13:00-13:05; Service partition: Partition C; Incremental capacity: +30MB;

[0131] Incremental capacity sub-information 4: Data container: Data container a; First time interval: 13:00-13:05; Service partition: Partition C1; Incremental capacity: +40MB;

[0132] Incremental capacity sub-information 5: Data container: Data container a; First time interval: 13:00-13:05; Service partition: Partition C2; Incremental capacity: -10MB;

[0133] Optionally, the above-mentioned incremental capacity sub-information 1-5 can be defined as the incremental capacity information of data container a within the first time interval 13:00-13:05.

[0134] Optionally, the second service partition can record the incremental capacity sub-information from the incremental capacity information mentioned above into the second capacity table. A row in the second capacity table corresponds to the incremental capacity sub-information of a certain data container belonging to a certain target service partition within a certain first time interval.

[0135] It should be noted that by refining the incremental capacity information into incremental capacity sub-information based on the target service partition, on the one hand, it ensures that even in scenarios where service partitions change dynamically, the metering service can still accurately capture the details of data container capacity changes, improving the reliability of calculating used capacity information. On the other hand, by calculating incremental capacity sub-information separately under the target service partition, it not only improves the flexibility of calculation but also reduces the complexity and resource consumption when performing elastic scaling operations in the data indexing service, enhancing the capacity metering capability of the object storage service and the overall stability and efficiency of the system.

[0136] To accurately determine the first time interval, in the object storage service capacity determination method provided in Embodiment 1 of this application, the first time interval is determined in the following way: when based on the first statistical incremental capacity information of the log stream set, the generation time of the first log entry in the log stream set is determined, and the first time interval is determined based on the generation time and the first statistical time interval; when based on the Nth statistical incremental capacity information of the log stream set, the first time interval corresponding to the Nth statistical is determined based on the time end point of the first time interval of the N-1th statistical and the first statistical time interval, where N is a positive integer greater than 1.

[0137] For example, when consuming log entries from the log stream collection and performing the first incremental capacity information statistics, the object storage service can obtain the starting log identifier of the log stream in the log stream collection based on the initialization information. Then, based on the log timestamps of the starting log entries of multiple log streams in the log stream collection, it can determine the generation time of the first log entry in the log stream collection. Assuming the generation time of the first log entry is T0 and the first statistical time interval is 5 minutes, then the first time interval can be [T0, T0+5min).

[0138] Optionally, when performing non-first incremental capacity information statistics, if the second incremental capacity information statistics are performed, then [T0+5min, (T0+5min)+5min) can be determined as the first time interval corresponding to the second statistics.

[0139] It should be noted that by using the above method, different methods are used to determine the first time interval under different statistical counts, thereby improving the accuracy of the determination of the first time interval, thus avoiding the omission or duplicate consumption of log entries, and improving the accuracy of the determined incremental capacity information.

[0140] To reduce the workload of statistically analyzing used capacity information, in the object storage service capacity determination method provided in Embodiment 1 of this application, based on the incremental capacity information of the data container, and according to the second statistical time interval, based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, the used capacity information at the end point of the second time interval is determined. Here, the second statistical time interval refers to the statistical time interval between the used capacity information of the data container, and the time length of the second time interval is the same as the second statistical time interval.

[0141] Optionally, the second statistical time interval may be the same as or different from the first statistical time interval.

[0142] Optionally, the second statistical time interval can be a user-preset time interval. The used capacity information can be periodically calculated according to the second statistical time interval to obtain the used capacity information corresponding to different time points (i.e., the end point of the second time interval).

[0143] Optionally, the length of the second time interval is the same as the second statistical time interval, and the start point of the second time interval can be determined based on the end point of the previous second time interval. For example, assuming the second statistical time interval is 1 hour and the end point of the previous second time interval is 12:00:00, then the start point of the current second time interval is 12:00:00, and the current second time interval is 12:00:00-13:00:00. Optionally, the start point of the first second time interval can be preset or determined based on the generation time of the first incremental capacity information.

[0144] Optionally, when the current time point reaches the end point of the second time interval, the used capacity information at the end point of the second time interval can be determined based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval. Optionally, the incremental capacity information corresponding to the second time interval can refer to the incremental capacity information associated with the target first time interval, and the target first time interval can be the first time interval whose end point belongs to the second time interval.

[0145] For example, the incremental capacity information corresponding to the second time interval is determined from the second capacity table, and then the incremental capacity information of the same data container is accumulated to obtain the total incremental capacity information of the data container in the second time interval. Then, the total incremental capacity information of the data container in the second time interval is added to the used capacity information of the data container at the start point of the second time interval to obtain the used capacity information of the data container at the end point of the second time interval.

[0146] Optionally, the used capacity information of the data container at the start point of the second time interval is essentially the same as the used capacity information of the data container at the end point of the previous second time interval. If the second time interval is the first second time interval, a preset value can be used as the used capacity information at the start point of the second time interval, or the used capacity information at the start point of the second time interval can be determined based on input from the administrator or user.

[0147] In an optional embodiment, after obtaining the used capacity information of the data container at the end of the second time interval, the obtained used capacity information can be recorded in the first capacity table mentioned above.

[0148] It should be noted that by calculating the used capacity information based on the second statistical time interval, the periodic statistics of used capacity information are realized, which can avoid the phenomenon of excessive occupation of system resources during real-time statistics and effectively reduce system resource consumption.

[0149] To improve the reliability of incremental capacity information, in the object storage service capacity determination method provided in Embodiment 1 of this application, after determining the incremental capacity information of the data container in the object storage service, the method further includes: recording the log consumption progress corresponding to the log stream set into the consumption progress information table; in the case of statistical anomalies in incremental capacity information, if it is detected that the statistical anomaly has been resolved, then based on the log consumption progress in the consumption progress information table, the incremental capacity information is continued to be statistically analyzed based on the log stream set.

[0150] Optionally, after determining the incremental capacity information of the data container, the log consumption progress corresponding to the log stream set can be recorded in the consumption progress information table. For example, the log identifier of the last processed log entry in the log stream set and the identifier of the log stream set can be used as the log consumption progress and then recorded in the consumption progress information table.

[0151] Optionally, if an abnormal situation (such as network failure, data read error, second service partition failure, etc.) is encountered during the process of calculating incremental capacity information, causing the statistics to fail to complete smoothly, the object storage service can find the first log entry to be consumed in this statistics from the log stream set according to the log consumption progress in the consumption progress information table, and then continue to consume logs and calculate incremental capacity information, provided that the statistical anomaly has been resolved (such as network recovery, data recovery, or the second service partition being started from a new server node).

[0152] For example, after obtaining the incremental capacity information of the data container within the first time interval, the obtained incremental capacity information can be recorded in the second capacity table. Then, the log consumption progress corresponding to this statistical analysis can be recorded in the consumption progress information table. After that, the incremental capacity of the memory statistics is cleared to zero, and the next round of incremental capacity information statistics begins. Mutual exclusion can be implemented when writing data to the table, for example, through session identifiers. If the session is found to be inactive, it needs to stop running.

[0153] It should be noted that the above methods effectively restore statistical anomalies, thereby significantly improving the reliability of statistical incremental capacity information.

[0154] In this embodiment, log streams of at least one first service partition corresponding to the data index service in the object storage service are obtained, enabling effective acquisition of capacity change information of the data container. By merging the log stream of the service partition resulting from the partition change into the log stream set when a partition change is detected in the first service partition, capacity change information of the data container after the partition change is obtained, avoiding the re-statistical analysis of capacity information before the partition change. This allows capacity measurement to avoid being affected by partition changes. When determining the used capacity information of the data container based on the data change information in the log entries in the log stream set, the operating cost can be effectively reduced. This achieves the goal of capacity measurement based on consuming log streams, realizing the technical effect of reducing operating costs. It also solves the technical problem in related technologies that, when measuring the capacity of data containers in object storage services, additional capacity measurement work is generated and the operating cost is high once the data index service partition used for data change operations on the data container changes.

[0155] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0157] Example 2

[0158] According to embodiments of this application, a capacity determination apparatus for an object storage service is also provided for implementing the above-described object storage service capacity determination method, such as... Figure 6 As shown, the device includes an acquisition unit 601, a first processing unit 602, and a first determination unit 603.

[0159] The acquisition unit 601 is used to acquire at least one log stream of the first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data change operations on the data container in the object storage service;

[0160] The first processing unit 602 is configured to, if a partition change is detected in the first service partition, merge the log stream of the service partition resulting from the partition change into the log stream set.

[0161] The first determining unit 603 is used to determine the used capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set.

[0162] In the object storage service capacity determination device provided in Embodiment 2 of this application, the acquisition unit 601 acquires at least one log stream of a first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data change operations on the data container in the object storage service; if the first processing unit 602 detects that a partition change has occurred in the first service partition, it merges the log stream of the service partition obtained from the partition change into the log stream set; the first determination unit 603 determines the used capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set. In this solution, log streams from at least one first service partition corresponding to the data index service in the object storage service are obtained, enabling effective acquisition of capacity change information for the data container. By merging the log streams of the service partition resulting from the partition change into a log stream set when a partition change is detected in the first service partition, capacity change information of the data container after the partition change is obtained, avoiding the need to re-calculate capacity information before the partition change. This allows capacity measurement to avoid being affected by partition changes. When determining the used capacity information of the data container based on data change information in the log entries in the log stream set, the operating cost can be effectively reduced. This achieves the goal of capacity measurement based on consuming log streams, realizing the technical effect of reducing operating costs. Furthermore, it solves the technical problem of high operating costs caused by additional capacity measurement work when the data index service partition used for data change operations in the object storage service changes.

[0163] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the object storage service capacity determination device includes: a detection unit, used to detect whether there is a target log entry in the log stream, wherein the target log entry is a log entry that represents a partition change of the first service partition; and a second determination unit, used to determine the log stream corresponding to the service partition obtained by the partition change based on the target log entry when the target log entry is detected in the log stream.

[0164] Optionally, in the capacity determination device for the object storage service provided in Embodiment 2 of this application, a log stream set is obtained through a second service partition. The second service partition is the service partition corresponding to the metering service in the object storage service. The partition change includes partition merging. When there are multiple first service partitions, the first processing unit 602 includes: a first determining subunit, used to determine the target first service partition from the merged first service partitions based on the partition information of the merged first service partition when a partition merging of the first service partition is detected and there are multiple second service partitions; and a processing subunit, used to merge the log stream of the service partition obtained by partition merging into the log stream set when the log stream of the target first service partition belongs to the log stream set.

[0165] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the first determination unit 603 includes: a second determination subunit, used to determine the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set; and a third determination subunit, used to determine the used capacity information of the data container based on the incremental capacity information of the data container.

[0166] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the second determination subunit includes: a first determination module, used to determine the incremental capacity information of the data container in the object storage service within the first time interval based on the data change information in the log entries generated in the log stream set within the first time interval according to a first statistical time interval, wherein the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0167] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the second determination module includes: a first determination submodule, used to determine the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval based on the data change information in the log entries generated within the first time interval in the log stream set, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained by partition changes; and a second determination submodule, used to determine the incremental capacity information of the data container within the first time interval based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval.

[0168] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the object storage service capacity determination device includes: a third determination unit, used to determine the generation time of the first log entry in the log stream set based on the first statistical incremental capacity information of the log stream set, and to determine a first time interval based on the generation time and the first statistical time interval; a fourth determination unit, used to determine the first time interval corresponding to the Nth statistical based on the time end point of the first time interval of the N-1th statistical based on the Nth statistical incremental capacity information of the log stream set, and the first statistical time interval, where N is a positive integer greater than 1.

[0169] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the third determination subunit includes: a second determination module, used to determine the used capacity information at the end of the time interval in the second time interval based on the used capacity information at the start of the time interval and the incremental capacity information corresponding to the second time interval, according to a second statistical time interval, wherein the second statistical time interval refers to the statistical time interval between the used capacity information of the data containers, and the time length of the second time interval is the same as the second statistical time interval.

[0170] Optionally, in the object storage service capacity determination device provided in Embodiment 2 of this application, the object storage service capacity determination device includes: a second processing unit, used to record the log consumption progress corresponding to the log stream set to the consumption progress information table; and a third processing unit, used to continue to calculate incremental capacity information based on the log stream set according to the log consumption progress in the consumption progress information table if the statistical anomaly of the incremental capacity information has been resolved.

[0171] It should be noted that the acquisition unit 601, the first processing unit 602, and the first determining unit 603 mentioned above correspond to steps S201 to S203 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0172] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0173] Example 3

[0174] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0175] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0176] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the object storage service capacity determination method: obtaining log streams of at least one first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data change operations on the data container in the object storage service; if a partition change is detected in the first service partition, the log stream of the service partition obtained by the partition change is merged into the log stream set; based on the data change information in the log entries in the log stream set, the used capacity information of the data container in the object storage service is determined.

[0177] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: before merging the log stream of the service partition obtained from the partition change into the log stream set, detecting whether there is a target log entry in the log stream, wherein the target log entry is a log entry that represents the partition change of the first service partition; if the target log entry is detected in the log stream, determining the log stream corresponding to the service partition obtained from the partition change based on the target log entry.

[0178] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: when a partition merging of the first service partition is detected and there are multiple second service partitions, the target first service partition is determined from the merged first service partition based on the partition information of the merged first service partition; if the log stream of the target first service partition belongs to the log stream set, the log stream of the service partition obtained by partition merging is incorporated into the log stream set.

[0179] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: determining the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream collection; and determining the used capacity information of the data container based on the incremental capacity information of the data container.

[0180] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: Based on the data change information in the log entries generated in the log stream set within the first time interval according to the first statistical time interval, determine the incremental capacity information of the data container in the object storage service within the first time interval, wherein the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0181] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: based on the data change information in the log entries generated in the log stream set within the first time interval, determine the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained from partition changes; based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, determine the incremental capacity information of the data container within the first time interval.

[0182] The aforementioned electronic device can also execute program code for the following steps in the object storage service capacity determination method: Based on the first statistical incremental capacity information of the log stream set, determine the generation time of the first log entry in the log stream set, and determine a first time interval based on the generation time and the first statistical time interval; Based on the Nth statistical incremental capacity information of the log stream set, determine the first time interval corresponding to the Nth statistical event based on the time termination point of the first time interval of the (N-1)th statistical event and the first statistical time interval, where N is a positive integer greater than 1.

[0183] The aforementioned electronic device can also execute the program code for the following steps in the object storage service capacity determination method: according to the second statistical time interval, based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, determine the used capacity information at the end point of the second time interval, wherein the second statistical time interval refers to the statistical time interval between the used capacity information of the data container, and the time length of the second time interval is the same as the second statistical time interval.

[0184] The aforementioned electronic device can also execute the following steps in the object storage service capacity determination method: after determining the incremental capacity information of the data container in the object storage service, the log consumption progress corresponding to the log stream set is recorded in the consumption progress information table; in the case of statistical anomalies in the incremental capacity information, if the statistical anomalies are detected to have been resolved, the incremental capacity information is continued to be statistically analyzed based on the log consumption progress in the consumption progress information table and the log stream set.

[0185] Optionally, Figure 7 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 7 As shown, the electronic device 70 may include: one or more ( Figure 7 (Only one is shown) Processor 702 and memory 704. The electronic device 70 may also include a memory controller to control and manage the memory 704; the electronic device 70 may also include a peripheral interface to connect to a radio frequency module, an audio module, and a display screen, etc.

[0186] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the object storage service capacity determination method and apparatus in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the aforementioned object storage service capacity determination method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0187] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: acquiring log streams of at least one first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data modification operations on the data container in the object storage service; if a partition change is detected in the first service partition, the log stream of the service partition obtained from the partition change is merged into the log stream set; based on the data change information in the log entries in the log stream set, the used capacity information of the data container in the object storage service is determined.

[0188] Optionally, the processor may also execute program code that performs the following steps: before merging the log stream of the service partition obtained from the partition change into the log stream set, detecting whether there is a target log entry in the log stream, wherein the target log entry is a log entry that represents the partition change of the first service partition; if the target log entry is detected in the log stream, determining the log stream corresponding to the service partition obtained from the partition change based on the target log entry.

[0189] Optionally, the processor may also execute program code that performs the following steps: when a partition merge is detected in the first service partition and there are multiple second service partitions, the target first service partition is determined from the merged first service partitions based on the partition information of the merged first service partition; if the log stream of the target first service partition belongs to the log stream set, the log stream of the service partition obtained by the partition merge is incorporated into the log stream set.

[0190] Optionally, the processor may also execute program code that performs the following steps: determining the incremental capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream collection; and determining the used capacity information of the data container based on the incremental capacity information of the data container.

[0191] Optionally, the processor may also execute program code that performs the following steps: Based on the data change information in the log entries generated in the log stream set within the first time interval, determine the incremental capacity information of the data container in the object storage service within the first time interval according to the first statistical time interval, wherein the first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

[0192] Optionally, the processor may also execute program code that performs the following steps: based on the data change information in the log entries generated in the log stream set within the first time interval, determine the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained from partition changes; based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, determine the incremental capacity information of the data container within the first time interval.

[0193] Optionally, the processor may also execute program code with the following steps: when the incremental capacity information of the log stream set is statistically analyzed for the first time, the generation time of the first log entry in the log stream set is determined, and a first time interval is determined based on the generation time and the first statistical time interval; when the incremental capacity information of the log stream set is statistically analyzed for the Nth time, the first time interval corresponding to the Nth statistical analysis is determined based on the time termination point of the first time interval of the (N-1)th statistical analysis and the first statistical time interval, where N is a positive integer greater than 1.

[0194] Optionally, the processor may also execute program code that performs the following steps: based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, determine the used capacity information at the end point of the second time interval according to the second statistical time interval, wherein the second statistical time interval refers to the statistical time interval between the used capacity information of the data containers, and the time length of the second time interval is the same as the second statistical time interval.

[0195] Optionally, the processor may also execute program code that performs the following steps: after determining the incremental capacity information of the data container in the object storage service, the log consumption progress corresponding to the log stream set is recorded in the consumption progress information table; in the event of statistical anomalies in the incremental capacity information, if the statistical anomalies are detected to have been resolved, the incremental capacity information is further statistically analyzed based on the log consumption progress in the consumption progress information table and the log stream set.

[0196] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 7 This does not limit the structure of the aforementioned electronic device. For example, electronic device 70 may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0197] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0198] Example 4

[0199] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the capacity determination method of the object storage service provided in Embodiment 1.

[0200] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0201] Example 5

[0202] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the capacity determination method for the object storage service provided in Embodiment 1.

[0203] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0204] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0206] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0207] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0208] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0209] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining the capacity of an object storage service, characterized in that, include: Obtain log streams from at least one first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data modification operations on the data container in the object storage service; If a partition change is detected in the first service partition, the log stream of the service partition resulting from the partition change is merged into the log stream set. Based on the data change information in the log entries of the log stream set, the used capacity information of the data container in the object storage service is determined.

2. The method according to claim 1, characterized in that, Before incorporating the log streams of the service partitions obtained from the partition changes into the log stream set, the method further includes: Detect whether a target log entry exists in the log stream, wherein the target log entry is a log entry that indicates a partition change has occurred in the first service partition; If the target log entry is detected in the log stream, the log stream corresponding to the service partition obtained from the partition change is determined based on the target log entry.

3. The method according to claim 1, characterized in that, The log stream set is obtained through a second service partition, which is the service partition corresponding to the metering service in the object storage service. The partition change includes: partition merging. If there are multiple first service partitions, and a partition change is detected in the first service partition, the log stream of the service partition obtained from the partition change is merged into the log stream set, including: If a partition merge is detected in the first service partition and there are multiple second service partitions, the target first service partition is determined from the merged first service partitions based on the partition information of the merged first service partitions. If the log stream of the target first service partition belongs to the log stream set, the log stream of the service partition obtained by partition merging is incorporated into the log stream set.

4. The method according to claim 1, characterized in that, Based on the data change information in the log entries of the log stream collection, the used capacity information of the data container in the object storage service is determined as follows: Based on the data change information in the log entries in the log stream set, the incremental capacity information of the data container in the object storage service is determined; Based on the incremental capacity information of the data container, the used capacity information of the data container is determined.

5. The method according to claim 4, characterized in that, Based on the data change information in the log entries of the log stream set, the incremental capacity information of the data container in the object storage service is determined as follows: Based on the data change information in the log entries generated within the first time interval in the log stream set according to the first statistical time interval, the incremental capacity information of the data container in the object storage service within the first time interval is determined. The first statistical time interval refers to the statistical time interval between the incremental capacity information of the data container, and the time length of the first time interval is the same as the first statistical time interval.

6. The method according to claim 5, characterized in that, Based on the data change information in the log entries generated within the first time interval in the log stream set, the incremental capacity information of the data container in the object storage service within the first time interval is determined as follows: Based on the data change information in the log entries generated in the log stream set within the first time interval, the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval is determined, wherein the multiple target service partitions include: at least one first service partition and service partitions obtained by partition changes; Based on the incremental capacity sub-information of the data container under multiple target service partitions within the first time interval, the incremental capacity information of the data container within the first time interval is determined.

7. The method according to claim 5, characterized in that, The first time interval is determined in the following way: Based on the first statistical analysis of the incremental capacity information of the log stream set, the generation time of the first log entry in the log stream set is determined, and the first time interval is determined based on the generation time and the first statistical time interval. When the incremental capacity information is statistically analyzed for the Nth time based on the log stream set, the first time interval corresponding to the Nth statistical analysis is determined based on the time end point of the first time interval of the (N-1)th statistical analysis and the first statistical time interval, where N is a positive integer greater than 1.

8. The method according to claim 4, characterized in that, Based on the incremental capacity information of the data container, determining the used capacity information of the data container includes: According to the second statistical time interval, based on the used capacity information at the start point of the second time interval and the incremental capacity information corresponding to the second time interval, the used capacity information at the end point of the second time interval is determined. The second statistical time interval refers to the statistical time interval between the used capacity information of the data containers, and the time length of the second time interval is the same as the second statistical time interval.

9. The method according to claim 4, characterized in that, After determining the incremental capacity information of the data container in the object storage service, the method further includes: Record the log consumption progress corresponding to the log stream set in the consumption progress information table; In the event of statistical anomalies in the incremental capacity information, if the statistical anomalies have been resolved, the incremental capacity information will continue to be statistically analyzed based on the log stream set according to the log consumption progress in the consumption progress information table.

10. A capacity determination device for an object storage service, characterized in that, include: The acquisition unit is used to acquire at least one log stream of a first service partition to obtain a log stream set, wherein the first service partition is the service partition corresponding to the data index service in the object storage service, and the data index service is used to perform data modification operations on the data container in the object storage service; The first processing unit is configured to, if a partition change is detected in the first service partition, merge the log stream of the service partition resulting from the partition change into the log stream set. The first determining unit is used to determine the used capacity information of the data container in the object storage service based on the data change information in the log entries in the log stream set.

11. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the capacity determination method for the object storage service of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the capacity determination method for the object storage service of any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the capacity determination method for an object storage service according to any one of claims 1 to 9.

Images