Data aggregation decoupling method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data aggregation decoupling method and device, electronic equipment and a storage medium, and can solve the problems that the progress of local aggregation cannot be globally controlled, and the consistency of data views of all storage operation units cannot be ensured. The storage system comprises a main control unit and a plurality of storage operation units, the main control unit is connected with the storage operation units, and the storage operation units perform distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress and send real-time aggregation information to the main control unit, the real-time aggregation information at least comprises a real-time aggregation progress; the main control unit analyzes the multiple pieces of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress is the minimum progress information in the multiple pieces of real-time aggregation progress, and the target aggregation progress is sent to each storage operation unit; and the storage operation unit determines a target aggregation interval according to the target aggregation progress, and continues to perform local aggregation according to the target aggregation interval.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of data aggregation decoupling, and in particular, to a data aggregation decoupling method, device, electronic device and storage medium. Background Art

[0002] In the storage system, there are multiple functions that are independent of each other but coupled with each other. They can be roughly divided into two categories, including local operation logic and distributed operation logic. Local operation logic, such as the local aggregation function, is designed to recycle garbage data on the storage operation unit, so it does not involve interaction with other operation units. Distributed operation logic, such as the EC aggregation function, is a distributed operation logic. During the operation, a certain operation unit needs to read data from other operation units, which requires reliance on the state consistency of global data. Therefore, the progress of local aggregation cannot be controlled globally, resulting in the inability to strictly ensure the consistency of the progress of each storage operation unit, and may also cause data anomalies when distributed aggregation obtains data. Summary of the invention

[0003] Based on this, it is necessary to provide a data aggregation decoupling method, device, electronic device and storage medium to address the above technical problems.

[0004] In a first aspect, an embodiment of the present application provides a data aggregation decoupling method, which is applied to a main control unit, wherein the storage system includes: the main control unit and multiple storage operation units, wherein the main control unit and the multiple storage operation units are respectively connected, and the data aggregation decoupling method includes: receiving real-time aggregation information respectively sent by each storage operation unit, and obtaining multiple real-time aggregation information, wherein the real-time aggregation information at least includes: real-time aggregation progress;

[0005] Parsing the multiple real-time aggregation information to obtain a target aggregation progress, where the target aggregation progress is the minimum progress information among the multiple real-time aggregation progress;

[0006] The target aggregation progress is sent to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation progress.

[0007] As an optional implementation, in the first aspect of the embodiment of the present application, the real-time aggregation information further includes: real-time version information; the parsing of the multiple real-time aggregation information to obtain the target aggregation progress includes:

[0008] Detecting whether the real-time version information corresponding to each storage operation unit is the same;

[0009] When it is detected that the real-time version information corresponding to all storage operation units is the same, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0010] As an optional implementation, in the first aspect of the embodiment of the present application, the real-time aggregation information further includes: identity information of the storage operation unit; the parsing of the multiple real-time aggregation information to obtain the target aggregation progress includes:

[0011] According to the identity information, detecting whether the real-time aggregate information sent by all storage operation units is received;

[0012] When it is detected that the real-time aggregation information sent by all the storage operation units has been received, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0013] In a second aspect, an embodiment of the present application provides a data aggregation decoupling method, which is applied to a storage operation unit, wherein the storage system includes: a main control unit and multiple storage operation units, wherein the main control unit and the multiple storage operation units are respectively connected to each other, and the data aggregation decoupling method includes: performing distributed aggregation according to an initial aggregation interval to obtain real-time aggregation progress;

[0014] Sending real-time aggregation information to the main control unit, the real-time aggregation information at least including: real-time aggregation progress;

[0015] receiving the target aggregation progress sent by the main control unit, and determining the target aggregation interval according to the target aggregation progress;

[0016] Continue local aggregation according to the target aggregation interval.

[0017] As an optional implementation manner, in the second aspect of the embodiment of the present application, the real-time aggregation information further includes: real-time version information; the sending of the real-time aggregation information to the main control unit includes:

[0018] Obtaining the real-time version information after distributed aggregation;

[0019] The real-time aggregation progress and the real-time version information are sent to the main control unit.

[0020] As an optional implementation, in the second aspect of the embodiment of the present application, the real-time aggregation information also includes: identity information of the storage operation unit.

[0021] As an optional implementation manner, in the second aspect of the embodiment of the present application, determining the target aggregation interval according to the target aggregation progress includes:

[0022] Determining a target aggregation upper limit according to the target aggregation progress;

[0023] Determine the target aggregation lower limit based on the aggregation interval of adjacent historical local aggregations;

[0024] The target aggregation interval is determined according to the target aggregation upper limit and the target aggregation lower limit.

[0025] As an optional implementation manner, in the second aspect of the embodiment of the present application, the continuing to perform local aggregation according to the target aggregation interval includes:

[0026] Detecting whether the target aggregation progress satisfies a first preset condition, and detecting whether the target aggregation interval satisfies a second preset condition;

[0027] When it is detected that the target aggregation progress satisfies the first preset condition and the target aggregation interval satisfies the second preset condition, local aggregation is continued according to the target aggregation interval.

[0028] In a third aspect, an embodiment of the present application provides a data aggregation decoupling device, which is applied to a main control unit. The storage system includes: the main control unit and multiple storage operation units, and the main control unit and the multiple storage operation units are respectively connected. The data aggregation decoupling device includes:

[0029] The transceiver module is used to receive the real-time aggregation information sent by each storage operation unit, and obtain a plurality of real-time aggregation information, wherein the real-time aggregation information at least includes: real-time aggregation progress;

[0030] A processing module, configured to parse the plurality of real-time aggregation information to obtain a target aggregation progress, wherein the target aggregation progress is the minimum progress information among the plurality of real-time aggregation progress;

[0031] The transceiver module is further used to send the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation progress.

[0032] In a fourth aspect, an embodiment of the present application provides a data aggregation decoupling device, which is applied to a storage operation unit. The storage system includes: a main control unit and multiple storage operation units, and the main control unit and the multiple storage operation units are respectively connected. The data aggregation decoupling device includes:

[0033] A processing module is used to perform distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress;

[0034] A transceiver module, configured to send real-time aggregation information to the main control unit, wherein the real-time aggregation information at least includes: real-time aggregation progress;

[0035] The transceiver module is further used to receive the target aggregation progress sent by the main control unit, and determine the target aggregation interval according to the target aggregation progress;

[0036] The processing module is further configured to continue local aggregation according to the target aggregation interval.

[0037] In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0038] A memory storing executable program code;

[0039] a processor coupled to the memory;

[0040] The processor calls the executable program code stored in the memory to execute the data aggregation decoupling method described in the first aspect or the second aspect of the embodiment of the present application.

[0041] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the data aggregation decoupling method described in the first aspect or the second aspect of the embodiment of the present application. The computer-readable storage medium includes a ROM / RAM, a magnetic disk or an optical disk, etc.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute part or all of the steps of any one of the methods of the first aspect.

[0043] In a sixth aspect, an embodiment of the present application provides an application publishing platform, which is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps of any one method of the first aspect.

[0044] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0045] The embodiments of the present application provide a data aggregation decoupling method, device, electronic device and storage medium, wherein the storage system includes: a main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, the storage operation units perform distributed aggregation according to the initial aggregation interval, obtain real-time aggregation progress, and send real-time aggregation information to the main control unit, the real-time aggregation information at least includes: real-time aggregation progress; the main control unit parses the multiple real-time aggregation information to obtain a target aggregation progress, the target aggregation progress is the smallest progress information among the multiple real-time aggregation progresses, and sends the target aggregation progress to each storage operation unit; the storage operation unit determines the target aggregation interval according to the target aggregation progress, and continues to perform local aggregation according to the target aggregation interval. In this solution, the behavior of local aggregation is controlled by setting an upper limit on the local aggregation range. Each storage independent operation unit collects the global progress of distributed aggregation and distributes the minimum aggregation progress to each storage operation unit. The local aggregation logic of the storage operation unit uses this progress as its aggregation upper limit. Although local aggregation and distributed aggregation operate at the same time, their operation ranges are different, which can achieve fine decoupling of local aggregation and distributed aggregation, thereby achieving non-interference between local aggregation and distributed aggregation functions. The local aggregation function also operates normally during distributed aggregation or reconstruction, ensuring the normal operation of the garbage collection function and eliminating capacity risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a flow diagram of a data aggregation decoupling method provided in an embodiment of the present application. Figure 1 ;

[0049] Figure 2 This is a flow diagram of a data aggregation decoupling method provided in an embodiment of the present application. Figure 2 ;

[0050] Figure 3 This is an operational architecture diagram of a storage system data aggregation decoupling provided by an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the structure of a data aggregation decoupling device provided in an embodiment of the present application. Figure 1 ;

[0052] Figure 5 This is a schematic diagram of the structure of a data aggregation decoupling device provided in an embodiment of the present application. Figure 2 ;

[0053] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The terms “first”, “second” and the like in the description and claims of the present application are used to distinguish different objects rather than to describe a specific order of the objects.

[0056] The terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0057] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] In the storage system, there are multiple functions that are independent of each other but coupled with each other. They can be roughly divided into two categories, including local operation logic and distributed operation logic. Local operation logic, such as the local aggregation function, is designed to recycle garbage data on the storage operation unit, so it does not involve interaction with other operation units. Distributed operation logic, such as the EC aggregation function, is a distributed operation logic. During the operation, a certain operation unit needs to read data from other operation units, so it relies on the consistency of the global data state. However, local aggregation is a single-point operation, and it does not perceive the operation status of other data shards. This may destroy the consistency of the global data state, thereby affecting the normal operation of distributed aggregation.

[0059] It should be noted that local aggregation is a business logic that runs regularly on an independent operation unit. It scans the metadata information of each object locally, analyzes and calculates the metadata, outputs garbage information, and performs garbage collection and data merging based on the information, thereby achieving the purpose of data cleaning, garbage collection, and freeing up space. This process only relies on its own metadata and does not perceive the global operation status. Local aggregation is usually used to reduce data transmission and improve processing efficiency, especially in data-intensive processing scenarios, such as the mapper stage in MapReduce jobs. Data is initially merged on the local node, reducing the amount of data transmission in subsequent processing stages. It is suitable for scenarios that require fast response because data can be processed without being transmitted to remote nodes. It can be achieved by using functions such as Hadoop Combiner to optimize the performance of MapReduce jobs.

[0060] It should be noted that distributed aggregation is a distributed operation logic, which is executed by the master of the object. By reading remote data, new low-space-occupying data is generated to achieve the purpose of data compression and merging to reduce global space occupancy. This function needs to read data from multiple remote ends, and multiple remote data need to be in the same view. Therefore, this function depends on the view of global data. Figure 1 Distributed aggregation is often used to process large-scale data sets or tasks that require cross-node collaboration. Data is transferred and merged between multiple nodes to achieve global data processing. It is suitable for processing large-scale data sets or scenarios that require cross-node collaboration. It can be implemented through distributed computing frameworks (such as Hadoop, Spark, etc.).

[0061] It should be noted that, comparing local aggregation and distributed aggregation, local aggregation is mainly processed within a single node or a local range, while distributed aggregation involves data processing between multiple nodes; in addition, local aggregation reduces the amount of data transmission by reducing the amount of data of intermediate results, while distributed aggregation may require the transmission of large amounts of data for global merging; and local aggregation is suitable for scenarios that require fast response or processing of small-scale data sets, while distributed aggregation is suitable for scenarios that process large-scale data sets or require cross-node collaboration.

[0062] To solve the above coupling, the two operation logics can be made absolutely mutually exclusive, that is, during the operation of the distributed aggregation logic, local aggregation will not be performed, and it will be performed after the distributed aggregation is completed. However, the existing strategy still has the following problems: local aggregation depends on the completion of distributed aggregation tasks, and a large amount of data cannot be recovered during the aggregation period, and there may be a risk of data space being filled up in high-capacity scenarios; in addition, the progress of local aggregation cannot be globally controlled, resulting in the progress of each storage operation unit cannot strictly guarantee the consistency of pace, which will still cause data anomalies when distributed aggregation obtains data.

[0063] In order to solve some or all of the above-mentioned technical problems, the embodiments of the present application provide a data aggregation decoupling method, device, electronic device and storage medium, wherein the storage system includes: a main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are connected respectively, the storage operation units perform distributed aggregation according to the initial aggregation interval, obtain real-time aggregation progress, and send real-time aggregation information to the main control unit, the real-time aggregation information at least includes: real-time aggregation progress; the main control unit parses the multiple real-time aggregation information to obtain a target aggregation progress, the target aggregation progress is the smallest progress information among the multiple real-time aggregation progresses, and sends the target aggregation progress to each storage operation unit; the storage operation unit determines the target aggregation interval according to the target aggregation progress, and continues to perform local aggregation according to the target aggregation interval. In this solution, the behavior of local aggregation is controlled by setting an upper limit on the local aggregation range. Each storage independent operation unit collects the global progress of distributed aggregation and distributes the minimum aggregation progress to each storage operation unit. The local aggregation logic of the storage operation unit uses this progress as its aggregation upper limit. Although local aggregation and distributed aggregation operate at the same time, their operation ranges are different, which can achieve fine decoupling of local aggregation and distributed aggregation, thereby achieving non-interference between local aggregation and distributed aggregation functions. The local aggregation function also operates normally during distributed aggregation or reconstruction, ensuring the normal operation of the garbage collection function and eliminating capacity risks.

[0064] like Figure 1 As shown, Figure 1A flowchart of a data aggregation decoupling method provided in an embodiment of the present application, the data aggregation decoupling method is implemented by interaction between a main control unit and a storage operation unit, the storage system may specifically include: a main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, and the multiple storage operation units are independent of each other, the storage operation unit can be understood as a single disk, or a functional entity in the disk for data storage and processing, and the main control unit is a management functional entity one layer higher than the storage operation unit, the method may include the following steps:

[0065] 101. The storage operation unit performs distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress.

[0066] In the embodiment of the present application, each initial aggregation interval can first complete the distributed aggregation by itself, and the aggregation interval of the distributed aggregation can be understood as dynamically updated. The initial aggregation interval can be understood as an interval determined according to the last distributed aggregation and the instruction of the main control unit, which can be represented by [AB]. After the storage operation unit completes the distributed aggregation according to the [AB] interval, it will refresh the aggregation progress Agg_progress to B, which can be understood as the real-time aggregation progress is B.

[0067] In some embodiments, each storage operation unit in the storage system performs distributed aggregation and refreshes the aggregation progress of the storage operation unit after completion. The aggregation progress of each storage operation unit is determined according to its own aggregation interval and can be different progress.

[0068] 102. The storage operation unit sends real-time aggregate information to the main control unit.

[0069] In the embodiment of the present application, the real-time aggregation information may at least include: real-time aggregation progress. After completing the distributed aggregation, each storage operation unit may report the real-time aggregation progress to the main control unit.

[0070] 103. The main control unit receives the real-time aggregate information sent by each storage operation unit, and obtains multiple real-time aggregate information.

[0071] In an embodiment of the present application, after the main control unit receives the real-time aggregation information sent by each storage operation unit, it can summarize multiple real-time aggregation information, that is, it can be understood as associating multiple real-time aggregation information with multiple storage operation units and saving them in a map.

[0072] 104. The main control unit analyzes multiple real-time aggregation information to obtain target aggregation progress.

[0073] In the embodiment of the present application, since distributed aggregation needs to ensure global data visibility Figure 1 Therefore, the main control unit needs to control the global progress through the real-time aggregation progress of each storage operation unit. Therefore, the main control unit can parse multiple real-time aggregation information to obtain the target aggregation progress, which can be the smallest progress information among multiple real-time aggregation progress.

[0074] It should be noted that the aggregation progress of each storage operation unit may be inconsistent. In order to ensure that the data of all storage operation units is consistent, Figure 1 In order to avoid aggregation progress loopholes in some storage operation units, it is necessary to select the aggregation progress with the smallest progress as the global progress.

[0075] 105. The main control unit sends the target aggregation schedule to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation schedule.

[0076] In an embodiment of the present application, the main control unit can send the target aggregation progress to each storage operation unit, so as to ensure that the aggregation progress of each storage operation unit is consistent.

[0077] 106. The storage operation unit receives the target aggregation progress sent by the main control unit, and determines the target aggregation interval according to the target aggregation progress.

[0078] In the embodiment of the present application, after receiving the target aggregation progress, the storage operation unit may determine a new aggregation interval, the upper limit of which may be determined according to the target aggregation progress, and the lower limit may be determined according to the interval of the last aggregation.

[0079] 107. The storage operation unit continues to perform local aggregation according to the target aggregation range.

[0080] In the embodiment of the present application, after the storage operation unit determines the target aggregation interval, it can determine the target aggregation interval as the aggregation interval for local aggregation, and perform local aggregation according to the target aggregation interval.

[0081] It should be noted that the target aggregation interval of local aggregation and the initial aggregation interval of distributed aggregation are different intervals, and the upper limit of local aggregation will not exceed the aggregation progress of distributed aggregation.

[0082] An embodiment of the present application provides a data aggregation decoupling method, which controls the behavior of local aggregation by setting an upper limit of the local aggregation interval. Each storage independent operation unit collects the global progress of distributed aggregation and distributes the minimum aggregation progress to each storage operation unit. The local aggregation logic of the storage operation unit uses the progress as its aggregation upper limit. Although local aggregation and distributed aggregation operate at the same time, their operation intervals are different. It is possible to achieve fine-grained decoupling of local aggregation and distributed aggregation, thereby achieving non-interference between local aggregation functions and distributed aggregation functions. The local aggregation function also operates normally during distributed aggregation or reconstruction, ensuring the normal operation of the garbage collection function and eliminating capacity risks.

[0083] like Figure 2 As shown, Figure 2 A flowchart of a data aggregation decoupling method provided in an embodiment of the present application, wherein the data aggregation decoupling method is implemented by interaction between a main control unit and a storage operation unit, and the method may further include the following steps:

[0084] 201. The storage operation unit performs distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress.

[0085] 202. The storage operation unit sends real-time aggregation information to the main control unit.

[0086] In some embodiments, the real-time aggregation information may also include: real-time version information; sending the real-time aggregation information to the main control unit may specifically include: obtaining the real-time version information after distributed aggregation; sending the real-time aggregation progress and real-time version information to the main control unit.

[0087] In the embodiment of the present application, the version information of the storage operation unit will also be updated after each aggregation. In order to ensure that the data of all storage operation units is Figure 1 To ensure consistency, it is necessary to ensure that the version information of all storage operation units is consistent. Therefore, after the storage operation unit completes the aggregation and refreshes the aggregation progress, it can send the real-time aggregation progress and real-time version information to the main control unit for analysis.

[0088] In some embodiments, the real-time aggregate information may also include: identity information of the storage operation unit.

[0089] In an embodiment of the present application, since the main control unit needs to summarize the aggregation progress of each storage operation unit and needs to ensure that the aggregation progress of each storage operation unit has been received, the storage operation unit can send the real-time aggregation progress and identity information to the main control unit, so that the main control unit can detect whether it has received the aggregation progress of all storage operation units.

[0090] In some embodiments, the real-time aggregation information may at least include the real-time aggregation progress. According to the above embodiments, the real-time aggregation information may also include the real-time version information, and the real-time aggregation information may also include the identity information; of course, the real-time version information and the identity information are in an and / or relationship, that is, the real-time aggregation information may include: the real-time aggregation progress and the real-time version information, or the real-time aggregation information may include: the real-time aggregation progress and the identity information, or the real-time aggregation information may include: the real-time aggregation progress, the real-time version information and the identity information.

[0091] 203. The main control unit receives the real-time aggregate information sent by each storage operation unit, and obtains multiple real-time aggregate information.

[0092] 204. The main control unit analyzes multiple real-time aggregation information to obtain target aggregation progress.

[0093] In some embodiments, when the real-time aggregation information also includes: real-time version information, multiple real-time aggregation information are parsed to obtain the target aggregation progress, which may specifically include: detecting whether the real-time version information corresponding to each storage operation unit is the same; when it is detected that the real-time version information corresponding to all storage operation units is the same, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0094] It should be noted that the main control unit will only analyze multiple real-time aggregation progress to obtain the target aggregation progress when it detects that the version information of all storage operation units is the same; if the version information of some storage operation units is different, the main control unit will not perform any operation until the main control unit receives real-time aggregation information including the same version information, and then it will analyze all real-time aggregation progress to obtain the target aggregation progress.

[0095] In some embodiments, when the real-time aggregation information also includes: the identity information of the storage operation unit, multiple real-time aggregation information are parsed to obtain the target aggregation progress, which may specifically include: detecting whether the real-time aggregation information sent by all storage operation units is received according to the identity information; when it is detected that the real-time aggregation information sent by all storage operation units has been received, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0096] It should be noted that after receiving the identity information and real-time aggregation progress of the storage operation unit, the main control unit can store the identity information and the real-time aggregation progress accordingly. That is, the identity information of the storage operation unit is used as the key value of the map, and its corresponding value is the real-time aggregation progress reported at the same time.

[0097] In some embodiments, when the real-time aggregation information includes real-time aggregation progress, real-time version information and identity information at the same time, the identity information of the storage operation unit is used as the key value of the map, and its corresponding value is the real-time aggregation progress information and real-time version information reported at the same time.

[0098] 205. The main control unit sends the target aggregation schedule to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation schedule.

[0099] 206. The storage operation unit receives the target aggregation progress sent by the main control unit.

[0100] 207. The storage operation unit determines a target aggregation upper limit according to the target aggregation progress.

[0101] In an embodiment of the present application, when determining the target aggregation upper limit, it is necessary to consider that the aggregation intervals of local aggregation and distributed aggregation cannot overlap, and the local aggregation cannot exceed the distributed aggregation. In other words, the target aggregation upper limit needs to be less than the target aggregation progress. Therefore, the target aggregation progress can be subtracted from the preset value to obtain the target aggregation upper limit.

[0102] 208. The storage operation unit determines a target aggregation lower limit according to an aggregation interval of adjacent historical local aggregations.

[0103] In the embodiment of the present application, since data aggregation is continuous, the aggregation interval is also continuous, and the upper limit of the previous local aggregation can be determined as the lower limit of this local aggregation, that is, the target aggregation lower limit.

[0104] 209. The storage operation unit determines a target aggregation range according to the target aggregation upper limit and the target aggregation lower limit.

[0105] 210. The storage operation unit detects whether the target aggregation progress satisfies a first preset condition, and detects whether the target aggregation interval satisfies a second preset condition.

[0106] In the embodiment of the present application, the first preset condition may be greater than the upper limit of the previous local aggregation, and the second preset condition may be non-overlapping with the initial aggregation interval, and of course, it may also be other customized conditions.

[0107] 211. When the storage operation unit detects that the target aggregation progress meets the first preset condition and the target aggregation interval meets the second preset condition, the storage operation unit continues to perform local aggregation according to the target aggregation interval.

[0108] In the embodiment of the present application, when it is detected that the target aggregation progress is greater than the upper limit of the previous local aggregation, and the target aggregation interval does not overlap with the initial aggregation interval, local aggregation is continued according to the target aggregation interval.

[0109] An embodiment of the present application provides a data aggregation decoupling method, which controls the behavior of local aggregation by setting an upper limit of the local aggregation interval. Each storage operation unit collects the global progress of distributed aggregation and distributes the minimum aggregation progress to each storage operation unit. The local aggregation logic of the storage operation unit uses the progress as its aggregation upper limit. Although local aggregation and distributed aggregation operate at the same time, their operation intervals are different. It is possible to achieve fine-grained decoupling of local aggregation and distributed aggregation, thereby achieving non-interference between local aggregation functions and distributed aggregation functions. The local aggregation function also operates normally during distributed aggregation or reconstruction, ensuring the normal operation of the garbage collection function and eliminating capacity risks.

[0110] In some embodiments, Figure 3 The following is an operational architecture diagram of the storage system for data aggregation decoupling. Two new operational modules are added: aggregation progress reporting and refreshing module and aggregation progress analysis and refreshing module. By adding the above two modules, local aggregation and distributed aggregation can be decoupled, and each can run in its own scope without interfering with each other.

[0111] The aggregation progress reporting and refreshing module is used for regular reporting of the aggregation progress and refreshing the upper limit of the local aggregation after receiving the message. The lower limit of the interval of distributed aggregation is the upper limit of the last aggregation, and the upper limit of the interval of aggregation is the current IO completion timestamp; after the current aggregation is completed according to the interval, the local distributed aggregation progress will be refreshed, that is, progress_N (real-time aggregation progress) will be equal to EC_high_epoch (the upper limit of the initial aggregation interval). The progress reporting module will regularly assemble the relevant information of this storage operation unit (operation unit ID + aggregation progress + current version information) and send it to the main control unit.

[0112] The aggregation progress analysis and refresh module is used to store the aggregation progress and refresh it after analysis. This module runs on the main control unit. After receiving messages from other storage operation units, it saves the relevant information (operation unit ID + aggregation progress + current version information) to the local map memory. The main control unit regularly parses the map information and traverses the information of each storage operation unit. If the version information reported by each unit is consistent, it is considered that all current operation units are reporting normally. Then the minimum distributed aggregation progress of each operation unit is obtained as the refresh aggregation progress. If the current refresh value is consistent with the last time, the current refresh is abandoned. If there is an update, it is refreshed. If Figure 3 As shown in , the main control unit modifies the local aggregation upper limit vos-high-epoch according to the aggregation progress progress-A, progress-B and progress-C reported by the three storage operation units respectively, and sends it to the three storage operation units respectively.

[0113] After the aggregation progress reporting and refresh module receives the new refresh progress and determines that the value meets the preset conditions, it will update the upper limit of the local aggregation to ensure that the local aggregation can operate normally. Ultimately, the scope of distributed aggregation and local aggregation is continuously and dynamically updated, and the range of local aggregation will not exceed the set upper limit, achieving mutual exclusion between the two and solving the coupling problem of distributed aggregation and local aggregation.

[0114] like Figure 4 As shown, an embodiment of the present application provides a data aggregation decoupling device, which is applied to a main control unit. The storage system includes: a main control unit and multiple storage operation units, and the main control unit and the multiple storage operation units are respectively connected. The data aggregation decoupling device may include:

[0115] The transceiver module 401 is used to receive the real-time aggregation information sent by each storage operation unit, and obtain multiple real-time aggregation information, where the real-time aggregation information at least includes: real-time aggregation progress;

[0116] The processing module 402 is used to parse the multiple real-time aggregation information to obtain a target aggregation progress, where the target aggregation progress is the minimum progress information among the multiple real-time aggregation progress;

[0117] The transceiver module 401 is further configured to send the target aggregation schedule to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation schedule.

[0118] In some embodiments, the real-time aggregate information further includes: real-time version information;

[0119] The processing module 402 is specifically used to detect whether the real-time version information corresponding to each storage operation unit is the same;

[0120] The processing module 402 is specifically configured to analyze a plurality of real-time aggregation progresses to obtain a target aggregation progress when it is detected that the real-time version information corresponding to all storage operation units is the same.

[0121] In some embodiments, the real-time aggregate information further includes: identity information of the storage operation unit;

[0122] The processing module 402 is specifically used to detect whether the real-time aggregation information sent by all storage operation units is received according to the identity information;

[0123] The processing module 402 is specifically configured to analyze multiple real-time aggregation progresses to obtain a target aggregation progress when it is detected that the real-time aggregation information sent by all storage operation units has been received.

[0124] In the embodiments of the present application, each module can implement the data aggregation decoupling method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0125] like Figure 5 As shown, an embodiment of the present application provides a data aggregation decoupling device, which is applied to a storage operation unit. The storage system includes: a main control unit and multiple storage operation units, and the main control unit and the multiple storage operation units are respectively connected. The data aggregation decoupling device may include:

[0126] Processing module 501, used to perform distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress;

[0127] The transceiver module 402 is used to send real-time aggregation information to the main control unit, where the real-time aggregation information at least includes: real-time aggregation progress;

[0128] The transceiver module 502 is further used to receive the target aggregation progress sent by the main control unit, and determine the target aggregation interval according to the target aggregation progress;

[0129] The processing module 501 is further configured to continue to perform local aggregation according to the target aggregation interval.

[0130] In some embodiments, the real-time aggregate information further includes: real-time version information;

[0131] Processing module 501, specifically used to obtain real-time version information after distributed aggregation;

[0132] The transceiver module 502 is specifically used to send the real-time aggregation progress and real-time version information to the main control unit.

[0133] In some embodiments, the real-time aggregate information further includes: identity information of the storage operation unit.

[0134] In some embodiments, the processing module 501 is specifically configured to determine a target aggregation upper limit according to the target aggregation progress;

[0135] The processing module 501 is specifically configured to determine a target aggregation lower limit according to an aggregation interval of adjacent historical local aggregations;

[0136] The processing module 501 is specifically configured to determine a target aggregation interval according to a target aggregation upper limit and a target aggregation lower limit.

[0137] In some embodiments, the processing module 501 is specifically used to detect whether the target aggregation progress meets the first preset condition, and detect whether the target aggregation interval meets the second preset condition;

[0138] The processing module 501 is specifically configured to continue to perform local aggregation according to the target aggregation interval when it is detected that the target aggregation progress meets a first preset condition and the target aggregation interval meets a second preset condition.

[0139] In the embodiments of the present application, each module can implement the data aggregation decoupling method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] like Figure 6 As shown, an embodiment of the present application further provides an electronic device, which may include:

[0141] A memory 601 storing executable program codes;

[0142] a processor 602 coupled to the memory 601;

[0143] The processor 602 calls the executable program code stored in the memory 601 to execute the data aggregation decoupling method performed by the electronic device in the above-mentioned method embodiments.

[0144] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the data aggregation decoupling method in the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] An embodiment of the present application also provides a computer program product, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the data aggregation decoupling method in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0147] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0149] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0150] In this application, a person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the computer-readable medium includes permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, Parallel Random Access Memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0152] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application. The above-mentioned multiple embodiments are not necessarily multiple independent embodiments. Dividing into multiple embodiments is only used to highlight the different technical features in different embodiments. Those skilled in the art should be aware that the above-mentioned multiple embodiments can also be combined in any way.

[0153] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0154] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0157] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data aggregation decoupling method, characterized in that: Applied to a main control unit, the storage system includes: the main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, and the method includes: Receiving the real-time aggregation information sent by each storage operation unit, and obtaining a plurality of real-time aggregation information, wherein the real-time aggregation information at least includes: real-time aggregation progress; Parsing the multiple real-time aggregation information to obtain a target aggregation progress, where the target aggregation progress is the minimum progress information among the multiple real-time aggregation progress; The target aggregation progress is sent to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation progress.

2. The method according to claim 1, characterized in that The real-time aggregation information also includes: real-time version information; the parsing of the multiple real-time aggregation information to obtain the target aggregation progress includes: Detecting whether the real-time version information corresponding to each storage operation unit is the same; When it is detected that the real-time version information corresponding to all storage operation units is the same, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

3. The method according to claim 1, characterized in that The real-time aggregation information also includes: identity information of the storage operation unit; the parsing of the plurality of real-time aggregation information to obtain the target aggregation progress includes: According to the identity information, detecting whether the real-time aggregate information sent by all storage operation units is received; When it is detected that the real-time aggregation information sent by all the storage operation units has been received, multiple real-time aggregation progresses are analyzed to obtain the target aggregation progress.

4. A data aggregation decoupling method, characterized in that: Applied to a storage operation unit, the storage system includes: a main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, and the method includes: Perform distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress; Sending real-time aggregation information to the main control unit, the real-time aggregation information at least including: real-time aggregation progress; receiving the target aggregation progress sent by the main control unit, and determining the target aggregation interval according to the target aggregation progress; Continue local aggregation according to the target aggregation interval.

5. The method according to claim 4, characterized in that The real-time aggregate information further includes: real-time version information; the sending of the real-time aggregate information to the main control unit includes: Obtaining the real-time version information after distributed aggregation; The real-time aggregation progress and the real-time version information are sent to the main control unit.

6. The method according to claim 4, characterized in that The real-time aggregation information also includes: identity information of the storage operation unit.

7. The method according to claim 4, characterized in that The determining a target aggregation interval according to the target aggregation progress includes: Determining a target aggregation upper limit according to the target aggregation progress; Determine the target aggregation lower limit based on the aggregation interval of adjacent historical local aggregations; The target aggregation interval is determined according to the target aggregation upper limit and the target aggregation lower limit.

8. The method according to claim 4, characterized in that The continuing to perform local aggregation according to the target aggregation interval includes: Detecting whether the target aggregation progress satisfies a first preset condition, and detecting whether the target aggregation interval satisfies a second preset condition; When it is detected that the target aggregation progress satisfies the first preset condition and the target aggregation interval satisfies the second preset condition, local aggregation is continued according to the target aggregation interval.

9. A data aggregation decoupling device, characterized in that: Applied to a main control unit, the storage system includes: the main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, and the device includes: The transceiver module is used to receive the real-time aggregation information sent by each storage operation unit, and obtain a plurality of real-time aggregation information, wherein the real-time aggregation information at least includes: real-time aggregation progress; A processing module, configured to parse the plurality of real-time aggregation information to obtain a target aggregation progress, wherein the target aggregation progress is the minimum progress information among the plurality of real-time aggregation progress; The transceiver module is further used to send the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation progress.

10. A data aggregation decoupling device, characterized in that: Applied to a storage operation unit, the storage system includes: a main control unit and multiple storage operation units, the main control unit and the multiple storage operation units are respectively connected, and the device includes: A processing module is used to perform distributed aggregation according to the initial aggregation interval to obtain real-time aggregation progress; A transceiver module, configured to send real-time aggregation information to the main control unit, wherein the real-time aggregation information at least includes: real-time aggregation progress; The transceiver module is further used to receive the target aggregation progress sent by the main control unit, and determine the target aggregation interval according to the target aggregation progress; The processing module is further configured to continue local aggregation according to the target aggregation interval.

11. An electronic device, characterized in that: The electronic device comprises: A memory storing executable program code; and a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the data aggregation decoupling method as described in any one of claims 1 to 3 or any one of claims 4 to 8.

12. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the data aggregation decoupling method as described in any one of claims 1 to 3 or any one of claims 4 to 8 is implemented.

Images