Data aggregation decoupling method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the application disclose a data aggregation decoupling method and device, electronic equipment and storage medium, which can solve the problem that the progress of local aggregation cannot be globally controlled and the consistency of data views of each storage operation unit cannot be guaranteed. The storage system comprises a master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are connected respectively, the storage operation units perform distributed aggregation according to an initial aggregation interval, obtain a real-time aggregation progress, and send real-time aggregation information to the master control unit, the real-time aggregation information at least comprises the real-time aggregation progress; the master control unit analyzes a plurality of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress is the smallest progress information in the plurality of real-time aggregation progress, and the target aggregation progress is sent to each storage operation unit; and the storage operation unit determines a target aggregation interval according to the target aggregation progress, and continues local aggregation according to the target aggregation interval.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data aggregation decoupling, and particularly relate to a data aggregation decoupling method and device, electronic equipment and storage medium. BACKGROUND

[0002] In a storage system, there are multiple functions that are independent of each other but are coupled to each other, which are generally divided into two categories, including local operation logic and distributed operation logic. The local operation logic, such as a local aggregation function, is to recover garbage data on the local storage operation unit, and thus does not involve interaction with other operation units. The distributed operation logic, such as an EC aggregation function, is a distributed operation logic, and in the operation process, a certain operation unit needs to read data from other operation units, which requires the consistency of the global data state. Therefore, the progress of the local aggregation cannot be globally controlled, which leads to the fact that the progress of each storage operation unit cannot be strictly guaranteed to be consistent, and may also cause data anomalies when the distributed aggregation obtains data. SUMMARY

[0003] Therefore, it is necessary to provide a data aggregation decoupling method and device, electronic equipment and storage medium to solve the above technical problems.

[0004] In a first aspect, the embodiments of the present application provide a data aggregation decoupling method applied to a master control unit, and a storage system includes the master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are connected respectively, and the data aggregation decoupling method includes: receiving real-time aggregation information respectively sent by each storage operation unit to obtain a plurality of real-time aggregation information, the real-time aggregation information at least including real-time aggregation progress;

[0005] analyzing the plurality of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress being the smallest progress information in the plurality of real-time aggregation progress;

[0006] sending the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval according to the target aggregation progress and performs aggregation.

[0007] As an optional implementation, in the first aspect of the embodiments of the present application, the real-time aggregation information further includes real-time version information, and the analyzing the plurality of real-time aggregation information to obtain a target aggregation progress includes:

[0008] detecting whether the real-time version information corresponding to each storage operation unit is the same;

[0009] In a case where it is detected that the real-time version information corresponding to all the storage operation units is the same, a plurality of real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0010] As an optional implementation, in the first aspect of the embodiment of the present application, the real-time aggregation information further comprises: identity information of the storage operation unit; and the analyzing the plurality of real-time aggregation information to obtain the target aggregation progress comprises:

[0011] According to the identity information, it is detected whether the real-time aggregation information sent by all the storage operation units is received;

[0012] In a case where it is detected that the real-time aggregation information sent by all the storage operation units is received, a plurality of real-time aggregation progresses are analyzed to obtain the target aggregation progress.

[0013] In a second aspect, the embodiment of the present application provides a data aggregation decoupling method applied to a storage operation unit, and a storage system comprises a master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are connected respectively, and the data aggregation decoupling method comprises: performing distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress;

[0014] Sending real-time aggregation information to the master control unit, the real-time aggregation information at least comprising: the real-time aggregation progress;

[0015] Receiving a target aggregation progress sent by the master control unit, and determining a target aggregation interval according to the target aggregation progress;

[0016] Continuing local aggregation according to the target aggregation interval.

[0017] As an optional implementation, in the second aspect of the embodiment of the present application, the real-time aggregation information further comprises: real-time version information; and the sending the real-time aggregation information to the master control unit comprises:

[0018] Obtaining the real-time version information after distributed aggregation;

[0019] Sending the real-time aggregation progress and the real-time version information to the master control unit.

[0020] As an optional implementation, in the second aspect of the embodiment of the present application, the real-time aggregation information further comprises: identity information of the storage operation unit.

[0021] As an optional implementation, in the second aspect of the embodiment of the present application, the determining the target aggregation interval according to the target aggregation progress comprises:

[0022] determining a target aggregation upper limit according to the target aggregation progress;

[0023] determining a target aggregation lower limit according to an aggregation interval of a neighboring historical local aggregation;

[0024] determining the target aggregation interval according to the target aggregation upper limit and the target aggregation lower limit.

[0025] As an optional implementation, in a second aspect of the embodiment, continuing the local aggregation according to the target aggregation interval comprises:

[0026] detecting whether the target aggregation progress meets a first preset condition and detecting whether the target aggregation interval meets a second preset condition;

[0027] continuing the local aggregation according to the target aggregation interval when it is detected that the target aggregation progress meets the first preset condition and the target aggregation interval meets the second preset condition.

[0028] In a third aspect, the embodiment provides a data aggregation decoupling device, applied to a master control unit, and stored in a system comprising the master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units being connected respectively, and the data aggregation decoupling device comprising:

[0029] a transceiving module, configured to receive real-time aggregation information respectively sent by each storage operation unit to obtain a plurality of real-time aggregation information, the real-time aggregation information at least comprising a real-time aggregation progress;

[0030] a processing module, configured to analyze the plurality of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress being the smallest progress information in the plurality of real-time aggregation progresses;

[0031] the transceiving module is further configured to send the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts an aggregation interval according to the target aggregation progress and performs aggregation.

[0032] In a fourth aspect, the embodiment provides a data aggregation decoupling device, applied to a storage operation unit, and stored in a system comprising a master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units being connected respectively, and the data aggregation decoupling device comprising:

[0033] a processing module, configured to perform distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress;

[0034] The transceiver module is configured to send real-time aggregation information to the master control unit, wherein the real-time aggregation information at least includes real-time aggregation progress;

[0035] The transceiver module is further configured to receive target aggregation progress sent by the master control unit, and determine a target aggregation interval according to the target aggregation progress.

[0036] The processing module is further configured to continue local aggregation according to the target aggregation interval.

[0037] In a fifth aspect, an embodiment of the present application provides an electronic device, which comprises:

[0038] a memory storing executable program codes;

[0039] a processor coupled to the memory;

[0040] The processor invokes the executable program codes stored in the memory to execute the data aggregation decoupling method in the first aspect or the second aspect of the present application.

[0041] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, which causes a computer to execute the data aggregation decoupling method in the first aspect or the second aspect of the present application. The computer readable storage medium includes ROM / RAM, magnetic or optical disk, etc.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which causes a computer to execute part or all steps of any one method of the first aspect when the computer program product runs on the computer.

[0043] In a sixth aspect, an embodiment of the present application provides an application publishing platform, which is configured to publish a computer program product, wherein the computer program product causes a computer to execute part or all steps of any one method of the first aspect when the computer program product runs on the computer.

[0044] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0045] The embodiment of the application provides a data aggregation decoupling method, device, electronic equipment and storage medium, and a storage system comprises a main control unit and a plurality of storage operation units. The main control unit and the plurality of storage operation units are connected respectively. The storage operation units perform distributed aggregation according to an initial aggregation interval, obtain a real-time aggregation progress, and send real-time aggregation information to the main control unit. The real-time aggregation information at least comprises the real-time aggregation progress. The main control unit analyzes the plurality of real-time aggregation information, obtains a target aggregation progress which is the smallest progress information in the plurality of real-time aggregation information, and sends the target aggregation progress to each storage operation unit. The storage operation unit determines a target aggregation interval according to the target aggregation progress, and continues local aggregation according to the target aggregation interval. In the scheme, the behavior of local aggregation is controlled by setting an upper limit of the local aggregation interval. Each storage independent running unit collects the global progress of distributed aggregation, distributes the smallest aggregation progress to each storage operation unit, and the local aggregation logic of the storage operation unit uses the progress as the upper limit of aggregation. The local aggregation and the distributed aggregation operate at the same time, but their operation intervals are different, so that fine decoupling of the local aggregation and the distributed aggregation can be realized, and the local aggregation function and the distributed aggregation function do not interfere with each other. During the distributed aggregation or reconstruction, the local aggregation function also operates normally, so that the garbage collection function operates normally, and the capacity risk is eliminated. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the present application, and together with the specification, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0048] Figure 1 is a flowchart of a data aggregation decoupling method provided by the embodiment of the application Figure 1 ;

[0049] Figure 2 is a flowchart of a data aggregation decoupling method provided by the embodiment of the application Figure 2 ;

[0050] Figure 3 is an operation architecture diagram of a storage system data aggregation decoupling provided by the embodiment of the application;

[0051] Figure 4 is a structure diagram of a data aggregation decoupling device provided by the embodiment of the applicationFigure 1 ;

[0052] Figure 2 is a structural schematic of a data aggregation decoupling device provided by an embodiment of the present application Figure 6 ;

[0053] Figure 1 is a structural schematic of an electronic device provided by an embodiment of the present application DETAILED DESCRIPTION

[0054] In order to more clearly understand the above objectives, features and advantages of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be explained that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. Apparently, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without any creative work, fall within the scope of protection of the present application.

[0055] The terms "first" and "second" and the like in the specification and claims of the present application are used to distinguish different objects, and are not used to describe a specific order of the objects.

[0056] The terms "include" and "have" and any variations thereof in the embodiments of the present application are intended to cover the non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those clearly listed steps or units, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] It should be explained that, in the embodiments of the present application, the words "exemplary" or "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0058] In storage systems, there are multiple functions that are independent yet interdependent, broadly categorized into two types: local operational logic and distributed operational logic. Local operational logic, such as local aggregation, aims to reclaim garbage data within its own storage unit and therefore does not involve interaction with other units. Distributed operational logic, such as EC aggregation, is distributed logic. During its operation, a unit may need to read data from other units, thus relying on the consistency of the global data state. Local aggregation, however, operates from a single point and is unaware of the operational state of other data shards. This can potentially disrupt the consistency of the global data state, thereby affecting the normal operation of distributed aggregation.

[0059] It's important to note that local aggregation is a business logic that runs periodically on an independent operating unit. It scans the metadata of each object locally, analyzes and calculates the metadata, outputs garbage information, and uses this information to perform garbage collection and data merging, thereby achieving data cleanup, garbage collection, and space freeing. This process relies only on its own metadata and is unaware of the global operation. Local aggregation is typically used to reduce data transfer volume and improve processing efficiency, especially in data-intensive processing scenarios, such as the mapper stage of a MapReduce job. Data is initially merged on the local node, reducing the amount of data transferred in subsequent processing stages. It is suitable for scenarios requiring rapid response because data can be processed without being transferred to remote nodes. It can be implemented using features such as Hadoop Combiner to optimize the performance of MapReduce jobs.

[0060] It's important to note that distributed aggregation operates in a distributed manner, executed by the object's master. It reads remote data and generates new, low-space-consumption data, achieving data compression and merging to reduce global space usage. This function requires reading data from multiple remote locations, all of which must be within the same view. Therefore, this function relies on the global data view. Figure 1 Consistency. Distributed aggregation is typically used to handle large-scale datasets or tasks requiring cross-node collaboration. Data is transferred and merged across multiple nodes to achieve global data processing. It is suitable for scenarios involving large-scale datasets or requiring cross-node collaboration. It can be implemented using distributed computing frameworks such as Hadoop and Spark.

[0061] It should be noted that, when comparing local aggregation and distributed aggregation, local aggregation is mainly processed in a single node or a local range, while distributed aggregation involves data processing between multiple nodes; in addition, local aggregation reduces data transmission by reducing the amount of intermediate results, while distributed aggregation may need to transmit a large amount of data for global merging; and local aggregation is suitable for scenarios that require fast response or processing of small-scale data sets, while distributed aggregation is suitable for processing large-scale data sets or scenarios that require cross-node collaboration.

[0062] To solve the above coupling, the two running logics can be absolutely mutually exclusive, that is, during the running of the distributed aggregation logic, the local aggregation is not performed, and after the distributed aggregation is completed, the local aggregation is performed. However, the existing strategy still has the following problems: The local aggregation depends on the completion of the distributed aggregation task, and a large amount of data cannot be recycled during aggregation, which may cause the risk of data space being full in a high-capacity scenario; in addition, the progress of the local aggregation cannot be globally controlled, which leads to the fact that the progress of each storage operation unit cannot be strictly guaranteed to be consistent, and data anomalies may still occur when the distributed aggregation obtains data.

[0063] To solve the above technical problems or all technical problems, the embodiments of the present application provide a data aggregation decoupling method and device, an electronic device and a storage medium. The storage system includes a master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are connected respectively, the storage operation units perform distributed aggregation according to an initial aggregation interval, obtain a real-time aggregation progress, and send real-time aggregation information to the master control unit, the real-time aggregation information at least includes the real-time aggregation progress; the master control unit analyzes a plurality of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress is the smallest progress information in the plurality of real-time aggregation progress, and sends the target aggregation progress to each storage operation unit; and the storage operation unit determines a target aggregation interval according to the target aggregation progress, and continues to perform local aggregation according to the target aggregation interval. In this scheme, the behavior of local aggregation is controlled by setting an upper limit of the local aggregation interval, each storage independent running unit collects the global progress of distributed aggregation, distributes the smallest aggregation progress to each storage operation unit, and the local aggregation logic of the storage operation unit uses the progress as its aggregation upper limit. Although local aggregation and distributed aggregation are operated at the same time, their operation intervals are different, which can realize fine decoupling of local aggregation and distributed aggregation, and further realize that the local aggregation function and the distributed aggregation function do not interfere with each other. During distributed aggregation or reconstruction, the local aggregation function also operates normally, which guarantees the normal operation of the garbage collection function and eliminates the capacity risk.

[0064] As shown in Figure 1 , the master control unit 100 is connected with the plurality of storage operation units 200, and the plurality of storage operation units 200 are connected with each other. Figure 1A flowchart of a data aggregation decoupling method provided in an embodiment of the present application is realized by interaction between a master control unit and a storage operation unit. The storage system can specifically include the master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are respectively connected, and the plurality of storage operation units are independent of each other. The storage operation unit can be understood as a single disk or a functional subject for data storage and processing in the disk, and the master control unit is a management functional subject at a higher level than the storage operation unit. The method can include the following steps:

[0065] 101. The storage operation unit performs distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress.

[0066] In the embodiment of the present application, each initial aggregation interval can first complete distributed aggregation by itself. The aggregation interval of distributed aggregation can be understood as dynamic update. The initial aggregation interval can be understood as an interval determined according to the last distributed aggregation and the indication of the master control unit, which can be represented as [A-B]. When the storage operation unit completes the distributed aggregation according to the [A-B] interval, the aggregation progress Agg_progress of this time is refreshed to B, that is, the real-time aggregation progress is B.

[0067] In some embodiments, each storage operation unit in the storage system performs distributed aggregation and refreshes the aggregation progress of the storage operation unit after completion. The aggregation progress of each storage operation unit is determined according to the respective aggregation interval and can be different.

[0068] 102. The storage operation unit sends real-time aggregation information to the master control unit.

[0069] In the embodiment of the present application, the real-time aggregation information can at least include the real-time aggregation progress. Each storage operation unit can report the real-time aggregation progress to the master control unit after completing the distributed aggregation.

[0070] 103. The master control unit receives the real-time aggregation information sent by each storage operation unit to obtain a plurality of real-time aggregation information.

[0071] In the embodiment of the present application, after the master control unit receives the real-time aggregation information sent by each storage operation unit, the plurality of real-time aggregation information can be summarized, that is, the plurality of real-time aggregation information and the plurality of storage operation units can be understood as being associated and saved in a map.

[0072] 104. The master control unit analyzes the plurality of real-time aggregation information to obtain a target aggregation progress.

[0073] In this embodiment of the application, distributed aggregation needs to ensure global data visibility. Figure 2 To ensure consistency, the main control unit needs to monitor the overall progress by analyzing the real-time aggregation progress of each storage operation unit. Therefore, the main control unit can parse multiple real-time aggregation information to obtain the target aggregation progress, which can be the smallest progress information among multiple real-time aggregation progresses.

[0074] It should be noted that the aggregation progress of each storage operation unit may be inconsistent, because in order to ensure data visibility across all storage operation units... Figure 2 To avoid aggregation progress vulnerabilities in some storage operation units, it is necessary to select the aggregation progress with the smallest progress as the global progress.

[0075] 105. The main control unit sends the target aggregation progress to each storage operation unit so that each storage operation unit can adjust the aggregation range and perform aggregation according to the target aggregation progress.

[0076] In this embodiment, the main control unit can send the target aggregation progress to each storage operation unit, which can ensure that the aggregation progress of each storage operation unit is consistent.

[0077] 106. The storage operation unit receives the target aggregation progress sent by the main control unit and determines the target aggregation interval based on the target aggregation progress.

[0078] In this embodiment of the application, after the storage operation unit receives the target aggregation progress, it can determine a new aggregation interval. The upper limit of the aggregation interval can be determined based on the target aggregation progress, and the lower limit can be determined based on the interval of the previous aggregation.

[0079] 107. The storage operation unit continues to perform local aggregation according to the target aggregation range.

[0080] In this embodiment of the application, after the storage operation unit determines the target aggregation range, it can determine the target aggregation range as the local aggregation range and perform local aggregation according to the target aggregation range.

[0081] It should be noted that the target aggregation range of local aggregation and the initial aggregation range of distributed aggregation are different ranges, and the upper limit of local aggregation will not exceed the aggregation progress of distributed aggregation.

[0082] The embodiment of the present application provides a data aggregation decoupling method, the behavior of local aggregation is controlled by setting the upper limit of a local aggregation interval, each storage independent operation unit collects a global progress of distributed aggregation, the minimum aggregation progress is distributed to each storage operation unit, and the local aggregation logic of the storage operation unit uses the progress as the aggregation upper limit. The operation intervals of the local aggregation and the distributed aggregation are different although the local aggregation and the distributed aggregation are simultaneously operated, so that fine decoupling of the local aggregation and the distributed aggregation can be realized, and then the local aggregation function and the distributed aggregation function do not interfere with each other, the local aggregation function also operates normally during distributed aggregation or reconstruction, the garbage collection function is ensured to operate normally, and the capacity risk is removed.

[0083] As shown in Figure 1 , Figure 3 A flowchart of a data aggregation decoupling method provided by the embodiment of the present application is shown, the data aggregation decoupling method is realized by interaction between a main control unit and a storage operation unit, and the method can further include the following steps.

[0084] 201. The storage operation unit performs distributed aggregation according to an initial aggregation interval, and obtains real-time aggregation progress.

[0085] 202. The storage operation unit sends real-time aggregation information to the main control unit.

[0086] In some embodiments, the real-time aggregation information can further include real-time version information, and sending the real-time aggregation information to the main control unit can specifically include: obtaining real-time version information after distributed aggregation; and sending the real-time aggregation progress and the real-time version information to the main control unit.

[0087] In the embodiment of the present application, the version information is also updated after each aggregation of the storage operation unit, in order to ensure that the data views of all storage operation units are consistent, therefore, the storage operation unit can send the real-time aggregation progress and the real-time version information to the main control unit together for analysis after refreshing the aggregation progress after aggregation. Figure 3

[0088] In some embodiments, the real-time aggregation information can further include identity information of the storage operation unit.

[0089] In the embodiment of the present application, the main control unit needs to collect the aggregation progress of each storage operation unit, and needs to ensure that the aggregation progress of each storage operation unit is received, therefore, the storage operation unit can send the real-time aggregation progress and the identity information to the main control unit together, so that the main control unit can detect whether the aggregation progress of all storage operation units is received.

[0090] ​In some embodiments, the real-time aggregation information at least includes a real-time aggregation progress, and according to the above embodiments, the real-time aggregation information further includes real-time version information, and the real-time aggregation information further includes identity information; of course, the real-time version information and the identity information are in an and / or relationship, that is, the real-time aggregation information can include the real-time aggregation progress and the real-time version information, or the real-time aggregation information can include the real-time aggregation progress and the identity information, or the real-time aggregation information can include the real-time aggregation progress, the real-time version information and the identity information.

[0091] 203、The master control unit receives the real-time aggregation information respectively sent by each storage operation unit, and obtains a plurality of real-time aggregation information.

[0092] 204、The master control unit analyzes the plurality of real-time aggregation information, and obtains a target aggregation progress.

[0093] In some embodiments, in the case where the real-time aggregation information further includes real-time version information, analyzing the plurality of real-time aggregation information to obtain the target aggregation progress can specifically include: detecting whether the real-time version information corresponding to each storage operation unit is the same; in the case where it is detected that the real-time version information corresponding to all storage operation units is the same, analyzing the plurality of real-time aggregation progresses to obtain the target aggregation progress.

[0094] It should be noted that the master control unit will analyze the plurality of real-time aggregation progresses to obtain the target aggregation progress only in the case where it is detected that the version information of all storage operation units is the same; if the version information of part of the storage operation units is different, the master control unit will not operate, and will analyze all real-time aggregation progresses to obtain the target aggregation progress only after the master control unit receives the real-time aggregation information including the same version information.

[0095] In some embodiments, in the case where the real-time aggregation information further includes identity information of the storage operation unit, analyzing the plurality of real-time aggregation information to obtain the target aggregation progress can specifically include: according to the identity information, detecting whether the real-time aggregation information sent by all storage operation units is received; in the case where it is detected that the real-time aggregation information sent by all storage operation units is received, analyzing the plurality of real-time aggregation progresses to obtain the target aggregation progress.

[0096] It should be noted that after the master control unit receives the identity information and the real-time aggregation progress of the storage operation unit, the identity information and the real-time aggregation progress can be stored correspondingly, that is, the identity information of the storage operation unit is taken as a key value of a map, and the corresponding value is the real-time aggregation progress reported at the same time.

[0097] In some embodiments, when the real-time aggregation information simultaneously includes the real-time aggregation progress, the real-time version information and the identity information, the identity information of the storage operation unit is stored as the key value of the map, and the corresponding value is the real-time aggregation progress information and the real-time version information reported simultaneously.

[0098] 205、The master control unit sends the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval and performs aggregation according to the target aggregation progress.

[0099] 206、The storage operation unit receives the target aggregation progress sent by the master control unit.

[0100] 207、The storage operation unit determines the target aggregation upper limit according to the target aggregation progress.

[0101] In the embodiments of the present application, when determining the target aggregation upper limit, it needs to be considered that the aggregation intervals of the local aggregation and the distributed aggregation cannot overlap, and the local aggregation cannot exceed the distributed aggregation, that is, the target aggregation upper limit needs to be less than the target aggregation progress, so the target aggregation upper limit can be obtained by subtracting a preset value from the target aggregation progress.

[0102] 208、The storage operation unit determines the target aggregation lower limit according to the aggregation interval of the adjacent historical local aggregation.

[0103] In the embodiments of the present application, since the data aggregation is continuous, the aggregation interval is also continuous, and the upper limit of the last local aggregation can be determined as the lower limit of the current local aggregation, that is, the target aggregation lower limit.

[0104] 209、The storage operation unit determines the target aggregation interval according to the target aggregation upper limit and the target aggregation lower limit.

[0105] 210、The storage operation unit detects whether the target aggregation progress meets the first preset condition, and detects whether the target aggregation interval meets the second preset condition.

[0106] In the embodiments of the present application, the first preset condition can be greater than the upper limit of the last local aggregation, and the second preset condition can be non-overlapping with the initial aggregation interval, and of course it can also be other self-defined conditions.

[0107] 211、The storage operation unit continues to perform local aggregation according to the target aggregation interval in the case that the target aggregation progress meets the first preset condition and the target aggregation interval meets the second preset condition.

[0108] In the embodiments of the present application, when it is detected that the target aggregation progress is greater than the upper limit of the last local aggregation, and the target aggregation interval is non-overlapping with the initial aggregation interval, the local aggregation is continued according to the target aggregation interval.

[0109] The embodiment of the application provides a data aggregation decoupling method, which controls the behavior of local aggregation by setting the upper limit of the local aggregation interval, each storage operation unit collects the global progress of distributed aggregation, and the minimum aggregation progress is distributed to each storage operation unit, and the local aggregation logic of the storage operation unit uses the progress as the upper limit of aggregation, and the local aggregation and the distributed aggregation operate at the same time, but the operation intervals are different, so that the local aggregation and the distributed aggregation can be finely decoupled, and the local aggregation function and the distributed aggregation function do not interfere with each other, and the local aggregation function also operates normally during the distributed aggregation or reconstruction, so that the garbage collection function operates normally, and the capacity risk is eliminated.

[0110] In some embodiments, as shown in Figure 4 The operation architecture diagram of the storage system for data aggregation decoupling is shown, and two operation modules, namely an aggregation progress reporting and refreshing module and an aggregation progress analysis and brushing module, are mainly added. By adding the two modules, the local aggregation and the distributed aggregation are decoupled, and operate in respective scopes without interfering with each other.

[0111] The aggregation progress reporting and refreshing module is used for periodic reporting of the aggregation progress and refreshing the upper limit of the local aggregation after receiving the message. The interval lower limit value of the distributed aggregation is the upper limit of the last aggregation, and the interval upper limit of the aggregation is the current IO completion timestamp; after the current aggregation is completed according to the interval, the local distributed aggregation progress is refreshed, that is, progress_N (real-time aggregation progress) is equal to EC_high_epoch (upper limit of the initial aggregation interval). The progress reporting module will periodically assemble the related information (operation unit ID+aggregation progress+current version information) of the storage operation unit, and send it to the main control unit.

[0112] The aggregation progress analysis and brushing module is used for storing and analyzing the aggregation progress and then brushing. The module runs on the main control unit, and after receiving the message of other storage operation units, saves the related information (operation unit ID+aggregation progress+current version information) in the local map memory. The main control unit periodically analyzes the map information, traverses the information of each storage operation unit, and if the version information reported by each unit is consistent, it is considered that all operation units are reporting normally. Then the minimum distributed aggregation progress of each operation unit is obtained as the brushing aggregation progress, if the brushing value this time is consistent with the last time, the brushing this time is abandoned, and if there is an update, the brushing is performed. As shown in Figure 5 According to the aggregation progress progress-A, progress-B and progress-C reported by the three storage operation units, the local aggregation upper limit vos-high-epoch is corrected and is respectively sent to the three storage operation units.

[0113] The aggregation progress reporting and refreshing module receives a new refreshing progress, and determines that the value meets a preset condition, and then updates the upper limit value of the local aggregation to ensure that the local aggregation can normally operate. Finally, the scope of the distributed aggregation and the local aggregation is dynamically updated, and the interval of the local aggregation does not exceed the set upper limit, the mutual exclusion of the two is realized, and the coupling problem of the distributed aggregation and the local aggregation is solved.

[0114] As shown in Figure 6 The data aggregation decoupling apparatus provided by the embodiment of the present application is applied to a master control unit, and the storage system includes the master control unit and a plurality of storage operation units, and the master control unit and the plurality of storage operation units are connected respectively. The data aggregation decoupling apparatus can include:

[0115] The transceiver module 401 is configured to receive real-time aggregation information respectively sent by each storage operation unit, and obtain a plurality of real-time aggregation information. The real-time aggregation information at least includes real-time aggregation progress.

[0116] The processing module 402 is configured to analyze the plurality of real-time aggregation information, and obtain a target aggregation progress. The target aggregation progress is the smallest progress information in the plurality of real-time aggregation progresses.

[0117] The transceiver module 401 is further configured to send the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval according to the target aggregation progress and performs aggregation.

[0118] In some embodiments, the real-time aggregation information further includes real-time version information.

[0119] The processing module 402 is specifically configured to detect whether the real-time version information corresponding to each storage operation unit is the same.

[0120] The processing module 402 is specifically configured to, in a case where it is detected that the real-time version information corresponding to all storage operation units is the same, analyze the plurality of real-time aggregation progresses, and obtain the target aggregation progress.

[0121] In some embodiments, the real-time aggregation information further includes identity information of the storage operation unit.

[0122] The processing module 402 is specifically configured to detect whether the real-time aggregation information sent by all storage operation units is received according to the identity information.

[0123] The processing module 402 is specifically configured to, in a case where it is detected that the real-time aggregation information sent by all storage operation units has been received, analyze the plurality of real-time aggregation progresses, and obtain the target aggregation progress.

[0124] In the embodiments of the present application, each module can implement the data aggregation decoupling method provided by the above-mentioned method embodiments, and achieve the same technical effects. To avoid repetition, details are not described here.

[0125] As shown in ​ The present application provides a data aggregation decoupling device, which is applied to a storage operation unit, and a storage system includes a master control unit and a plurality of storage operation units. The master control unit and the plurality of storage operation units are connected respectively. The data aggregation decoupling device can include:

[0126] The processing module 501 is configured to perform distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress.

[0127] The transceiver module 402 is configured to send real-time aggregation information to the master control unit, and the real-time aggregation information at least includes the real-time aggregation progress.

[0128] The transceiver module 502 is further configured to receive a target aggregation progress sent by the master control unit, and determine a target aggregation interval according to the target aggregation progress.

[0129] The processing module 501 is further configured to continue local aggregation according to the target aggregation interval.

[0130] In some embodiments, the real-time aggregation information further includes real-time version information.

[0131] The processing module 501 is specifically configured to obtain real-time version information after distributed aggregation.

[0132] The transceiver module 502 is specifically configured to send the real-time aggregation progress and the real-time version information to the master control unit.

[0133] In some embodiments, the real-time aggregation information further includes identity information of the storage operation unit.

[0134] In some embodiments, the processing module 501 is specifically configured to determine a target aggregation upper limit according to the target aggregation progress.

[0135] The processing module 501 is specifically configured to determine a target aggregation lower limit according to an aggregation interval of adjacent historical local aggregation.

[0136] The processing module 501 is specifically configured to determine the target aggregation interval according to the target aggregation upper limit and the target aggregation lower limit.

[0137] In some embodiments, the processing module 501 is specifically configured to detect whether the target aggregation progress meets a first preset condition, and detect whether the target aggregation interval meets a second preset condition.

[0138] The processing module 501 is specifically configured to continue the local aggregation according to the target aggregation interval when it is detected that the target aggregation progress meets the first preset condition and the target aggregation interval meets the second preset condition.

[0139] In the embodiments of the present application, each module can implement the data aggregation decoupling method provided by the above method embodiments, and achieve the same technical effects. To avoid repetition, details are not described here.

[0140] As shown in ​ The embodiments of the present application also provide an electronic device, which can include:

[0141] The memory 601 stores executable program codes;

[0142] The processor 602 is coupled with the memory 601;

[0143] The processor 602 calls the executable program codes stored in the memory 601 to execute the data aggregation decoupling method performed by the electronic device in each method embodiment.

[0144] The embodiments of the present application provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement each process of the data aggregation decoupling method in the above method embodiments, and can achieve the same technical effects. To avoid repetition, details are not described here.

[0145] The embodiments of the present application also provide a computer program product, which stores a computer program. The computer program is executed by a processor to implement each process of the data aggregation decoupling method in the above method embodiments, and can achieve the same technical effects. To avoid repetition, details are not described here.

[0146] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media including computer usable program codes.

[0147] It should be understood that all the devices and methods disclosed in the embodiments of the present application can be implemented by other ways. The device embodiments described above are only schematic, and for instance, the flowcharts and the block diagrams in the figures illustrate the possible implementation modes of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or the block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order from that noted in the figures. For example, two consecutive blocks can actually be executed in parallel or in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or the flowcharts, and the combination of blocks in the block diagrams and / or the flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0148] In the present application, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0149] In the present application, the memory can include a non-persistent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory, etc., such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0150] In the present application, those of ordinary skill in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by relevant hardware instructed by a program, and the program can be stored in a computer readable storage medium, including permanent and non-permanent, removable and non-removable storage medium. The storage medium can realize information storage by any method or technology, and the information can be computer readable instructions, data structure, program module or other data. Examples of computer storage medium include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), other types of random access memory (RAM), read-only memory (ROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, the computer readable medium does not include transitory computer readable media such as modulated data signals and carriers.

[0151] It is to be noted that, in the present document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", "includes", "including", or the like, are intended to encompass non-exclusive inclusions, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, the term "coupled" or "coupling" is intended to mean a direct connection between two elements, or an indirect connection through one or more intermediate elements.

[0152] It is to be understood that the terminology "one embodiment" or "an embodiment" used throughout this document means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Therefore, the appearances of the phrases "in one embodiment" or "in an embodiment" appearing in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It will also be appreciated by those of skill in the art that the embodiments described herein are merely exemplary in nature, and that the acts and modules described in the embodiments are not necessarily required to be performed in the particular order mentioned or illustrated herein. The embodiments described herein are also not necessarily all independent, but can be combined in any suitable manner.

[0153] In various embodiments of the present application, it is to be understood that the sequence of the above-mentioned processes does not mean the inevitable order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0154] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. they may be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0155] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0156] The integrated units described above, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer accessible memory. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the present application or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of parts or all steps of the above-mentioned method for enabling a computer device (which can be a personal computer, a server or a network device, etc., and specifically can be a processor in the computer device) to execute the embodiments of the present application.

[0157] The above is only a specific implementation of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data aggregation decoupling method, characterized in that, The storage system comprises a master control unit and a plurality of storage operation units, the master control unit and the plurality of storage operation units are connected respectively, and the method comprises the following steps: The storage operation unit performs distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress; The storage operation unit sends real-time aggregation information to the master control unit, and the real-time aggregation information at least comprises the real-time aggregation progress; The master control unit receives the real-time aggregation information sent by each storage operation unit respectively to obtain a plurality of real-time aggregation information; The master control unit analyzes the plurality of real-time aggregation information to obtain a target aggregation progress, and the target aggregation progress is the smallest progress information in the plurality of real-time aggregation progresses; The master control unit sends the target aggregation progress to each storage operation unit, so that each storage operation unit adjusts the aggregation interval according to the target aggregation progress and performs aggregation; The storage operation unit receives the target aggregation progress sent by the master control unit, and determines a target aggregation upper limit according to the target aggregation progress; The storage operation unit determines a target aggregation lower limit according to the aggregation interval of adjacent historical local aggregation; The storage operation unit determines a target aggregation interval according to the target aggregation upper limit and the target aggregation lower limit; The storage operation unit continues local aggregation according to the target aggregation interval.

2. The method of claim 1, wherein, The real-time aggregation information further comprises real-time version information; the master control unit analyzes the plurality of real-time aggregation information to obtain a target aggregation progress, which comprises the following steps: The master control unit detects whether the real-time version information corresponding to each storage operation unit is the same; The master control unit analyzes the plurality of real-time aggregation progresses to obtain the target aggregation progress when it is detected that the real-time version information corresponding to all storage operation units is the same.

3. The method of claim 1, wherein, The real-time aggregation information further comprises identity information of the storage operation unit; the master control unit analyzes the plurality of real-time aggregation information to obtain a target aggregation progress, which comprises the following steps: The master control unit detects whether the real-time aggregation information sent by all storage operation units is received according to the identity information; The master control unit analyzes the plurality of real-time aggregation progresses to obtain the target aggregation progress when it is detected that the real-time aggregation information sent by all storage operation units has been received.

4. The method of claim 1, wherein, The real-time aggregation information further comprises real-time version information; the storage operation unit sends real-time aggregation information to the master control unit, which comprises the following steps: The storage operation unit obtains the real-time version information after distributed aggregation; The storage operation unit sends the real-time aggregation progress and the real-time version information to the master control unit.

5. The method of claim 1, wherein, The storage operation unit continues local aggregation according to the target aggregation interval, which comprises the following steps: The storage operation unit detects whether the target aggregation progress meets a first preset condition, and detects whether the target aggregation interval meets a second preset condition; The storage operation unit continues local aggregation according to the target aggregation interval when the target aggregation progress meets the first preset condition and the target aggregation interval meets the second preset condition.

6. A data aggregation decoupling apparatus for implementing the data aggregation decoupling method of claim 1, wherein, The storage system includes a master control unit and a plurality of storage operation units connected between the master control unit and the plurality of storage operation units, and the device includes: The processing module is configured to cause the storage operation unit to perform distributed aggregation according to an initial aggregation interval to obtain a real-time aggregation progress. The transceiver module is configured to cause the storage operation unit to send real-time aggregation information to the master control unit, the real-time aggregation information including at least the real-time aggregation progress. The transceiver module is further configured to cause the master control unit to receive real-time aggregation information sent by each storage operation unit to obtain a plurality of real-time aggregation information. The processing module is further configured to cause the master control unit to analyze the plurality of real-time aggregation information to obtain a target aggregation progress, the target aggregation progress being the smallest progress information in the plurality of real-time aggregation progresses. The transceiver module is further configured to cause the master control unit to send the target aggregation progress to each storage operation unit to cause each storage operation unit to adjust an aggregation interval according to the target aggregation progress and perform aggregation. The transceiver module is further configured to cause the storage operation unit to receive the target aggregation progress sent by the master control unit. The processing module is further configured to cause the storage operation unit to determine a target aggregation upper limit according to the target aggregation progress. The processing module is further configured to cause the storage operation unit to determine a target aggregation lower limit according to an aggregation interval of adjacent historical local aggregation. The processing module is further configured to cause the storage operation unit to determine a target aggregation interval according to the target aggregation upper limit and the target aggregation lower limit. The processing module is further configured to cause the storage operation unit to continue local aggregation according to the target aggregation interval.

7. An electronic device, comprising: The electronic device includes: a memory storing executable program code; and a processor coupled to the memory; The processor invokes the executable program code stored in the memory to execute the data aggregation decoupling method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the data aggregation decoupling method of any one of claims 1 to 5. ​

Images