An edge data management and control method and device with deduplication function

Abstract

This invention discloses an edge data management method and apparatus with deduplication functionality. The method includes: when it is determined that the storage service client is in storage operation mode, switching the backup client to backup operation mode and generating a data interaction link between the storage service client and the backup client; detecting the data transmission status of a first time-series database; when it is determined that the data transmission status is a first state with data interaction, controlling the backup client to obtain target interaction data of the first time-series database within a preset detection period based on the data interaction link; controlling the backup client to perform deduplication processing on the target interaction data according to a determined deduplication processing algorithm to obtain the deduplication processing result; and sending the deduplication processing result to the storage service client to write the deduplication processing result into the corresponding second time-series database of the storage service client. Therefore, implementing this invention can improve the disaster recovery capabilities of edge cloud data and reduce the occurrence of data loss.

Description

Technical Field

[0001] This invention relates to the field of edge data processing technology, and in particular to an edge data management method and apparatus with deduplication function. Background Technology

[0002] In the industrial internet field, IoT gateways located in the edge cloud collect massive amounts of data, including video streams, device metrics, and license plate information. Furthermore, with the advancement of technology and the increasing number of enterprise users, the amount of data that the edge cloud needs to process is growing exponentially.

[0003] However, compared to the surging data volume, the disaster recovery capabilities of edge clouds have always been relatively weak. In other words, edge clouds typically suffer from a lack of disaster recovery capabilities. Therefore, if the IoT gateway corresponding to the edge cloud fails, a large amount of data loss will occur. It is therefore crucial to improve the disaster recovery capabilities of edge cloud data and reduce the occurrence of data loss. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an edge data management method and device with deduplication function, which can improve the disaster recovery capability of edge cloud data and reduce the occurrence of data loss.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a method for edge data management with deduplication functionality, the method comprising:

[0006] When it is determined that the storage service client is in storage running state, the backup client is switched to backup running state and a data interaction link is generated between the storage service client and the backup client.

[0007] The data transmission status of the first time-series database is detected, and the data transmission status includes a first state where data interaction exists or a second state where data interaction does not exist.

[0008] When the data transmission status is determined to be the first status, the backup client is controlled to obtain the target interaction data of the first time series database within a preset detection period based on the data interaction link.

[0009] The backup client is controlled to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm, so as to obtain the deduplication processing result corresponding to the target interaction data.

[0010] The deduplication result is sent to the storage service terminal, and the storage service terminal is controlled to write the deduplication result into the second time-series database corresponding to the storage service terminal.

[0011] As an optional implementation, in the first aspect of the present invention, controlling the backup client to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm to obtain the deduplication processing result corresponding to the target interaction data includes:

[0012] Based on the determined deduplication algorithm, the backup client is controlled to divide the target interaction data according to the data type of the target interaction data to obtain the data division result corresponding to the target interaction data. The data division result includes at least one first data block of a first data type and at least one second data block of a second data type.

[0013] The backup client determines, according to the deduplication algorithm, the first-level data blocks in all the second data blocks that will undergo deduplication and the second-level data blocks that will not undergo deduplication.

[0014] The backup client performs the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication algorithm to obtain the deduplication results corresponding to all the first data blocks and the first-level data blocks;

[0015] The deduplication result and the secondary data block are determined as the deduplication processing result corresponding to the target interactive data.

[0016] As an optional implementation, in the first aspect of the present invention, the step of the backup client performing the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication processing algorithm to obtain the deduplication results corresponding to all the first data blocks and the first-level data blocks includes:

[0017] All the first data blocks and the first-level data blocks are identified as the target deduplicated data blocks;

[0018] The backup client determines the data attributes corresponding to the target deduplicated data block according to the deduplication algorithm;

[0019] The backup client performs a data compression operation on the target deduplicated data block according to the data attributes corresponding to the target deduplicated data block, and obtains the data compression result corresponding to the target deduplicated data block, which serves as the deduplication result for all the first data blocks and the first-level data blocks.

[0020] The data compression result corresponding to the target deduplicated data block includes the data compression result corresponding to each of the first data blocks and the data compression result corresponding to the first-level data block.

[0021] As an optional implementation, in the first aspect of the present invention, the first data block of the first data type is specifically a data block containing duplicate data in a continuous data block; the second data block of the second data type is specifically a data block containing no duplicate data in a continuous data block.

[0022] As an optional implementation, in the first aspect of the present invention, before sending the deduplication result to the storage service terminal, the method further includes:

[0023] Analyze the deduplication results to obtain the deduplication verification value corresponding to the target interaction data. The deduplication verification value is used to determine the deduplication status of the target interaction data.

[0024] The deduplication check value is compared with the pre-determined simulated deduplication value to obtain the data difference between the deduplication check value and the simulated deduplication value. The simulated deduplication value is the estimated deduplication value obtained after performing the deduplication processing operation on the simulated interactive data.

[0025] When the data difference is within a preset standard deviation range, the target interactive data is determined to be data that meets the deduplication requirements, and the operation of sending the deduplication result to the storage service terminal is triggered.

[0026] As an optional implementation, in the first aspect of the present invention, when the data difference is outside a preset standard deviation range, the method further includes:

[0027] Based on the data difference and the deduplication algorithm, the backup client is controlled to perform a second deduplication operation on the target interactive data until the numerical difference between the deduplication verification data and the simulated deduplication value corresponding to the target interactive data is determined to be within the standard deviation range. Then, the target interactive data is determined to be data that meets the deduplication requirements, and the operation of sending the deduplication result to the storage service client is triggered.

[0028] As an optional implementation, in the first aspect of the present invention, during the process of controlling the backup client to perform a secondary deduplication operation on the target interactive data based on the data difference and the deduplication algorithm, the method further includes:

[0029] Each time the secondary deduplication operation is performed on the target interactive data, the number of times the secondary deduplication operation is performed on the target interactive data is updated accordingly.

[0030] When the number of operations is greater than the preset number of operations, and the numerical difference between the deduplication verification data corresponding to the target interaction data and the simulated deduplication value is outside the standard deviation range, deduplication information for the target interaction data is generated based on the number of operations. The deduplication information is used to stop the secondary deduplication processing operation for the target interaction data and to provide feedback to the processing personnel who process the target interaction data, so as to trigger the processing personnel to perform data analysis processing operation on the target interaction data based on the deduplication information.

[0031] A second aspect of the present invention discloses an edge data management device with deduplication function, the device comprising:

[0032] The switching interaction module is used to switch the backup client to the backup running state and generate a data interaction link between the storage service client and the backup client when it is determined that the storage service client is in the storage running state.

[0033] The detection module is used to detect the data transmission status of the first time-series database, wherein the data transmission status includes a first state in which data interaction exists or a second state in which data interaction does not exist.

[0034] The control module is used to control the backup client to obtain the target interaction data of the first time series database within a preset detection period based on the data interaction link when the data transmission status is determined to be the first status.

[0035] The deduplication module is used to control the backup client to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm, so as to obtain the deduplication processing result corresponding to the target interaction data.

[0036] The sending module is used to send the deduplication result to the storage service terminal.

[0037] The control module is also used to control the storage service terminal to write the deduplication result into the second time-series database corresponding to the storage service terminal.

[0038] As an optional implementation, in the second aspect of the present invention, the deduplication module controls the backup client to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm, and the specific method for obtaining the deduplication processing result corresponding to the target interaction data includes:

[0039] Based on the determined deduplication algorithm, the backup client is controlled to divide the target interaction data according to the data type of the target interaction data to obtain the data division result corresponding to the target interaction data. The data division result includes at least one first data block of a first data type and at least one second data block of a second data type.

[0040] The backup client determines, according to the deduplication algorithm, the first-level data blocks in all the second data blocks that will undergo deduplication and the second-level data blocks that will not undergo deduplication.

[0041] The backup client performs the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication algorithm to obtain the deduplication results corresponding to all the first data blocks and the first-level data blocks;

[0042] The deduplication result and the secondary data block are determined as the deduplication processing result corresponding to the target interactive data.

[0043] As an optional implementation, in a second aspect of the present invention, the deduplication module, by the backup client performing the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication algorithm, obtains the deduplication results corresponding to all the first data blocks and the first-level data blocks in the following specific ways:

[0044] All the first data blocks and the first-level data blocks are identified as the target deduplicated data blocks;

[0045] The backup client determines the data attributes corresponding to the target deduplicated data block according to the deduplication algorithm;

[0046] The backup client performs a data compression operation on the target deduplicated data block according to the data attributes corresponding to the target deduplicated data block, and obtains the data compression result corresponding to the target deduplicated data block, which serves as the deduplication result for all the first data blocks and the first-level data blocks.

[0047] The data compression result corresponding to the target deduplicated data block includes the data compression result corresponding to each of the first data blocks and the data compression result corresponding to the first-level data block.

[0048] As an optional implementation, in a second aspect of the present invention, the first data block of the first data type is specifically a data block containing duplicate data in a continuous data block; the second data block of the second data type is specifically a data block containing no duplicate data in a continuous data block.

[0049] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0050] The analysis module is used to analyze the deduplication processing result before the sending module sends the deduplication processing result to the storage service terminal, and obtain the deduplication verification value corresponding to the target interaction data. The deduplication verification value is used to determine the deduplication status of the target interaction data.

[0051] The comparison module is used to compare the deduplication verification value with the pre-determined simulated deduplication value to obtain the data difference between the deduplication verification value and the simulated deduplication value. The simulated deduplication value is the estimated deduplication value obtained after performing the deduplication processing operation on the simulated interactive data.

[0052] The determination module is used to determine that the target interactive data meets the deduplication requirements when the data difference is within a preset standard deviation range, and to trigger the operation of sending the deduplication result to the storage service terminal.

[0053] As an optional implementation, in a second aspect of the present invention, the control module is further configured to, when the data difference is outside a preset standard deviation range, control the backup client to perform a secondary deduplication operation on the target interactive data based on the data difference and the deduplication algorithm, until it is determined that the numerical difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is within the standard deviation range, then determine that the target interactive data meets the deduplication requirements and trigger the operation of sending the deduplication result to the storage service client.

[0054] As an optional implementation, in a second aspect of the invention, the apparatus further includes:

[0055] The update module is used to update the number of times the secondary deduplication operation is performed on the target interactive data each time the backup client performs the secondary deduplication operation on the target interactive data, in the process of controlling the backup client to perform the secondary deduplication operation on the target interactive data according to the data difference and the deduplication algorithm.

[0056] The generation module is used to generate deduplication information for the target interactive data based on the number of operations when the number of operations is greater than a preset number of operations, and the numerical difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is outside the standard deviation range. The deduplication information is used to stop the secondary deduplication processing operation for the target interactive data and to provide feedback to the processing personnel who process the target interactive data, so as to trigger the processing personnel to perform data analysis processing operation on the target interactive data based on the deduplication information.

[0057] A third aspect of the present invention discloses another edge data management device with deduplication function, the device comprising:

[0058] Memory containing executable program code;

[0059] A processor coupled to the memory;

[0060] The processor calls the executable program code stored in the memory to execute the edge data management method with deduplication function disclosed in the first aspect of the present invention.

[0061] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the edge data management method with deduplication function disclosed in the first aspect of the present invention.

[0062] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0063] This invention provides an edge data management method with deduplication functionality. The method includes: when it is determined that the storage service provider is in a storage running state, switching the backup client to a backup running state and generating a data interaction link between the storage service provider and the backup client; detecting the data transmission status of a first time-series database, where the data transmission status includes a first state with data interaction or a second state without data interaction; when the data transmission status is determined to be the first state, controlling the backup client to obtain target interaction data from the first time-series database within a preset detection period based on the data interaction link; controlling the backup client to perform deduplication processing on the target interaction data according to a determined deduplication processing algorithm, obtaining the deduplication processing result corresponding to the target interaction data; sending the deduplication processing result to the storage service provider and controlling the storage service provider to write the deduplication processing result into the second time-series database corresponding to the storage service provider. As can be seen, implementing this invention can automatically build a data link between the backup database and the storage service, thereby automatically performing data deduplication on the acquired target interactive data when the data transmission state is in the first state. This realizes the periodic backup and deduplication of edge cloud data, improves the disaster recovery capability of edge cloud data, and reduces the occurrence of data loss and irrecoverability due to gateway anomalies. In addition, this deduplication transmission operation reduces the transmission bandwidth, which is conducive to improving transmission efficiency. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating an edge data management method with deduplication functionality disclosed in an embodiment of the present invention.

[0066] Figure 2 This is a flowchart illustrating another edge data management method with deduplication functionality disclosed in an embodiment of the present invention.

[0067] Figure 3 This is a schematic diagram of the structure of an edge data management device with deduplication function disclosed in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of another edge data management device with deduplication function disclosed in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the structure of another edge data management device with deduplication function disclosed in an embodiment of the present invention. Detailed Implementation

[0070] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0072] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0073] This invention discloses an edge data management method and apparatus with deduplication functionality. It can automatically construct a data link between the backup database and the storage service, thereby automatically performing deduplication on the acquired target interactive data when the data transmission state is in the first state. This achieves periodic backup and deduplication of edge cloud data, improving the disaster recovery capability of edge cloud data and reducing the occurrence of data loss due to gateway anomalies. Furthermore, this deduplication transmission operation reduces transmission bandwidth, which is beneficial for improving transmission efficiency. Detailed explanations follow.

[0074] Example 1

[0075] Please see Figure 1 , Figure 1 This is a flowchart illustrating an edge data management method with deduplication functionality disclosed in an embodiment of the present invention. Figure 1 The described edge data management method with deduplication function can be applied to edge data management devices with deduplication function, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the edge data management method with deduplication functionality can include the following operations:

[0076] 101. When it is determined that the storage service client is in storage running state, switch the backup client to backup running state and generate a data interaction link between the storage service client and the backup client.

[0077] 102. Detect the data transmission status of the first time-series database. The data transmission status includes a first state where data interaction exists or a second state where data interaction does not exist.

[0078] 103. When the data transmission status is determined to be in the first state, the backup client is controlled to obtain the target interaction data of the first time-series database within the preset detection period based on the data interaction link.

[0079] In this embodiment of the invention, the preset detection cycle can be configured and adjusted by the user, and the default cycle can be set to 10 seconds.

[0080] In this embodiment of the invention, the data is collected by the device, and no backup is made if no data is collected.

[0081] 104. Based on the determined deduplication algorithm, control the backup client to perform deduplication on the target interactive data to obtain the deduplication result corresponding to the target interactive data.

[0082] 105. Send the deduplication result to the storage service terminal and control the storage service terminal to write the deduplication result to the corresponding second time-series database of the storage service terminal.

[0083] It is evident that implementation Figure 1 The described edge data management method with deduplication function can automatically build a data link between the backup library and the storage service. When the data transmission status is in the first state, it can automatically perform data deduplication on the acquired target interactive data, realize the periodic backup and deduplication of edge cloud data, improve the disaster recovery capability of edge cloud data, and reduce the occurrence of data loss and irrecoverability due to gateway anomalies. In addition, the deduplication transmission operation reduces the transmission bandwidth, which is conducive to improving transmission efficiency.

[0084] In an optional embodiment, the method of controlling the backup client to perform deduplication on the target interaction data according to the determined deduplication algorithm to obtain the deduplication result corresponding to the target interaction data specifically includes:

[0085] Based on the determined deduplication algorithm, the backup client is controlled to partition the target interaction data according to the data type of the target interaction data, and the data partitioning result corresponding to the target interaction data is obtained. The data partitioning result includes at least one first data block of the first data type and at least one second data block of the second data type. Specifically, the first data block of the first data type is a data block with duplicate data in a continuous data block; the second data block of the second data type is a data block without duplicate data in a continuous data block.

[0086] The backup client determines, based on the deduplication algorithm, which first-level data blocks in all second-level data blocks will undergo deduplication and which second-level data blocks will not undergo deduplication.

[0087] The backup client performs deduplication processing on all first data blocks and first-level data blocks according to the deduplication processing algorithm, and obtains the deduplication results corresponding to all first data blocks and first-level data blocks.

[0088] The deduplication results and secondary data blocks are identified as the deduplication processing results corresponding to the target interactive data.

[0089] In this optional embodiment, further optionally, the method by which the backup client performs deduplication processing on all first data blocks and first-level data blocks according to the deduplication processing algorithm to obtain the deduplication results corresponding to all first data blocks and first-level data blocks may specifically include:

[0090] All first-level data blocks and first-level data blocks are identified as target deduplication data blocks;

[0091] The backup client determines the data attributes corresponding to the target deduplicated data block based on the deduplication algorithm.

[0092] The backup client performs data compression on the target deduplicated data block according to the data attributes corresponding to the target deduplicated data block, and obtains the data compression result corresponding to the target deduplicated data block, which serves as the deduplication result for all first data blocks and first-level data blocks.

[0093] The data compression result corresponding to the target deduplicated data block includes the data compression result corresponding to each first data block and the data compression result corresponding to the first-level data block.

[0094] In this optional embodiment, the deduplication algorithm can specifically be the Repetitive Leakage (RLE) algorithm. The RLE algorithm divides the data into two cases according to a linear sequence: consecutive repeating data blocks and consecutive non-repeating data blocks. For the first case, the consecutive repeating data blocks are compressed by adding a data block to an attribute representing the number of blocks. For the second case, the RLE algorithm has two processing methods: one is to process the consecutive non-repeating data blocks in the same way as in the first case, except that the attribute representing the number of blocks is always 1; the other is to not process the data at all and directly use the original data as the compressed data.

[0095] Taking strings as an example, the encoding rules for RLE are as follows: In a string, a substring consisting of 2 to 9 identical characters is represented by two characters. The first character is the length of the string, which is 2 to 9. The second character is the value of the identical character. If a string has more than 9 identical characters in a substring, the first 9 characters are encoded first, and then the remaining substrings consisting of identical characters are encoded in the same way. For example, AAAAAABCCCC is encoded as 6A1B14C. In a string, if there is a substring in which no character appears consecutively, it is represented by starting with the character 1, followed by the substring, and ending with a character. If there is a substring in the string that contains only one 1 character, it is output as two 1 characters, for example, 12344 is encoded as 11123124.

[0096] As can be seen, in this optional embodiment, the target interactive data can be divided according to data type first, so as to facilitate subsequent data processing operations on data of the same type, thereby improving the efficiency and accuracy of subsequent data processing operations; it can also further identify all data that need to be deduplicated, and perform deduplication processing on all target deduplicated data blocks, thereby improving the efficiency and accuracy of deduplication processing on target deduplicated data blocks.

[0097] Example 2

[0098] Please see Figure 2 , Figure 2 This is a flowchart illustrating another edge data management method with deduplication functionality disclosed in an embodiment of the present invention. Figure 2 The described edge data management method with deduplication function can be applied to edge data management devices with deduplication function, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the edge data management method with deduplication functionality can include the following operations:

[0099] 201. When it is determined that the storage service client is in storage running state, switch the backup client to backup running state and generate a data interaction link between the storage service client and the backup client.

[0100] 202. Detect the data transmission status of the first time-series database. The data transmission status includes a first state where data interaction exists or a second state where data interaction does not exist.

[0101] 203. When the data transmission status is determined to be in the first state, the backup client is controlled to obtain the target interaction data of the first time-series database within the preset detection period based on the data interaction link.

[0102] 204. Based on the determined deduplication algorithm, control the backup client to perform deduplication on the target interactive data to obtain the deduplication result corresponding to the target interactive data.

[0103] 205. Analyze the deduplication results to obtain the deduplication verification value corresponding to the target interactive data.

[0104] In this embodiment of the invention, the deduplication verification value is used to determine the deduplication status of the target interactive data.

[0105] 206. Based on the pre-determined simulated deduplication value, compare it with the deduplication verification value to obtain the data difference between the deduplication verification value and the simulated deduplication value.

[0106] In this embodiment of the invention, the simulated deduplication value is the estimated deduplication value obtained after performing deduplication processing on the simulated interactive data.

[0107] 207. When the data difference is within the preset standard deviation range, the target interactive data is determined to be data that meets the deduplication requirements.

[0108] 208. Send the deduplication result to the storage service terminal and control the storage service terminal to write the deduplication result to the corresponding second time-series database of the storage service terminal.

[0109] For further descriptions of steps 201-204 and 208 in this embodiment of the invention, please refer to the other specific descriptions of steps 101-104 and 105 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0110] It is evident that implementation Figure 2The described edge data management method with deduplication function can automatically analyze the deduplication processing result and obtain the data difference between the simulated deduplication value and the deduplication verification value. If the data difference is within the preset standard deviation range, the target interactive data is determined to be data that meets the deduplication processing requirements. That is, a corresponding deduplication verification process is set up, which improves the accuracy and reliability of the deduplication processing result obtained by performing deduplication processing operation on the target interactive data.

[0111] In an optional embodiment, when the data difference is outside a preset standard deviation range, the method further includes:

[0112] Based on the data difference and the deduplication algorithm, the backup client is controlled to perform a second deduplication operation on the target interactive data until the numerical difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is within the standard deviation range. At this point, the target interactive data is determined to meet the deduplication requirements and the operation of sending the deduplication result to the storage service client is triggered.

[0113] In this optional embodiment, during the process of controlling the backup client to perform a secondary deduplication operation on the target interactive data based on the data difference and the deduplication algorithm, the method may further include:

[0114] For each secondary deduplication operation performed on the target interaction data, the corresponding number of secondary deduplication operations performed on the target interaction data is updated.

[0115] When the number of operations exceeds the preset number of operations, and the difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is outside the standard deviation range, deduplication information for the target interactive data is generated based on the number of operations. The deduplication information is used to stop the secondary deduplication processing operation for the target interactive data and to provide feedback to the processing personnel who process the target interactive data, so as to trigger the processing personnel to perform data analysis processing operations on the target interactive data based on the deduplication information.

[0116] As can be seen, in this optional embodiment, a secondary verification process is further set up, which can perform secondary deduplication processing on data that does not meet the deduplication requirements, thereby improving the accuracy of the final deduplication result. In addition, the number of times the secondary deduplication processing is performed on data that does not meet the deduplication requirements can be controlled based on the number of operations, reducing the waste of resources caused by repeatedly performing secondary deduplication on a certain target interactive data, and improving the reliability of the secondary deduplication process to a certain extent.

[0117] Example 3

[0118] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an edge data management device with deduplication function disclosed in an embodiment of the present invention. The edge data management device with deduplication function can be an edge data management terminal with deduplication function, an edge data management equipment with deduplication function, an edge data management system with deduplication function, or an edge data management server with deduplication function. The edge data management server with deduplication function can be a local server, a remote server, or a cloud server (also known as a cloud server). When the edge data management server with deduplication function is not a cloud server, the non-cloud server can communicate with the cloud server. This embodiment of the present invention does not impose any limitations. Figure 3 As shown, the edge data management device with deduplication function may include a switching interaction module 301, a detection module 302, a control module 303, a deduplication processing module 304, and a sending module 305, wherein:

[0119] The switching interaction module 301 is used to switch the backup client to the backup running state and generate a data interaction link between the storage service client and the backup client when it is determined that the storage service client is in the storage running state.

[0120] The detection module 302 is used to detect the data transmission status of the first time-series database. The data transmission status includes a first state where data interaction exists or a second state where data interaction does not exist.

[0121] The control module 303 is used to control the backup client to obtain the target interactive data of the first time-series database within a preset detection period based on the data interaction link when the data transmission status is determined to be the first state.

[0122] The deduplication module 304 is used to control the backup client to perform deduplication operations on the target interactive data according to the determined deduplication algorithm, so as to obtain the deduplication result corresponding to the target interactive data.

[0123] The sending module 305 is used to send the deduplication result to the storage service.

[0124] The control module 303 is also used to control the storage service terminal to write the deduplication result to the second time-series database corresponding to the storage service terminal.

[0125] It is evident that implementation Figure 3The described edge data management device with deduplication function can automatically build a data link between the backup library and the storage service. When the data transmission state is in the first state, it can automatically perform data deduplication on the acquired target interactive data, realize the periodic backup and deduplication of edge cloud data, improve the disaster recovery capability of edge cloud data, and reduce the occurrence of data loss and irrecoverability due to gateway failure. In addition, the deduplication transmission operation reduces the transmission bandwidth, which is conducive to improving transmission efficiency.

[0126] In an optional embodiment, the deduplication module 304 controls the backup client to perform deduplication processing on the target interactive data according to the determined deduplication processing algorithm, and the specific methods for obtaining the deduplication processing result corresponding to the target interactive data include:

[0127] Based on the determined deduplication algorithm, the backup client is controlled to partition the target interaction data according to the data type of the target interaction data, and the data partitioning result corresponding to the target interaction data is obtained. The data partitioning result includes at least one first data block of the first data type and at least one second data block of the second data type. Specifically, the first data block of the first data type is a data block with duplicate data in a continuous data block; the second data block of the second data type is a data block without duplicate data in a continuous data block.

[0128] The backup client determines, based on the deduplication algorithm, which first-level data blocks in all second-level data blocks will undergo deduplication and which second-level data blocks will not undergo deduplication.

[0129] The backup client performs deduplication processing on all first data blocks and first-level data blocks according to the deduplication processing algorithm, and obtains the deduplication results corresponding to all first data blocks and first-level data blocks.

[0130] The deduplication results and secondary data blocks are identified as the deduplication processing results corresponding to the target interactive data.

[0131] In this optional embodiment, the deduplication module 304 performs deduplication processing on all first data blocks and first-level data blocks by the backup client according to the deduplication processing algorithm, and obtains the deduplication results corresponding to all first data blocks and first-level data blocks in the following specific ways:

[0132] All first-level data blocks and first-level data blocks are identified as target deduplication data blocks;

[0133] The backup client determines the data attributes corresponding to the target deduplicated data block based on the deduplication algorithm.

[0134] The backup client performs data compression on the target deduplicated data block according to the data attributes corresponding to the target deduplicated data block, and obtains the data compression result corresponding to the target deduplicated data block, which serves as the deduplication result for all first data blocks and first-level data blocks.

[0135] The data compression result corresponding to the target deduplicated data block includes the data compression result corresponding to each first data block and the data compression result corresponding to the first-level data block.

[0136] As can be seen, in this optional embodiment, the target interactive data can be divided according to data type first, so as to facilitate subsequent data processing operations on data of the same type, thereby improving the efficiency and accuracy of subsequent data processing operations; it can also further identify all data that need to be deduplicated, and perform deduplication processing on all target deduplicated data blocks, thereby improving the efficiency and accuracy of deduplication processing on target deduplicated data blocks.

[0137] In another alternative embodiment, such as Figure 4 As shown, the device also includes an analysis module 306, a comparison module 307, and a determination module 308, wherein:

[0138] The analysis module 306 is used to analyze the deduplication processing result before the sending module 305 sends the deduplication processing result to the storage service terminal, and obtain the deduplication verification value corresponding to the target interaction data. The deduplication verification value is used to determine the deduplication status of the target interaction data.

[0139] The comparison module 307 is used to compare the deduplication verification value with the pre-determined simulated deduplication value to obtain the data difference between the deduplication verification value and the simulated deduplication value. The simulated deduplication value is the estimated deduplication value obtained after performing deduplication processing on the simulated interactive data.

[0140] The determination module 308 is used to determine that the target interactive data meets the deduplication requirements when the data difference is within the preset standard deviation range, and to trigger the operation of sending the deduplication result to the storage service terminal.

[0141] It is evident that implementation is as follows Figure 4 The described edge data management device with deduplication function can automatically analyze the deduplication processing result and compare the simulated deduplication value with the deduplication verification value to obtain the data difference between the two. If the data difference is within a preset standard deviation range, the target interactive data is determined to be data that meets the deduplication processing requirements. That is, a corresponding deduplication verification process is set up, which improves the accuracy and reliability of the deduplication processing result obtained by performing deduplication processing operation on the target interactive data.

[0142] In another optional embodiment, the control module 303 is further configured to, when the data difference is outside the preset standard deviation range, control the backup client to perform a secondary deduplication operation on the target interactive data according to the data difference and the deduplication algorithm, until it is determined that the numerical difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is within the standard deviation range, determine that the target interactive data meets the deduplication requirements and trigger the execution of the above-mentioned operation of sending the deduplication result to the storage service terminal.

[0143] Optional, such as Figure 4 As shown, the device also includes an update module 309 and a generation module 310, wherein:

[0144] The update module 309 is used to control the backup client to perform a second deduplication operation on the target interactive data based on the data difference and the deduplication algorithm. Each time a second deduplication operation is performed on the target interactive data, the number of operations for performing the second deduplication operation on the target interactive data is updated accordingly.

[0145] The generation module 310 is used to generate deduplication information for the target interactive data based on the number of operations when the number of operations is greater than the preset number of operations and the difference between the deduplication verification data corresponding to the target interactive data and the simulated deduplication value is outside the standard deviation range. The deduplication information is used to stop the secondary deduplication processing operation for the target interactive data and to provide feedback to the processing personnel who process the target interactive data, so as to trigger the processing personnel to perform data analysis processing operation on the target interactive data based on the deduplication information.

[0146] It is evident that implementation is as follows Figure 4 The described edge data management device with deduplication function further incorporates a secondary verification process, which can perform secondary deduplication on data that does not meet the deduplication requirements, thereby improving the accuracy of the final deduplication result. In addition, it can control the number of times secondary deduplication is performed on data that does not meet the deduplication requirements based on the number of operations, reducing the waste of resources caused by repeatedly performing secondary deduplication on a certain target interactive data, and improving the reliability of the secondary deduplication process to a certain extent.

[0147] Example 4

[0148] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of another edge data management device with deduplication function disclosed in an embodiment of the present invention. Figure 5 As shown, the edge data management device with deduplication function may include:

[0149] Memory 401 storing executable program code;

[0150] Processor 402 coupled to memory 401;

[0151] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the edge data management method with deduplication function described in Embodiment 1 or Embodiment 2 of the present invention.

[0152] Example 5

[0153] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the edge data management method with deduplication function described in Embodiment 1 or Embodiment 2 of this invention.

[0154] Example 6

[0155] This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the edge data management method with deduplication function described in Embodiment 1 or Embodiment 2.

[0156] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0158] Finally, it should be noted that the edge data management method and apparatus with deduplication function disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An edge data management method with deduplication function, characterized in that, The method includes: When it is determined that the storage service client is in storage running state, the backup client is switched to backup running state and a data interaction link is generated between the storage service client and the backup client. The data transmission status of the first time-series database is detected, and the data transmission status includes a first state where data interaction exists or a second state where data interaction does not exist. When the data transmission status is determined to be the first status, the backup client is controlled to obtain the target interaction data of the first time series database within a preset detection period based on the data interaction link. The backup client is controlled to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm, so as to obtain the deduplication processing result corresponding to the target interaction data. The deduplication result is sent to the storage service terminal, and the storage service terminal is controlled to write the deduplication result into the second time-series database corresponding to the storage service terminal. The step of controlling the backup client to perform deduplication on the target interaction data according to the determined deduplication algorithm, and obtaining the deduplication result corresponding to the target interaction data, includes: Based on the determined deduplication algorithm, the backup client is controlled to divide the target interaction data according to the data type of the target interaction data to obtain the data division result corresponding to the target interaction data. The data division result includes at least one first data block of a first data type and at least one second data block of a second data type. The backup client determines, according to the deduplication algorithm, the first-level data blocks in all the second data blocks that will undergo deduplication and the second-level data blocks that will not undergo deduplication. The backup client performs the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication algorithm to obtain the deduplication results corresponding to all the first data blocks and the first-level data blocks; The deduplication result and the secondary data block are determined as the deduplication processing result corresponding to the target interactive data.

2. The edge data management method with deduplication function according to claim 1, characterized in that, The step involves the backup client performing the deduplication operation on all the first data blocks and the first-level data blocks according to the deduplication algorithm to obtain the deduplication results corresponding to all the first data blocks and the first-level data blocks, including: All the first data blocks and the first-level data blocks are identified as the target deduplicated data blocks; The backup client determines the data attributes corresponding to the target deduplicated data block according to the deduplication algorithm; The backup client performs a data compression operation on the target deduplicated data block according to the data attributes corresponding to the target deduplicated data block, and obtains the data compression result corresponding to the target deduplicated data block, which serves as the deduplication result for all the first data blocks and the first-level data blocks. The data compression result corresponding to the target deduplicated data block includes the data compression result corresponding to each of the first data blocks and the data compression result corresponding to the first-level data block.

3. The edge data management method with deduplication function according to claim 1 or 2, characterized in that, The first data block of the first data type is specifically a data block containing duplicate data in a continuous data block; the second data block of the second data type is specifically a data block containing no duplicate data in a continuous data block.

4. The edge data management method with deduplication function according to any one of claims 1-3, characterized in that, Before sending the deduplication result to the storage service, the method further includes: Analyze the deduplication results to obtain the deduplication verification value corresponding to the target interaction data. The deduplication verification value is used to determine the deduplication status of the target interaction data. The deduplication verification value is compared with the pre-determined simulated deduplication value to obtain the data difference between the deduplication verification value and the simulated deduplication value. The simulated deduplication value is the estimated deduplication value obtained after performing the deduplication processing operation on the target interactive data. When the data difference is within a preset standard deviation range, the target interactive data is determined to be data that meets the deduplication requirements, and the operation of sending the deduplication result to the storage service terminal is triggered.

5. The edge data management method with deduplication function according to claim 4, characterized in that, When the data difference is outside a preset standard deviation range, the method further includes: Based on the data difference and the deduplication algorithm, the backup client is controlled to perform a second deduplication operation on the target interactive data until the numerical difference between the deduplication verification value and the simulated deduplication value corresponding to the target interactive data is determined to be within the standard deviation range. At this point, the target interactive data is determined to be data that meets the deduplication requirements, and the operation of sending the deduplication result to the storage service client is triggered.

6. The edge data management method with deduplication function according to claim 5, characterized in that, In the process of controlling the backup client to perform a secondary deduplication operation on the target interactive data based on the data difference and the deduplication algorithm, the method further includes: Each time the secondary deduplication operation is performed on the target interactive data, the number of times the secondary deduplication operation is performed on the target interactive data is updated accordingly. When the number of operations is greater than the preset number of operations, and the difference between the deduplication verification value and the simulated deduplication value corresponding to the target interactive data is outside the standard deviation range, deduplication information for the target interactive data is generated based on the number of operations. The deduplication information is used to stop the secondary deduplication processing operation for the target interactive data and to provide feedback to the processing personnel who process the target interactive data, so as to trigger the processing personnel to perform data analysis processing operation on the target interactive data based on the deduplication information.

7. An edge data management and control device with deduplication function, characterized in that, The apparatus is used to perform the edge data management method with deduplication function as described in any one of claims 1-6, and the apparatus comprises: The switching interaction module is used to switch the backup client to the backup running state and generate a data interaction link between the storage service client and the backup client when it is determined that the storage service client is in the storage running state. The detection module is used to detect the data transmission status of the first time-series database, wherein the data transmission status includes a first state in which data interaction exists or a second state in which data interaction does not exist. The control module is used to control the backup client to obtain the target interaction data of the first time series database within a preset detection period based on the data interaction link when the data transmission status is determined to be the first status. The deduplication module is used to control the backup client to perform deduplication processing on the target interaction data according to the determined deduplication processing algorithm, so as to obtain the deduplication processing result corresponding to the target interaction data. The sending module is used to send the deduplication result to the storage service terminal. The control module is also used to control the storage service terminal to write the deduplication result into the second time-series database corresponding to the storage service terminal.

8. An edge data management and control device with deduplication function, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the edge data management method with deduplication function as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the edge data management method with deduplication function as described in any one of claims 1-6.

Images