Information deduplication method and system based on annular memory, medium and equipment

Abstract

The invention provides an information deduplication method and system based on an annular memory, a medium and equipment, and the method comprises the steps: 1, carrying out the creation of a module object through the introduction of a predetermined memory and a memory size, and setting the maximum deduplication length, namely the maximum length of a data repetitive sequence, in the creation process; 2, after a module object is created, entering a parameter checking and initializing stage; 3, after initialization is completed, data are called through an interface, repeated detection is conducted, and organization and updating of data blocks are achieved; 4, taking an input data block as a duplicate judgment standard, reducing an index range according to length data of duplicate removal mark information and recent duplicate information in a data head structure, and performing duplicate removal on an information data block with the highest closest duplicate degree in the index range and an index of the information data block; and 5, storing the data blocks. According to the method, the utilization efficiency of the storage space is optimized, and meanwhile, the accuracy and readability of data recording are improved.

Description

Technical Field

[0001] The present invention relates to the field of information storage technology, and in particular to an information deduplication method, system, medium and device based on a ring memory. Background Art

[0002] With the development of industrial control software systems, the complexity of the system has gradually increased. In order to facilitate the development and maintenance of software, it is very important to store and manage the data generated during the operation of the system. In particular, considering the high degree of data duplication, if effective deduplication is not performed, a large amount of storage space will be wasted. In the existing duplicate detection technology, due to the randomness and repetitiveness of system information, there are often many repeated patterns, which are difficult to effectively remove duplicates, occupy too much storage space, and reduce the readability and analysis efficiency of information. Therefore, in order to improve the stability and performance of industrial control software systems, an innovative solution is needed that can effectively store and deduplicate data to ensure that the system can record various fault conditions in real time and accurately.

[0003] Patent document CN106897197B discloses a technology for managing and deduplicating keywords such as error identifiers and function identifiers in log information. The patent mainly focuses on achieving efficient processing of error information through intelligent calculation of keywords.

[0004] Patent application document CN106844143A discloses a log deduplication processing method and device, which uses a hash value comparison mechanism to detect similar logs and update the number of similar logs, store sample logs and initialize statistical information, and realize deduplication and merging of similar logs. This patent does not provide deduplication support for complex repetitive sequences. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide an information deduplication method, system, medium and device based on ring memory.

[0006] The information deduplication method based on the ring memory provided by the present invention includes:

[0007] Step 1: Create a module object by passing in the predetermined memory and memory size. During the creation process, set the maximum deduplication length, that is, the maximum length of the data duplication sequence;

[0008] Step 2: After the module object is created, it enters the parameter checking and initialization phase;

[0009] If the data from the last shutdown exists in the memory passed in by the user, the historical information and metadata area are located, and the legitimacy of the data is verified to check whether the data needs to be restored; if the data from the last shutdown does not exist, the storage area size, boundary information, location and quantity information of the index area are initialized according to the default configuration, and the deduplication algorithm is initialized at the same time;

[0010] Step 3: After the initialization is completed, the data is called through the interface and duplicate detection is performed to realize the organization and update of the data block. During the update process, according to the arrangement mode and duplication of the historical data and the input data, it is determined whether to directly store or perform deduplication operations. At the same time, the data header structure is updated to record the current data quantity and read and write position information;

[0011] Step 4: Using the input data block as the duplication criterion, the index range is narrowed according to the deduplication flag information in the data header structure and the length data of the most recent duplicate information, and the most recent duplicate information data block and its index within the index range are deduplicated;

[0012] Step 5: Store the data blocks.

[0013] Preferably, the deduplication mode is determined according to the index position difference n of the two duplicate data blocks in the storage area and the most recent deduplication quantity;

[0014] If the two duplicate data are not adjacent, determine whether the index position difference n is greater than the number of recent deduplications. If it is less than the number of recent deduplications, update the data time. If it is greater than or equal to the number of recent deduplications, match the sequence repetition of n data and perform deduplication.

[0015] When two duplicate data are adjacent, delete the duplicate data at the end and set the number of duplicates, update time, and the number of recent deduplications for the adjacent data.

[0016] Based on the length of the most recent repeated information, determine the most recent index position that is repeated with the input data within the data index range. By comparing the most recent repeated information, determine the type of repeated pattern and the frequency of occurrence, and then execute the deduplication algorithm. When encountering a continuous, nested or regular sequence of repeated information, calculate whether there are multiple index positions within the data index range that are repeated with the input data, and then execute the deduplication algorithm separately.

[0017] Preferably, when there are multiple duplicate data, the processing method is to retain a single data;

[0018] When there are multiple sets of repeated data sequences with a length greater than 2, the processing method is to retain the shortest independent sequence;

[0019] When there are multiple sets of repeated data sequences with a length greater than 2, and one or more repeated data are nested in them, the processing method is to retain only the shortest independent sequence and delete the repeated parts;

[0020] When multiple repetitive patterns appear continuously, the processing method is to identify the repetitive patterns, compress the repetitive parts of each sequence, and retain the repetition number information and storage time of adjacent repetitive data.

[0021] Preferably, the data blocks are stored in a data structure of a fixed-size, head-to-tail connected buffer, and whenever new data is stored, the data quantity, related read and write position information, and index information stored in the data header structure are updated;

[0022] By updating the order and length of the data blocks, the position offset of each data in the buffer is updated;

[0023] Perform regular data cleanup operations to delete old data and memory that is no longer needed;

[0024] The relevant information data after executing the deduplication algorithm is written into the preset data storage area.

[0025] The information deduplication system based on the ring memory provided by the present invention includes:

[0026] Module M1: Creates a module object by passing in a predetermined memory and memory size. During the creation process, the maximum deduplication length, i.e., the maximum length of the data duplication sequence, is also set;

[0027] Module M2: After the module object is created, it enters the parameter checking and initialization phase;

[0028] If the data from the last shutdown exists in the memory passed in by the user, the historical information and metadata area are located, and the legitimacy of the data is verified to check whether the data needs to be restored; if the data from the last shutdown does not exist, the storage area size, boundary information, location and quantity information of the index area are initialized according to the default configuration, and the deduplication algorithm is initialized at the same time;

[0029] Module M3: After initialization, call the data through the interface and perform duplicate detection to realize the organization and update of data blocks. During the update process, determine whether to directly store or perform deduplication operations based on the arrangement mode and duplication of historical data and input data. At the same time, update the data header structure and record the current data quantity and read and write position information;

[0030] Module M4: using the input data block as a duplication criterion, narrowing the index range according to the deduplication flag information in the data header structure and the length data of the most recent duplicate information, and deduplicating the most recent duplicate information data block and its index within the index range;

[0031] Module M5: stores data blocks.

[0032] Preferably, the deduplication mode is determined according to the index position difference n of the two duplicate data blocks in the storage area and the most recent deduplication quantity;

[0033] If the two duplicate data are not adjacent, determine whether the index position difference n is greater than the number of recent deduplications. If it is less than the number of recent deduplications, update the data time. If it is greater than or equal to the number of recent deduplications, match the sequence repetition of n data and perform deduplication.

[0034] When two duplicate data are adjacent, delete the duplicate data at the end and set the number of duplicates, update time, and the number of recent deduplications for the adjacent data.

[0035] Based on the length of the most recent repeated information, determine the most recent index position that is repeated with the input data within the data index range. By comparing the most recent repeated information, determine the type of repeated pattern and the frequency of occurrence, and then execute the deduplication algorithm. When encountering a continuous, nested or regular sequence of repeated information, calculate whether there are multiple index positions within the data index range that are repeated with the input data, and then execute the deduplication algorithm separately.

[0036] Preferably, when there are multiple duplicate data, the processing method is to retain a single data;

[0037] When there are multiple sets of repeated data sequences with a length greater than 2, the processing method is to retain the shortest independent sequence;

[0038] When there are multiple sets of repeated data sequences with a length greater than 2, and one or more repeated data are nested in them, the processing method is to retain only the shortest independent sequence and delete the repeated parts;

[0039] When multiple repetitive patterns appear continuously, the processing method is to identify the repetitive patterns, compress the repetitive parts of each sequence, and retain the repetition number information and storage time of adjacent repetitive data.

[0040] Preferably, the data blocks are stored in a data structure of a fixed-size, head-to-tail connected buffer, and whenever new data is stored, the data quantity, related read and write position information, and index information stored in the data header structure are updated;

[0041] By updating the order and length of the data blocks, the position offset of each data in the buffer is updated;

[0042] Perform regular data cleanup operations to delete old data and memory that is no longer needed;

[0043] The relevant information data after executing the deduplication algorithm is written into the preset data storage area.

[0044] According to the computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the information deduplication method based on the ring memory are implemented.

[0045] The electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the information deduplication method based on the ring memory are implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The present invention focuses on the design of efficient data storage and accurate deduplication. By adopting a ring buffer and an index storage ring buffer data structure, the system realizes efficient data storage. For different complex repetitive patterns, the system flexibly selects an appropriate deduplication mode according to a preset strategy, including the processing of ordered sequences and overlapping sequences. Through a flexible deduplication strategy, storage redundancy is effectively reduced and the deduplication accuracy of the system is improved.

[0048] (2) The present invention compares the information data stored in the ring memory with the newly input data for redundancy. When it is identified that the new data or the combination pattern containing the new data has repetitiveness with the preceding sequence, the duplicate data is eliminated and the identification of the original data area is updated. This method optimizes the utilization efficiency of the storage space and improves the accuracy and readability of the data record.

[0049] (3) The present invention can not only effectively solve the deficiencies of the prior art in data management, but also achieve remarkable beneficial effects in storage efficiency, deduplication accuracy, system stability, etc., and provides an innovative solution for data processing of industrial control software systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0051] Figure 1 This is a block diagram of the information deduplication system;

[0052] Figure 2 is a schematic diagram of the storage unit structure;

[0053] Figure 3 is a deduplication flow chart of the deduplication unit;

[0054] Figure 4 De-duplicate results for different data. DETAILED DESCRIPTION

[0055] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0056] Example 1

[0057] The present invention provides an information deduplication method based on a ring memory, designs a data structure for storing information and a corresponding deduplication algorithm, and stores and deduplicates information data in an input storage area. In a typical deduplication process, the information data stored in the ring memory is redundantly compared with the newly input data. When it is recognized that the new data or the combination pattern containing the new data is repeated with the preceding sequence, the duplicate data is eliminated and the identification of the original data area is updated. This method optimizes the utilization efficiency of the storage space and improves the accuracy and readability of the data record. Figure 1 : is a system block diagram showing the above method plus data block storage technology, which mainly includes the following parts:

[0058] 1. Input data stream and data block: The input data stream is variable-length input data containing specific descriptive information. The data block is the smallest unit of information storage in the system, which contains information about the length of the information, error level, whether it belongs to a certain group, number of repetitions, generation timestamp, system encoding, and custom parameters.

[0059] 2. Duplicate detection unit: When new information data needs to be stored in the system, the duplicate detection unit first receives the data. After that, the duplicate detection unit will perform a redundancy comparison with the data stored in the ring memory. The purpose of this step is to find out whether there is data similar to or duplicated with the new data. By comparing the content, combination pattern or other features of the new data with the stored data, the duplicate detection unit can quickly identify whether there is duplicate data. If it is found that the stored data is duplicated with the new data, the information deduplication unit will perform the corresponding processing operation.

[0060] 3. Information deduplication unit: Figure 3 The figure shows the deduplication flow chart of the deduplication unit. After receiving new data, the deduplication unit will compare the information bits of the data in the storage unit and the stored key information to detect whether there is a data block or data block combination in the preset storage space that meets the preset similarity condition compared with the sample information. When similar conditions or repetitiveness are identified, the duplicate data is eliminated and the identification of the original data area is updated to ensure that only unique data is stored in the system.

[0061] 4. Data storage unit: The storage unit includes metadata area, data index area and ring data storage area. The data index area is responsible for storing the index of the data block in the storage area, while the ring data storage area is used to store the actual information data.

[0062] Metadata area: It is placed in the header area of ​​the entire storage medium and is used to manage the data storage area as a whole. It contains key information describing the data structure, such as the size and boundary record information of the storage area, the location and quantity information of the index area, the information recording the current data quantity and read / write position, and various control information. In addition to the control information, the data structure also maintains the flag information of whether to deduplicate the input data, the length of the most recent duplicate information, and other calculation data for the deduplication algorithm.

[0063] Ring storage area: The ring data storage area is the area where information data is actually stored. It consists of a metadata area and multiple data blocks. The metadata header is used to store meta-information related to the data block, such as the number of data blocks, read and write positions, checksums, etc. Each data block contains a data header and actual information data, and the actual information data is stored in the main body of the data block. The entire ring storage area is a data structure with fixed capacity and connected end to end. Its cyclic characteristics allow the system to manage data in a sliding window manner, which can continuously receive new data and overwrite old data.

[0064] Data index area: The data index area is a key component that is used to store index information for data blocks in the circular data storage area. The index information may include key attributes such as the location, size, and timestamp of the data block, which is used to quickly locate and access specific data blocks.

[0065] Figure 2 The main process of the information data deduplication system is displayed, and the steps are as follows:

[0066] Step 1: First, create a module object. Create the object by passing in the specified memory and memory size. During the creation process, set the maximum deduplication length, that is, the maximum length of the data duplicate sequence. This setting provides basic parameters for the subsequent deduplication algorithm to flexibly respond to data processing requirements in different situations.

[0067] Step 2: After the module object is created, it enters the parameter checking and initialization phase. During this process, if the memory passed in by the user contains data from the last shutdown, the system can locate the historical information and metadata area, verify the legitimacy of the data, and check whether the data needs to be restored. If not, the key parameters such as the storage area size, boundary information, and the location and quantity of the index area are initialized according to the default configuration. At the same time, the deduplication algorithm is initialized to ensure the correct setting of the maximum deduplication length. This step prepares for subsequent data storage and deduplication operations.

[0068] Step 3: Input information data. Once the initialization is completed, the system can accept external input data. By calling the corresponding interface, the data is passed to the duplicate detection unit to implement the organization and update operations of the data block. During the update process, the system will determine whether to directly store or perform deduplication operations based on the arrangement pattern and repetition of the historical data and the input data. At the same time, the data header structure is updated to record the current data quantity, read and write position and other information to ensure the orderly storage of data. The judgment conditions are: if the repetition rate between two data exceeds a certain threshold, it is judged as duplicate data; if each data between two groups of data is duplicate data, and the relative position and quantity ratio in the sequence are consistent, the two groups are judged as duplicate data groups.

[0069] Step 4: Data deduplication. Using the simple duplication detection method, the basis for information duplication detection is the input data block as the duplication detection standard. According to the deduplication flag information in the data header structure and the length of the most recent duplicate information, the index range to be considered is narrowed down. The data blocks within the index range are judged, and if the most recent duplicate information data block and its index are found, deduplication is performed.

[0070] The system determines the deduplication mode based on the index position difference n of the two duplicate data blocks in the storage area and the most recent deduplication quantity.

[0071] If two duplicate data are not adjacent, determine whether the index distance n is greater than the number of recent deduplication. If it is less than the number of recent deduplication, update the data time; if it is greater than or equal to the number of recent deduplication, match the sequence repetition of n data and perform subsequent deduplication operations.

[0072] When two duplicate data are adjacent, delete the duplicate data at the end and set the number of duplicates, update time, and the number of recent deduplications for the adjacent data.

[0073] Based on the length of the most recent duplicate information, the most recent index position that is a duplicate of the input data within the data index range can be determined. By comparing the most recent duplicate information, the system can more quickly determine the type of duplicate pattern, the frequency of occurrence, etc., thereby more effectively executing the deduplication algorithm. When encountering a continuous, nested, or regular sequence of duplicate information, the system will also calculate whether there are multiple index positions within the data index range that are duplicates of the input data. This index range information is usually determined by the system based on a preset strategy to search for possible duplicate data patterns within the range. These factors can help the system determine which deduplication mode to use, so that it can more flexibly respond to different duplicate data situations.

[0074] Finally, the deduplication processed data blocks are sent to the storage unit to update the relevant metadata and data index area. The update of the ring data area and regular data cleaning operations are triggered to ensure the normal operation of the system and the effective use of storage space. The following are several system preset deduplication standards and results, such as Figure 3 The system will flexibly select the appropriate processing method based on the preset strategy.

[0075] When there are multiple duplicate data, the processing method is to retain a single data.

[0076] When there are multiple sets of repeated data sequences with a length greater than 2, the processing method is to retain the shortest independent sequence.

[0077] When there are multiple sets of repeated data sequences with a length greater than 2, and one or more repeated data are nested in them, the processing method is to retain only the shortest independent sequence and delete the repeated parts to ensure the uniqueness of each message.

[0078] When multiple repetitive patterns appear continuously, the processing method is to identify the repetitive patterns, compress the repetitive parts of each sequence, and retain key information such as the number of repetitions of adjacent repetitive data and storage time to distinguish different patterns.

[0079] Step 5: The data storage unit stores the data block. After the preparation and inspection of the above steps, the system will perform the following key operations.

[0080] Step 5.1: In the storage unit, the system ensures the real-time and accuracy of the data by updating the data area. Figure 2 As shown, the data blocks in the storage unit are stored in a data structure of a fixed-size, head-to-tail connected buffer. Whenever new data is stored, the data quantity in the data header structure and the related read and write position information are updated, and the original data content is updated according to the stored index information to ensure that the operation of the data does not affect the consistency of the overall data.

[0081] Step 5.2: Update index data. Complementing step 4 is the update of index information. The system updates the position offset of each piece of data in the data storage ring buffer by updating the order and length between the data blocks. This step enables the system to quickly locate specific data in subsequent fast queries and operations. (The relative offset of the storage data block is the basis for deduplication. It is the relative position order between the latest n pieces of data input (n is the default value), and the position offset of each piece of data in the data storage ring buffer can be regarded as the absolute order of each piece of data)

[0082] Step 5.3: Trigger ring data area maintenance. The system performs regular data cleanup operations to delete old data and junk memory that are no longer needed. This step helps to keep the storage area clean, reduce the burden of data parsing, and ensure the timeliness of the data.

[0083] Step 5.4: Store information data. The system writes the relevant information data after executing the deduplication algorithm into the data storage area.

[0084] Example 2

[0085] The present invention provides an information deduplication method based on a ring memory, which, based on Embodiment 1, further includes:

[0086] 1. Expansion of modular design: The present invention can be applied to larger-scale systems through further modular design to meet the needs of complex industrial control software systems. The ring buffer and deduplication algorithm are modularized to make them easier to embed and apply to various software systems.

[0087] 2. Diversified initialization parameter settings: Further expand the setting range of initialization parameters to make them more diversified to meet the different needs of different industries, different application scenarios and devices. Users can more flexibly set the storage area size, boundary information, etc. to meet the requirements of specific application environments.

[0088] In the process of this method, first, the user creates a module object, creates the object by specifying the memory and memory size, and sets the maximum deduplication length, providing basic parameters for the subsequent deduplication algorithm. Then, the module object enters the parameter checking and initialization phase. The system checks the memory and parameters passed in by the user, initializes key parameters such as the storage area size, boundary information, and the location and quantity information of the index area, and initializes the deduplication algorithm to ensure the correct setting of the maximum deduplication length.

[0089] Once the initialization is completed, the system begins to accept external input information data. The user calls the corresponding interface to pass the information data to the ring buffer to achieve orderly storage of data. At this time, the data header structure is responsible for recording the current data quantity, read and write position and other information to ensure orderly storage of data.

[0090] After the information data is input, the system enters the deduplication operation stage. According to the preset deduplication strategy, the system monitors the storage of data. When the system detects information to be deduplicated, the deduplication algorithm is triggered. The deduplication algorithm calculates the nearest index position that is repeated with the input data within the data index range based on the deduplication flag information in the data header structure and the length of the most recent repeated information, and determines the deduplication mode. If the system determines that deduplication is not necessary, it will directly proceed to the next step.

[0091] Deduplication is based on key information in the metadata structure and storage data format. By comparing the most recent duplicate information, the system can more accurately determine whether similar data exists, thereby executing the deduplication algorithm more effectively. The algorithm considers the most recent index position within the data index range that is a duplicate of the input data, and searches for possible duplicate data patterns within the range according to the preset strategy. It also considers other factors, such as the type of duplicate pattern and the frequency of occurrence.

[0092] Subsequently, the system selects a deduplication mode based on the calculated index information. There are two options: single-repeat mode and multiple-repeat mode. Single-repeat mode is suitable for independent duplicate items, while multiple-repeat mode is suitable for continuous, overlapping or regular repeated information sequences. The system performs specific data deduplication operations based on the selected deduplication mode, including retaining a single piece of data, retaining the shortest independent sequence, and other processing methods.

[0093] Finally, the system performs data update and storage operations to ensure the real-time and accuracy of the data. This includes updating the data area, updating the index information, and regular maintenance of the ring data area to delete old data and junk memory that are no longer needed. The entire process enables the system to efficiently manage and deduplicate information data through the design of the ring buffer and the flexible application of the deduplication algorithm, providing an innovative solution for the industrial control software system. The present invention realizes efficient storage and deduplication of data by providing static storage space and predefined parameters.

[0094] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0095] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for deduplication of information based on ring memory, characterized in that: include: Step 1: Create a module object by passing in the predetermined memory and memory size. During the creation process, set the maximum deduplication length, that is, the maximum length of the data duplication sequence; Step 2: After the module object is created, it enters the parameter checking and initialization phase; If the data from the last shutdown exists in the memory passed in by the user, the historical information and metadata area are located, and the legitimacy of the data is verified to check whether the data needs to be restored; if the data from the last shutdown does not exist, the storage area size, boundary information, location and quantity information of the index area are initialized according to the default configuration, and the deduplication algorithm is initialized at the same time; Step 3: After the initialization is completed, the data is called through the interface and duplicate detection is performed to realize the organization and update of the data block. During the update process, according to the arrangement mode and duplication of the historical data and the input data, it is determined whether to directly store or perform deduplication operations. At the same time, the data header structure is updated to record the current data quantity and read and write position information; Step 4: Using the input data block as the duplication criterion, the index range is narrowed according to the deduplication flag information in the data header structure and the length data of the most recent duplicate information, and the most recent duplicate information data block and its index within the index range are deduplicated; Step 5: Store the data blocks.

2. The information deduplication method based on ring memory according to claim 1 is characterized in that: Determine the deduplication mode based on the index position difference n of the two duplicate data blocks in the storage area and the most recent deduplication quantity; If the two duplicate data are not adjacent, determine whether the index position difference n is greater than the number of recent deduplications. If it is less than the number of recent deduplications, update the data time. If it is greater than or equal to the number of recent deduplications, the sequence repetition of matching length n data is executed to perform deduplication; When two duplicate data are adjacent, delete the duplicate data at the end and set the number of duplicates, update time, and the number of recent deduplications for the adjacent data. Based on the length of the most recent repeated information, determine the most recent index position that is repeated with the input data within the data index range. By comparing the most recent repeated information, determine the type of repeated pattern and the frequency of occurrence, and then execute the deduplication algorithm. When encountering a continuous, nested or regular sequence of repeated information, calculate whether there are multiple index positions within the data index range that are repeated with the input data, and then execute the deduplication algorithm separately.

3. The information deduplication method based on ring memory according to claim 1 is characterized in that: When there are multiple duplicate data, the processing method is to retain a single data; When there are multiple sets of repeated data sequences with a length greater than 2, the processing method is to retain the shortest independent sequence; When there are multiple sets of repeated data sequences with a length greater than 2, and one or more repeated data are nested in them, the processing method is to retain only the shortest independent sequence and delete the repeated parts; When multiple repetitive patterns appear continuously, the processing method is to identify the repetitive patterns, compress the repetitive parts of each sequence, and retain the repetition number information and storage time of adjacent repetitive data.

4. The method for deduplication of information based on ring memory according to claim 1, characterized in that: The data blocks are stored in a fixed-size, head-to-tail connected buffer data structure. Whenever new data is stored, the data quantity in the data header structure, the related read and write position information, and the stored index information are updated; By updating the order and length of the data blocks, the position offset of each data in the buffer is updated; Perform regular data cleanup operations to delete old data and memory that is no longer needed; The relevant information data after executing the deduplication algorithm is written into the preset data storage area.

5. An information deduplication system based on ring memory, characterized in that: include: Module M1: Creates a module object by passing in a predetermined memory and memory size. During the creation process, the maximum deduplication length, i.e., the maximum length of the data duplication sequence, is also set; Module M2: After the module object is created, it enters the parameter checking and initialization phase; If the data from the last shutdown exists in the memory passed in by the user, the historical information and metadata area are located, and the legitimacy of the data is verified to check whether the data needs to be restored; if the data from the last shutdown does not exist, the storage area size, boundary information, location and quantity information of the index area are initialized according to the default configuration, and the deduplication algorithm is initialized at the same time; Module M3: After initialization, call the data through the interface and perform duplicate detection to realize the organization and update of data blocks. During the update process, determine whether to directly store or perform deduplication operations based on the arrangement mode and duplication of historical data and input data. At the same time, update the data header structure and record the current data quantity and read and write position information; Module M4: using the input data block as a duplication criterion, narrowing the index range according to the deduplication flag information in the data header structure and the length data of the most recent duplicate information, and deduplicating the most recent duplicate information data block and its index within the index range; Module M5: stores data blocks.

6. The information deduplication system based on ring memory according to claim 5, characterized in that: Determine the deduplication mode based on the index position difference n of the two duplicate data blocks in the storage area and the most recent deduplication quantity; If the two duplicate data are not adjacent, determine whether the index position difference n is greater than the number of recent deduplications. If it is less than the number of recent deduplications, update the data time. If it is greater than or equal to the number of recent deduplications, the sequence repetition of matching length n data is executed to perform deduplication; When two duplicate data are adjacent, delete the duplicate data at the end and set the number of duplicates, update time, and the number of recent deduplications for the adjacent data. Based on the length of the most recent repeated information, determine the most recent index position that is repeated with the input data within the data index range. By comparing the most recent repeated information, determine the type of repeated pattern and the frequency of occurrence, and then execute the deduplication algorithm. When encountering a continuous, nested or regular sequence of repeated information, calculate whether there are multiple index positions within the data index range that are repeated with the input data, and then execute the deduplication algorithm separately.

7. The information deduplication system based on ring memory according to claim 5, characterized in that: When there are multiple duplicate data, the processing method is to retain a single data; When there are multiple sets of repeated data sequences with a length greater than 2, the processing method is to retain the shortest independent sequence; When there are multiple sets of repeated data sequences with a length greater than 2, and one or more repeated data are nested in them, the processing method is to retain only the shortest independent sequence and delete the repeated parts; When multiple repetitive patterns appear continuously, the processing method is to identify the repetitive patterns, compress the repetitive parts of each sequence, and retain the repetition number information and storage time of adjacent repetitive data.

8. The information deduplication system based on ring memory according to claim 5, characterized in that: The data blocks are stored in a fixed-size, head-to-tail connected buffer data structure. Whenever new data is stored, the data quantity in the data header structure, the related read and write position information, and the stored index information are updated; By updating the order and length of the data blocks, the position offset of each data in the buffer is updated; Perform regular data cleanup operations to delete old data and memory that is no longer needed; The relevant information data after executing the deduplication algorithm is written into the preset data storage area.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the information deduplication method based on a ring memory as described in any one of claims 1 to 4 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the information deduplication method based on a ring memory as described in any one of claims 1 to 4 are implemented.

Images