A method for repetitive processing when writing data to storage in real time

Abstract

This invention discloses a method, apparatus, medium, and device for deduplication during real-time data writing and storage, belonging to the field of data storage technology. The core of the method includes: responding to a data write request and determining the final persistent target storage path for the data to be stored; using this path as the core anchor point to determine the deduplication range, performing a deduplication operation, and determining whether duplicate historical data exists through precise content matching and / or semantic similarity matching; if duplicates exist, deduplication is first performed on the historical data, and then new data is written, with deduplication and writing controlled by strong consistency transactions; if no duplicates exist, the data is written directly. This invention balances deduplication efficiency and comprehensiveness, achieving high-precision deduplication in two dimensions, ensuring data consistency and security. It can be implemented in any way, including software, hardware, firmware, and hardware-software collaboration, and is adaptable to storage applications across all scenarios.

Description

Technical Field

[0001] This invention belongs to the field of data storage and data deduplication technology, and particularly relates to a method, apparatus, medium and device for deduplication processing when writing data to storage in real time. Background Technology

[0002] With the rapid development of big data and information technology, the amount of data stored by individual and enterprise users is growing exponentially. Duplicate data and semantically similar data occupy a large amount of storage space, which not only significantly increases the cost of storage hardware, but also reduces the efficiency of data reading, writing, retrieval and management.

[0003] Existing data deduplication solutions are mainly divided into two categories: offline deduplication and online deduplication. Both have the core drawback of being unable to simultaneously address both: 1. Offline deduplication solutions can only perform batch deduplication on static datasets that have already been stored. They cannot adapt to dynamic scenarios where data is written in real time, and there is a problem of deduplication lag—redundant data is only cleaned up after it has occupied storage space, which cannot reduce storage redundancy from the source. 2. Although online deduplication solutions can perform deduplication during the data writing process, the design of the deduplication scope has inherent flaws: either they adopt a global scan of the entire storage pool, which has extremely low deduplication efficiency and seriously slows down the data writing performance; or they only perform deduplication on a single-level directory specified by the user, which is very easy to miss duplicate data in subdirectories, and cannot balance deduplication efficiency and deduplication comprehensiveness. 3. Existing online deduplication solutions can only achieve accurate content deduplication through hash value matching. They cannot identify similar data with the same semantics but different content formats, layouts, and expressions. A large amount of semantically redundant data cannot be effectively cleaned up, resulting in poor deduplication performance. 4. The deduplication process and data writing operations lack strong consistency control. In the event of power failure, program crash, hardware failure, or other scenarios during the deduplication process, problems such as loss of historical data, failure to write new data, and inconsistency of stored data are very likely to occur, and data security and integrity cannot be guaranteed. 5. The deduplicated historical data lacks a comprehensive lifecycle management mechanism. Users cannot flexibly control the retention, restoration, and deletion of duplicate data, and user operation permissions are restricted, making it impossible to perform unified management of data across the entire storage domain, resulting in a very poor user experience. 6. Existing solutions are mostly customized designs for specific scenarios and can only be implemented through a single software or hardware method. They cannot be adapted to all types of hardware such as personal terminals, enterprise-level servers, and embedded storage devices, nor can they cover all forms of implementation such as software, firmware, and software-hardware collaboration. Their adaptability and universality are extremely poor. Summary of the Invention

[0004] I. Technical problems to be solved The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide a method, apparatus, medium and device for deduplication when writing data to storage in real time. It aims to solve the core technical problems of existing online deduplication schemes, such as the inability to balance deduplication efficiency and deduplication comprehensiveness, the inability to achieve semantic-level high-precision deduplication, poor data consistency and security, insufficient user management flexibility and limited adaptability to various scenarios. Technical solution

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for recurring data writes to storage in real time includes the following steps: S1 responds to a data write request by receiving the data to be stored, parsing and determining the final persistent target storage path corresponding to the data to be stored; S2 Determines a preset deduplication range with the final persistent target storage path as the core anchor point, performs a deduplication operation within the preset deduplication range, and determines whether there is historical data that meets the preset duplication judgment condition with the data to be stored. The preset deduplication range covers at least all logical storage intervals corresponding to the final persistent target storage path and all logical storage intervals corresponding to all its hierarchical subdirectories. The preset duplicate determination conditions include content exact matching conditions and / or semantic similarity matching conditions; wherein, the content exact matching condition is that the binary hash values ​​of the historical data and the data to be stored are completely consistent; the semantic similarity matching condition is that the semantic similarity between the historical data and the data to be stored is greater than or equal to the preset similarity threshold. S3 If there is historical data that meets the preset duplicate determination condition, then the historical data is first subjected to preset deduplication processing. After the preset deduplication processing is completed, the data to be stored or the data obtained after performing deduplication optimization processing on the data to be stored is written to the final persistent target storage path. S4 If there is no historical data that meets the preset duplicate determination condition, the data to be stored is written to the final persistent target storage path.

[0006] Furthermore, in step S2, the deduplication operation adopts a hierarchical deduplication logic, and the execution process is as follows: first, pre-filtering is performed according to the file type and file size of the data to be stored; only for the data to be stored that meets the preset file conditions, the content precise matching initial screening is performed first; only if the initial screening does not find a matching result, the semantic similarity matching secondary screening is performed; the preset file conditions include at least one of the following: the file type belongs to a preset deduplication type and the file size is within a preset size range.

[0007] Further, in step S3, the entire process of the preset duplicate removal process and the write operation are controlled by a strong consistency transaction, and the strong consistency transaction satisfies: the preset duplicate removal process and the write operation are prerequisites for each other's execution. If any link fails to execute, both links are fully rolled back to the initial state before this data write request to ensure data consistency.

[0008] Further, the method includes at least one of the following: ① In step S2, the preset duplicate check range supports custom configuration, and the custom configuration rules include at least one of the duplicate check-free directory exclusion rule, the specified traversal level rule, and the cross-directory global duplicate check rule; ② Step S1 further includes: if it is confirmed after parsing that the final persistent target storage path does not exist, then according to the hierarchical structure of the final persistent target storage path, the corresponding full-scale directory hierarchy is automatically created, and after completion, step S2 is executed.

[0009] Further, the preset duplicate removal process is a duplicate removal-related operation for historical duplicate data to reduce storage redundancy, including but not limited to the operation of moving historical data into a preset isolation storage area, the operation of deleting historical data, the combined storage operation of historical data and data to be stored, and the compression and archiving operation of historical data; the duplicate removal optimization process is an optimization process for data to be stored to reduce storage redundancy, including but not limited to compression processing, duplicate removal pointer mapping processing, and redundant field elimination processing.

[0010] Further, for the historical data moved into the preset isolation storage area, the method further includes the full storage domain data management and user permission control steps配套 with the preset duplicate removal process: perform full life cycle management on the historical data in the isolation storage area according to the preset retention rules, and the preset retention rules include at least one of dynamically adjusting the retention duration according to the preset importance level of the data, triggering automatic cleaning according to the storage capacity occupancy rate, and performing restoration or permanent deletion according to the operation manually triggered by the user; the user has the full storage domain complete deletion permission and can manually trigger the permanent deletion operation of any data under any logical storage path in the normal storage area and any historical data in the isolation storage area; when the restoration operation manually triggered by the user is executed, the corresponding historical data is restored to its original logical storage path before it is moved into the preset isolation storage area.

[0011] Furthermore, the semantic similarity calculation process is as follows: extract the effective text content from the data to be stored and the historical data respectively, generate fixed-dimensional semantic fingerprint vectors corresponding to the effective text content through a pre-trained semantic coding model commonly used in this field, calculate the spatial distance similarity between the two semantic fingerprint vectors, and use the spatial distance similarity as the semantic similarity between the two; the spatial distance similarity includes any one of cosine similarity, Manhattan distance similarity, and Pearson correlation coefficient similarity; the effective text content is the recognizable text content after removing format characters, whitespace characters, and invalid identifiers.

[0012] The present invention also provides a device for repetitive processing when writing data to storage in real time, comprising a group of functional modules for implementing the method described in any of the above claims. The group of functional modules includes at least a data receiving module, a deduplication execution module, a transaction management module, and a data writing module. The modules communicate and cooperate with each other. The group of functional modules can be implemented by any one or more of the following methods: software code, programmable hardware logic circuit, non-programmable hardware logic circuit, firmware, and a combination of software and hardware. The data receiving module is used to perform the operation of step S1 above; The plagiarism detection module is used to perform the operation of step S2 above; The transaction management module is used to perform the above-mentioned strong consistency transaction full-process management operations; The data writing module is used to perform the writing operations in steps S3 and S4 above.

[0013] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the preceding claims.

[0014] The present invention also provides an electronic device, including a processor and a computer-readable storage medium, the computer-readable storage medium storing computer-executable instructions executable by the processor, wherein the processor executes the computer-executable instructions to implement the method described in any of the preceding claims. Beneficial effects

[0015] Compared with the prior art, the present invention has the following outstanding beneficial technical effects: 1. Thoroughly balances deduplication efficiency and comprehensiveness: The deduplication scope is set with the final persistent target storage path as the core anchor point, covering the target path and all its subdirectories by default. This avoids the severe performance loss caused by global scanning of the entire storage pool and completely solves the problem of missing duplicate data in subdirectories when checking single-level directories. At the same time, through hierarchical deduplication logic, the initial screening is completed quickly through pre-filtering and precise matching, and semantic matching is performed only on necessary data for secondary screening. This greatly improves deduplication efficiency while ensuring comprehensiveness.

[0016] 2. Achieve high-precision deduplication in two dimensions: It supports both precise content matching and semantic similarity matching. It can identify completely duplicate binary files as well as similar files with the same semantic core but different formats, layouts, and expressions. It fundamentally solves the industry pain point of not being able to clean up semantically redundant data, and the deduplication effect is far superior to existing technologies.

[0017] 3. Comprehensive protection of data consistency and security: Through strong consistency transaction control, the entire process of deduplication and write operations is clearly defined, and the two stages are mutually dependent. If either stage fails, the entire process will be rolled back to the initial state, which completely avoids the problems of historical data loss, new data writing failure, and data inconsistency in abnormal scenarios, and greatly improves the security and integrity of data storage.

[0018] 4. Comprehensive lifecycle management and full access control: Provides isolated storage and full lifecycle management mechanism for deduplicated historical data, while granting users full deletion and management permissions across the entire storage domain. Users can flexibly control the retention, restoration, and cleanup of all data, completely solving the problems of limited user operation permissions and insufficient management flexibility in existing technologies.

[0019] 5. Universal Adaptability Across All Scenarios and Form Factors: The solution can be implemented through any means, including software code, programmable / non-programmable hardware logic circuits, firmware, and hardware-software collaboration. It can adapt to all types of hardware, such as personal computers, mobile terminals, enterprise-level network-attached storage servers, and industrial embedded storage devices, covering personal, enterprise, and industrial applications across all scenarios. It has extremely strong universality and adaptability. Detailed Implementation

[0020] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0021] Example 1: File Storage Deduplication Method Implemented by Personal Terminal Software This embodiment provides a file storage deduplication method implemented through software code on a desktop operating system personal terminal. The specific steps are as follows: The S1 terminal file manager responds to the user's file save request, receives the document data to be stored, parses the save path selected by the user, and determines that the path is the final persistent target storage path "D:\Work Documents\Project Plans\"; S2 determines the preset deduplication scope with "D:\Work Documents\Project Plans\" as the core anchor point, which by default covers all logical storage areas and all subdirectories corresponding to this path. At the same time, the user has customized the deduplication exclusion rules, excluding the "D:\Work Documents\Project Plans\Temporary Cache\" directory; the deduplication operation is performed within the above deduplication scope. The preset duplicate determination conditions are content exact matching condition + semantic similarity matching condition. The content exact matching condition is that the 256-bit secure hash algorithm value of the historical file and the file to be stored are completely consistent. The semantic similarity matching condition is that the semantic similarity between the two is ≥0.92 (preset similarity threshold). If a historical document that meets the duplicate detection criteria is matched, S3 first performs a preset deduplication process on the historical document and moves it to the preset isolated storage area "D:\Duplicate File Isolation Area\". After the move operation is completed, the document to be stored is written to the path "D:\Work Documents\Project Plan\". The deduplication process and the writing operation are controlled by strong consistency transactions. If the move to the isolation area fails or the document writing fails, both operations are rolled back to the initial state before the current writing request. The historical document is kept in the original path and no new document is written. S4 If no historical data that meets the duplicate detection criteria is found, the document to be stored is directly written to the target storage path.

[0022] Furthermore, in this embodiment, the deduplication operation in step S2 adopts a hierarchical deduplication logic: first, a pre-filter is performed to determine that the file type of the document to be stored is a document format (belonging to the preset deduplication type) and the file size is 2.3 megabytes (within the preset size range of 10 kilobytes to 100 megabytes), which meets the preset file conditions; then, a preliminary screening of precise content matching is performed to calculate the 256-bit secure hash algorithm value of the document to be stored and compare it with the hash value of historical files within the deduplication range. Only when the preliminary screening does not find a completely identical result will a secondary screening of semantic similarity matching be performed.

[0023] The semantic similarity calculation process is as follows: extract the effective text content from the document to be stored and the historical documents respectively, remove invalid identifiers such as format characters, headers and footers, and whitespace characters, generate 768-dimensional semantic fingerprint vectors respectively through the pre-trained semantic coding model commonly used in this field, calculate the cosine similarity between the two vectors, and use the cosine similarity as the semantic similarity between the two.

[0024] For historical documents moved to the isolated storage area, the system performs corresponding full lifecycle management and access control: according to the preset retention rules, ordinary documents are retained for 30 days, and documents marked as high importance are retained for 180 days. When the disk storage capacity utilization rate exceeds 90%, documents whose retention period has expired are automatically cleaned up. Users can manually trigger the restoration operation (restore to the original logical storage path) or permanent deletion operation of historical documents in the isolation area interface. At the same time, users can perform permanent deletion operations on files under any logical storage path in the normal storage area and any document in the isolation area in the file manager, and have full deletion permissions for the entire storage domain.

[0025] Example 2: Storage Deduplication Method Implemented in Industrial Embedded Device Firmware This embodiment provides a real-time data deduplication method implemented through firmware in an industrial embedded storage device. The specific steps are as follows: S1 The microcontroller unit of the embedded device responds to the data write request from the industrial sensor, receives the device operation log data to be stored, parses and determines the preset final persistent target storage path " / mnt / emmc / log / device01 / "; after parsing, it confirms that the path does not exist, automatically creates the corresponding full directory level according to the path hierarchy, and then executes step S2. S2 determines the preset deduplication range with " / mnt / emmc / log / device01 / " as the core anchor point, covering all logical storage areas and all levels of subdirectories corresponding to this path, and performs deduplication operations within this range; the preset deduplication judgment condition is the content exact match condition, that is, the message digest version 5 algorithm value of the historical log data and the data to be stored are completely consistent; S3 If historical log data with completely identical algorithm values ​​is matched, the historical data will be deleted first. After the deletion is completed, the log data to be stored will be written to the target storage path. The deletion and writing operations are controlled by strong consistency transactions. If either step fails, both steps will be rolled back to the initial state before the current write request. If no duplicate data is found in S4, the log data to be stored will be written directly to the target storage path.

[0026] The solution in this embodiment is implemented through embedded firmware, which is adapted to the low-performance and high-reliability requirements of industrial scenarios. It reduces the redundancy of embedded storage chips by precise matching and deduplication, and ensures the integrity and consistency of industrial log data through strong consistency transaction management.

[0027] Example 3: Deduplication Processing Device Implemented with Hardware Logic Circuits This embodiment provides a deduplication device implemented by a field-programmable gate array (FPGA) programmable hardware logic circuit. It includes a functional module group for implementing the method described in Embodiment 1. The functional module group includes a data receiving module, a deduplication execution module, a transaction management module, and a data writing module that are interconnected. Each module is implemented using a hardware description language and works together to complete the entire process of data deduplication and writing. The data receiving module receives data to be stored from external interfaces, parses and determines the final persistent target storage path; the deduplication execution module has built-in hash calculation acceleration circuit and semantic matching acceleration circuit to perform hierarchical deduplication operations; the transaction management module has built-in hardware transaction state machine to manage the entire process of strong consistency transactions for deduplication and write operations; and the data writing module is used to interface with physical storage media and perform data writing operations.

[0028] The device in this embodiment can be integrated into the main control chip of a solid-state drive or network-attached storage device, achieving high-performance real-time deduplication through hardware acceleration. At the same time, it can be adapted to the implementation method of non-programmable hardware logic circuits of application-specific integrated circuits by modifying the hardware description language.

[0029] Example 4: Deduplication Scheme Implemented Through Enterprise-Level Server Hardware and Software Collaboration This embodiment provides a hardware and software co-implemented deduplication scheme in an enterprise-level network attached storage server. The server's central processing unit executes software code to implement path resolution, user permission control, and full lifecycle management of isolated data. A dedicated deduplication acceleration chip enables hardware acceleration of hash calculation and semantic similarity calculation. The firmware of the disk array controller enables strong consistency transaction control. All units work together to implement the method described in any one of claims 1 to 7, adapting to the high performance and high reliability requirements of enterprise-level massive data storage scenarios.

[0030] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for repetitive processing when writing data to storage in real time, characterized in that, Includes the following steps: S1 responds to a data write request by receiving the data to be stored, parsing and determining the final persistent target storage path corresponding to the data to be stored; S2 Determines a preset deduplication range with the final persistent target storage path as the core anchor point, performs a deduplication operation within the preset deduplication range, and determines whether there is historical data that meets the preset duplication judgment condition with the data to be stored. The preset deduplication range covers at least all logical storage intervals corresponding to the final persistent target storage path and all logical storage intervals corresponding to all its hierarchical subdirectories. The preset duplicate determination conditions include content exact matching conditions and / or semantic similarity matching conditions; wherein, the content exact matching condition is that the binary hash values ​​of the historical data and the data to be stored are completely consistent; the semantic similarity matching condition is that the semantic similarity between the historical data and the data to be stored is greater than or equal to the preset similarity threshold. S3 If there is historical data that meets the preset duplicate determination condition, then the historical data is first subjected to preset deduplication processing. After the preset deduplication processing is completed, the data to be stored or the data obtained after performing deduplication optimization processing on the data to be stored is written to the final persistent target storage path. S4 If there is no historical data that meets the preset duplicate determination condition, the data to be stored is written to the final persistent target storage path.

2. The method according to claim 1, characterized in that, In step S2, the deduplication operation adopts a hierarchical deduplication logic, and the execution process is as follows: first, pre-filtering is performed according to the file type and file size of the data to be stored; only for the data to be stored that meets the preset file conditions, the content precise matching initial screening is performed first; only if the initial screening does not find a matching result, the semantic similarity matching secondary screening is performed; the preset file conditions include at least one of the following: the file type belongs to a preset deduplication type and the file size is within a preset size range.

3. The method according to claim 1, characterized in that, In step S3, the entire process of the preset deduplication process and the write operation are controlled by a strong consistency transaction. The strong consistency transaction satisfies the following: the preset deduplication process and the write operation are mutually prerequisites for execution. If either step fails, both steps are rolled back to the initial state before the current data write request to ensure data consistency.

4. The method according to claim 1, characterized in that, Includes at least one of the following: ① In step S2, the preset deduplication range supports custom configuration. The custom configuration rules include at least one of the following: deduplication-free directory exclusion rules, specified traversal level rules, and cross-directory global deduplication rules. ② Step S1 further includes: if it is confirmed after parsing that the final persistent target storage path does not exist, then according to the hierarchical structure of the final persistent target storage path, the corresponding full directory hierarchy is automatically created, and then step S2 is executed.

5. The method according to claim 1, characterized in that, The preset deduplication process refers to deduplication-related operations for duplicate historical data to reduce storage redundancy, including but not limited to operations such as moving historical data into a preset isolation storage area, deleting historical data, merging and storing historical data with data to be stored, and compressing and archiving historical data; the deduplication optimization process refers to an optimization process for data to be stored to reduce storage redundancy, including but not limited to compression processing, deduplication pointer mapping processing, and redundant field elimination processing.

6. The method according to claim 5, characterized in that, For historical data moved into the preset isolation storage area, the method further includes steps of full storage domain data management and user permission control that are配套 with the preset deduplication process: performing full life cycle management on the historical data in the isolation storage area according to preset retention rules, where the preset retention rules include at least one of dynamically adjusting the retention duration according to the preset importance level of the data, triggering automatic cleaning according to the storage capacity occupancy rate, and performing restoration or permanent deletion according to operations manually triggered by the user; the user has the complete deletion permission for the full storage domain and can manually trigger the permanent deletion operation for any data under any logical storage path in the normal storage area and any historical data in the isolation storage area; when the restoration operation manually triggered by the user is executed, the corresponding historical data is restored to its original logical storage path before it was moved into the preset isolation storage area.

7. The method according to claim 1, characterized in that, The calculation process of the semantic similarity is as follows: respectively extract the effective text content in the data to be stored and historical data, generate fixed-dimensional semantic fingerprint vectors corresponding to the effective text content through a pre-trained semantic encoding model common in the art, calculate the spatial distance similarity between the two semantic fingerprint vectors, and use the spatial distance similarity as their semantic similarity; the spatial distance similarity includes any one of cosine similarity, Manhattan distance similarity, and Pearson correlation coefficient similarity; the effective text content is recognizable text content after removing format characters, blank characters, and invalid identifiers.

8. A device for repetitive processing of data written to storage in real time, characterized in that, It includes a functional module group for implementing the method described in any one of claims 1 to 7. The functional module group at least includes a data receiving module, a duplicate checking execution module, a transaction control module, and a data writing module. Each module communicates and cooperates with each other; the functional module group can be implemented by any one or more of software code, programmable hardware logic circuits, non-programmable hardware logic circuits, firmware, and software and hardware cooperation; The data receiving module is used to perform the operation of step S1 in claim 1; The duplicate checking execution module is used to perform the operation of step S2 in claim 1; The transaction control module is used to perform the strong consistency transaction full process control operation described in claim 3; The data writing module is used to perform the writing operations of steps S3 and S4 in claim 1.

9. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored on the computer-readable storage medium. When the computer-executable instructions are executed by a processor, the method described in any one of claims 1 to 7 is implemented; the computer-readable storage medium includes but is not limited to firmware storage media, software program storage media, and embedded storage media.

10. An electronic device, characterized in that, The device includes a processor and a computer-readable storage medium storing computer-executable instructions that can be executed by the processor, wherein the processor executes the computer-executable instructions to implement the method as described in any one of claims 1 to 7; the electronic device includes, but is not limited to, a pure hardware logic device, a hardware-software hybrid device, and an embedded device with firmware.