A method for automatically inheriting bloodline compliance attributes of a WORM storage file

Abstract

The application discloses a WORM storage file bloodline compliance attribute automatic inheritance method and belongs to the field of distributed storage and data compliance management. The method completes WORM compliance semantic standardization definition, analyzes file compliance attribute generation rule templates and binds with file unique identification; captures file full operation identification and generates derivative file generation behavior, generates bloodline identification and binds associated information; constructs a full-link bloodline relationship directed acyclic graph with compliance files as root nodes and persists; based on bloodline relationship matching compliance rule templates, adopts transactional atomic operation to complete automatic inheritance and synchronous change of derivative file compliance attributes, and synchronously updates audit logs. The application realizes automatic transmission of derivative file compliance attributes, guarantees compliance cycle continuity and audit link integrity, reduces manual configuration workload, and reduces compliance risks caused by manual operation.

Description

Technical Field

[0001] This invention relates to the field of distributed storage and data compliance management, and in particular to a method for automatically inheriting the lineage compliance attributes of WORM storage files. Background Technology

[0002] WORM (Write Once Read Many) storage, as a core technology for ensuring data immutability, is an indispensable storage foundation in various scenarios requiring data retention. With the advancement of digitalization, WORM storage has been widely applied in data archiving and compliance management scenarios across multiple industries, supporting the long-term secure storage of various business data. The development of distributed storage architecture has driven the evolution of WORM storage from traditional single-node storage to large-scale cluster deployments. Current mainstream distributed WORM storage systems support multiple standard access protocols, adapting to the access needs of different business systems. Existing technologies have enabled WORM compliance attribute configuration for individual files and directories, allowing for basic control over file retention periods and read / write permissions. File lineage tracing technology is gradually being applied in the field of data governance, enabling basic recording of data flow processes and providing support for data traceability and asset management. Simultaneously, compliance auditing technologies are continuously developing, logging common file operations to meet basic audit traceability needs.

[0003] Existing distributed WORM storage systems mostly use manual, static methods for configuring compliance attributes, lacking a unified and standardized WORM compliance semantic definition. This prevents unified resolution and cross-system transfer of compliance attributes across different systems. Current technologies cannot automatically identify derivative relationships between files, comprehensively capture all derivative operations throughout a file's lifecycle, or construct a complete lineage chain covering all derived files. New files derived from compliant files do not automatically inherit the original file's WORM compliance attributes, requiring manual configuration one by one. Furthermore, the configuration of compliance attributes cannot be atomically synchronized with the file creation process, resulting in time windows where compliance attributes are not fully configured. When the compliance attributes of the original file change, the changes cannot be automatically synchronized to all related derived files, causing interruptions in the compliance cycle of compliant files and disrupting the integrity of the compliance audit chain. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for automatic inheritance of lineage compliance attributes of WORM storage files.

[0005] The objective of this invention is achieved through the following technical solution: A method for automatically inheriting lineage compliance attributes of WORM storage files is provided, which includes the following steps: S1. Complete the WORM compliance semantic standardization definition, parse the WORM compliance attributes of the file, generate a compliance rule template, bind the compliance rule template with the file's unique identifier and persist it to the metadata; S2. Capture all operations on the file, identify the source file based on the file's unique identifier, identify the behavior of generating derived files, generate a unique lineage identifier for the derived file, and bind the derived file to the parent file's file unique identifier, operation type, operation time, operation subject, and data source range; S3. Using WORM compliance files as root nodes, derived files as child nodes, and lineage relationships as directed edges, construct a full-link lineage relationship directed acyclic graph based on lineage identifiers and the unique file identifiers of parent files, and persist the lineage relationship directed acyclic graph to the WORM compliance metadata pool. S4. Based on the directed acyclic graph of lineage, obtain the unique file identifier of the parent file corresponding to the derived file, match the compliance rule template corresponding to the parent file, generate the WORM compliance attribute of the derived file, and use transactional atomic operations to complete the creation of the derived file and the synchronization of the WORM compliance attribute configuration, and synchronously update the directed acyclic graph of lineage and the WORM compliance audit log.

[0006] Furthermore, step S1 includes the following sub-steps: S1.1. Decompose WORM attributes into inheritable fields, synchronizeable fields, and verifiable fields to complete the standardized definition of WORM core compliance semantics. The standardized fields include compliance retention period, unmodifiable flag, undeletable lock flag, audit level, compliance level, data retention requirements, storage media lock requirements, and access control rules. S1.2. Based on the WORM compliance semantics of the parsing file, which is based on inheritable fields, synchronizeable fields, and verifiable fields, remove personalized non-compliant fields, generate a compliance rule template, and clearly mark inheritable fields, mandatory inheritance fields, and unchangeable fields in the compliance rule template; S1.3. Bind the compliance rule template to the unique file identifier, persist it to the metadata using an append write method, and simultaneously write it to the WORM compliance audit log.

[0007] Furthermore, step S2 includes the following sub-steps: S2.1. Embed full-link tracing hooks at all operation entry points of POSIX, object, NFS and CIFS protocols to capture all file operations in real time and record the initiating entity, operation time, operation path and operation data range of the operation; S2.2. Determine the derivative file generation behavior based on operation characteristics. The derivative file generation behavior includes file copying, file exporting, file format conversion, file content extraction, file copy generation, file fragment download and retransmission, and cross-protocol read and write to generate new files; S2.3. Generate a unique lineage identifier for each derived file, bind the parent file's unique file identifier, operation type, operation time, operation subject, and data source range, and output it to the lineage relationship directed acyclic graph construction process.

[0008] Furthermore, step S3 includes the following sub-steps: S3.1. Using WORM compliant files as the root node, derived files as child and grandchild nodes, and lineage relationships as directed edges, construct a directed acyclic graph of the entire lineage relationship based on the lineage identifier and the unique file identifier of the parent file; S3.2. Record the corresponding file unique identifier, compliance rule template, operation subject, generation time and blood relationship in each node of the directed acyclic graph of bloodline relationship, and support backtracking from any derived file to the root WORM compliance file; S3.3. The directed acyclic graph of lineage is persisted to the WORM compliance metadata pool by appending to the graph, and the WORM compliance audit log is updated simultaneously to record the generation time and the generating entity of the directed acyclic graph of lineage.

[0009] Furthermore, step S4 includes the following sub-steps: S4.1. When a derived file is generated, obtain the unique file identifier of the parent file corresponding to the derived file based on the directed acyclic graph of lineage, and verify the validity and completeness of the compliance rule template of the parent file; S4.2. Based on the unique file identifier of the parent file, match the corresponding compliance rule template of the parent file, extract the mandatory inheritance field and inheritable field in the compliance rule template, and generate the WORM compliance attribute of the derived file; S4.3. When creating the metadata of the derived file, synchronously write the inherited WORM compliance attributes, first complete the compliance attribute configuration locking, verify the integrity of attribute writing, and then open the data writing permissions of the file; S4.4. Synchronously update the directed acyclic graph of bloodline relationships and the WORM compliance audit log, recording all inherited fields, inheritance time, and operation subject.

[0010] Furthermore, in step S2, derivative files generated across protocols such as POSIX, Object, NFS, and CIFS are included in the full-link lineage tracing scope. The data source scope includes file byte offset, file data block identifier, and file content hash value. For derivative files that only use part of the source file's data, it is supported to configure differentiated inheritance rules based on the data source scope. For the set regulatory scenarios, it is supported to enable permanent locking of the WORM attribute of the derivative file. Permanent locking of the WORM attribute is achieved by writing to the WORM compliance metadata pool and is completed by appending. After the permanent lock takes effect, the WORM compliance attribute of the derivative file can only have its retention period extended and other compliance attributes cannot be changed.

[0011] Furthermore, in step S4, when the compliance attributes of the root WORM file undergo compliance changes, these changes include extending the compliance retention period, upgrading the compliance level, adjusting the audit level, and changing storage media locking requirements. Based on the lineage relationship, the derived files under the root node are traversed in a directed acyclic graph, and atomic compliance attribute synchronization changes are performed. The change operation adopts a full-chain transactional atomic commit, which includes four stages: pre-commit, verification, formal commit, and rollback. If any derived file change fails, all changes are rolled back. At the same time, an immutable WORM compliance audit log is generated, which records the attributes before the change, the attributes after the change, the entity initiating the change, and the change time.

[0012] Furthermore, in step S4, the WORM compliance attributes of files are verified in real time at all file operation entry points of the storage protocol layer, metadata operation interface, and data read / write interface. Real-time verification is triggered when a file is opened, written, its attributes are modified, or deleted. Operations that violate inheritance rules are blocked. Operations that violate inheritance rules include modifying locked files, deleting locked files, shortening the compliance retention period, and lowering the compliance level. For derived files that have not completed the inheritance of compliance attributes, an unready flag is set in the metadata, and all data write and modification operations are blocked. The unready flag is cleared and the corresponding permissions are granted only after atomic inheritance is completed.

[0013] Furthermore, in step S4.2, the WORM compliance attributes of the derived file are generated according to the set rules. The mandatory inheritance fields include compliance retention period, unmodifiable flag, undeletable lock flag and compliance level. The mandatory inheritance fields must be fully inherited. The retention deadline of the derived file must not be earlier than the retention deadline of the parent file. The lock flag must be consistent with the parent file. The inheritable fields include audit level, access control rules, storage media lock requirements and data retention requirements. The inheritable fields are fully inherited by default. They can be adjusted upwards based on regulatory requirements, but cannot be adjusted downwards.

[0014] Furthermore, in step S4.2, when a derived file originates from two or more WORM parent files, the derived file generation scenarios include file merging, data concatenation, and multi-source data aggregation. The inheritance rule adopts the principle of choosing the higher value over the lower value. The retention period is taken as the maximum value of the retention periods of all parent files, the compliance level is taken as the highest value of the compliance level of all parent files, the locking flag is taken as the most stringent value of the locking flag of all parent files, and the audit level is taken as the highest value of the audit level of all parent files. The WORM compliance attributes of the generated derived file are written to the metadata of the derived file through transactional atomic operations.

[0015] The beneficial effects of this invention are: (1) By constructing a directed acyclic graph of the full-link lineage relationship and combining it with transactional atomic operations, the automatic inheritance and synchronization of WORM compliance attributes of derived files are realized, eliminating the window period of compliance attribute configuration and ensuring the continuity of the compliance cycle; (2) Perform full-chain transactional synchronization for changes to the compliance attributes of the original document, and record the full amount of operations in an immutable audit log to ensure the integrity and traceability of the compliance audit chain and adapt to various regulatory compliance requirements; (3) It covers the tracking of derivative files in multi-protocol scenarios and supports the configuration of differentiated inheritance rules based on the same data source range, which greatly reduces the workload of manually configuring the compliance rules of derivative files and reduces the compliance risks caused by human operation. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the steps of an automatic inheritance method for lineage compliance attributes in WORM storage files; Figure 2 The flowchart illustrates the specific steps of a method for automatically inheriting lineage compliance attributes of WORM storage files, as provided in this embodiment. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 See Figure 1 This embodiment provides a method for automatically inheriting the lineage compliance attributes of WORM storage files. The method includes the following steps: S1. Complete the WORM compliance semantic standardization definition, parse the WORM compliance attributes of the file, generate a compliance rule template, bind the compliance rule template with the file's unique identifier and persist it to the metadata; S2. Capture all operations on the file, identify the source file based on the file's unique identifier, identify the behavior of generating derived files, generate a unique lineage identifier for the derived file, and bind the derived file to the parent file's file unique identifier, operation type, operation time, operation subject, and data source range; S3. Using WORM compliance files as root nodes, derived files as child nodes, and lineage relationships as directed edges, construct a full-link lineage relationship directed acyclic graph based on lineage identifiers and the unique file identifiers of parent files, and persist the lineage relationship directed acyclic graph to the WORM compliance metadata pool. S4. Based on the directed acyclic graph of lineage, obtain the unique file identifier of the parent file corresponding to the derived file, match the compliance rule template corresponding to the parent file, generate the WORM compliance attribute of the derived file, and use transactional atomic operations to complete the creation of the derived file and the synchronization of the WORM compliance attribute configuration, and synchronously update the directed acyclic graph of lineage and the WORM compliance audit log.

[0019] In some embodiments, step S1 includes the following sub-steps: S1.1. Decompose WORM attributes into inheritable fields, synchronizeable fields, and verifiable fields to complete the standardized definition of WORM core compliance semantics. The standardized fields include compliance retention period, unmodifiable flag, undeletable lock flag, audit level, compliance level, data retention requirements, storage media lock requirements, and access control rules. S1.2. Based on the WORM compliance semantics of the parsing file, which is based on inheritable fields, synchronizeable fields, and verifiable fields, remove personalized non-compliant fields, generate a compliance rule template, and clearly mark inheritable fields, mandatory inheritance fields, and unchangeable fields in the compliance rule template; S1.3. Bind the compliance rule template to the unique file identifier, persist it to the metadata using an append write method, and simultaneously write it to the WORM compliance audit log.

[0020] In some embodiments, step S2 includes the following sub-steps: S2.1. Embed full-link tracing hooks at all operation entry points of POSIX, object, NFS and CIFS protocols to capture all file operations in real time and record the initiating entity, operation time, operation path and operation data range of the operation; S2.2. Determine the derivative file generation behavior based on operation characteristics. The derivative file generation behavior includes file copying, file exporting, file format conversion, file content extraction, file copy generation, file fragment download and retransmission, and cross-protocol read and write to generate new files; S2.3. Generate a unique lineage identifier for each derived file, bind the parent file's unique file identifier, operation type, operation time, operation subject, and data source range, and output it to the lineage relationship directed acyclic graph construction process.

[0021] In some embodiments, step S3 includes the following sub-steps: S3.1. Using WORM compliant files as the root node, derived files as child and grandchild nodes, and lineage relationships as directed edges, construct a directed acyclic graph of the entire lineage relationship based on the lineage identifier and the unique file identifier of the parent file; S3.2. Record the corresponding file unique identifier, compliance rule template, operation subject, generation time and blood relationship in each node of the directed acyclic graph of bloodline relationship, and support backtracking from any derived file to the root WORM compliance file; S3.3. The directed acyclic graph of lineage is persisted to the WORM compliance metadata pool by appending to the graph, and the WORM compliance audit log is updated simultaneously to record the generation time and the generating entity of the directed acyclic graph of lineage.

[0022] In some embodiments, step S4 includes the following sub-steps: S4.1. When a derived file is generated, obtain the unique file identifier of the parent file corresponding to the derived file based on the directed acyclic graph of lineage, and verify the validity and completeness of the compliance rule template of the parent file; S4.2. Based on the unique file identifier of the parent file, match the corresponding compliance rule template of the parent file, extract the mandatory inheritance field and inheritable field in the compliance rule template, and generate the WORM compliance attribute of the derived file; S4.3. When creating the metadata of the derived file, synchronously write the inherited WORM compliance attributes, first complete the compliance attribute configuration locking, verify the integrity of attribute writing, and then open the data writing permissions of the file; S4.4. Synchronously update the directed acyclic graph of bloodline relationships and the WORM compliance audit log, recording all inherited fields, inheritance time, and operation subject.

[0023] In some embodiments, in step S2, derived files generated across protocols such as POSIX, Object, NFS, and CIFS are included in the full-link lineage tracing scope. The data source scope includes file byte offset, file data block identifier, and file content hash value. For derived files that only use part of the source file data, it is supported to configure differentiated inheritance rules based on the data source scope. For the set regulatory scenarios, it is supported to enable permanent locking of the WORM attribute of the derived file. Permanent locking of the WORM attribute is achieved by writing to the WORM compliance metadata pool and is completed by appending. After the permanent lock takes effect, the WORM compliance attribute of the derived file can only have its retention period extended and other compliance attributes cannot be changed.

[0024] In some embodiments, in step S4, when the compliance attributes of the root WORM file undergo compliance changes, the compliance changes include extending the compliance retention period, upgrading the compliance level, adjusting the audit level, and changing the storage media locking requirements. Based on the lineage relationship, the derived files under the root node are traversed in a directed acyclic graph, and atomic compliance attribute synchronization changes are performed. The change operation adopts a full-link transactional atomic commit, which includes four stages: pre-commit, verification, formal commit, and rollback. If any derived file change fails, all are rolled back. At the same time, an immutable WORM compliance audit log is generated, which records the attributes before the change, the attributes after the change, the entity that initiated the change, and the change time.

[0025] In some embodiments, in step S4, the WORM compliance attributes of files are verified in real time at all file operation entry points of the storage protocol layer, metadata operation interface, and data read / write interface. Real-time verification is triggered when a file is opened, written, its attributes are modified, or deleted. Operations that violate inheritance rules are blocked. Operations that violate inheritance rules include modifying a locked file, deleting a locked file, shortening the compliance retention period, and lowering the compliance level. For derived files that have not completed the inheritance of compliance attributes, an unready flag is set in the metadata, and all data write and modification operations are blocked. The unready flag is cleared and the corresponding permissions are granted only after atomic inheritance is completed.

[0026] In some embodiments, in step S4.2, the WORM compliance attributes of the derived file are generated according to set rules. The mandatory inheritance fields include compliance retention period, unmodifiable flag, undeletable lock flag and compliance level. The mandatory inheritance fields must be fully inherited. The retention deadline of the derived file must not be earlier than the retention deadline of the parent file. The lock flag must be consistent with the parent file. The inheritable fields include audit level, access control rules, storage media lock requirements and data retention requirements. The inheritable fields are fully inherited by default. They can be adjusted upwards based on regulatory requirements, but cannot be adjusted downwards.

[0027] In some embodiments, in step S4.2, when the derived file originates from two or more WORM parent files, the derived file generation scenarios include file merging, data splicing, and multi-source data aggregation. The inheritance rule adopts the principle of choosing the higher value, the retention period is the maximum value of the retention periods of all parent files, the compliance level is the highest value of the compliance levels of all parent files, the locking flag is the most stringent value of the locking flags of all parent files, and the audit level is the highest value of the audit levels of all parent files. The WORM compliance attributes of the generated derived file are written to the metadata of the derived file through transactional atomic operations.

[0028] Example 2 This embodiment provides a specific implementation process for a method of automatically inheriting compliance attributes of file lineage in WORM storage. This method is applicable to compliance archiving scenarios in distributed storage systems. By combining file lineage tracking, compliance semantic parsing, and atomic inheritance of compliance attributes, it achieves automatic transfer and synchronization of compliance attributes of derived files. Figure 2 As shown, the specific implementation process is as follows: Step 1. WORM Compliance Semantic Standardization and Template Generation: Step 1.1. WORM Attribute Decomposition and Standardized Definition: WORM storage is a write-once, read-many storage technology that ensures that written data cannot be modified or deleted within a set time frame. It is widely used in data archiving scenarios that require compliance with regulatory requirements. In this embodiment, WORM attributes are first decomposed into inheritable fields, synchronizeable fields, and verifiable fields to complete the standardized definition of the core compliance semantics of WORM. Standardized fields include compliance retention period, immutable flag, non-deletable lock flag, audit level, compliance level, data retention requirements, storage media lock requirements, and access control rules.

[0029] The system includes the following components: Compliance Retention Period (compare to data retention period, including start and end times); Immutable Flag (identify file status as immutable, blocking any modification); Non-Deleteable Lock Flag (identify file status as non-deletable, blocking any deletion); Audit Level (determine the level of detail for auditing file operations, with different levels corresponding to different audit log entries); Compliance Level (distinguish the stringency of compliance requirements for different files, with higher levels indicating stricter compliance constraints); Data Retention Requirements (specify data retention specifications, including backup frequency, backup storage location, and backup retention time); Storage Media Locking Requirements (specify the locking conditions that storage media must meet, including physical and logical locking); and Access Control Rules (specify different user access permissions for files, including read, write, and execute permissions).

[0030] In some embodiments, standardized fields can be adjusted according to the regulatory requirements of different industries. For example, in the financial industry, compliance fields related to transaction flow can be added, and in the medical industry, compliance fields related to patient privacy protection can be added.

[0031] Step 1.2. Compliance Semantic Parsing and Template Generation: Based on inheritable, synchronizeable, and verifiable fields, the WORM compliance semantics of the file are parsed, personalized non-compliant fields are removed, and a compliance rule template is generated. Personalized non-compliant fields include metadata fields unrelated to WORM compliance requirements, such as file creation time, file modification time, file size, and file owner.

[0032] The compliance rule template should clearly indicate inheritable fields, mandatory inheritance fields, and unchangeable fields. Among them, mandatory inheritance fields are fields that derived files must inherit from the parent file and cannot be modified in any way; inheritable fields are fields that derived files inherit from the parent file by default, but can be adjusted upwards according to regulatory requirements; unchangeable fields are fields that cannot be modified once set, including the file's unique identifier and the compliance rule template version number.

[0033] Step 1.3. Template Binding and Persistence: Bind the compliance rule template to the file unique identifier. The file unique identifier is a string used to uniquely identify a file, which in distributed storage systems typically includes a file inode or an object ID. A file inode is a structure in the POSIX file system used to store file metadata, and each file corresponds to a unique inode number; an object ID is a string used to uniquely identify an object in an object storage system, usually automatically generated by the system.

[0034] The bound information is persisted to the metadata using an append-only write method. Append-only writes allow data to be added to the end of a file without modifying or deleting existing data, ensuring data immutability. Simultaneously, data is written to the WORM compliance audit log, which records all WORM compliance-related operations for subsequent compliance audits.

[0035] In some specific implementations, WORM compliance attribute fields are stored according to a unified encoding format, with each field occupying a fixed length of storage space. The field encoding format and inheritance rules are shown in Table 1. Each field is stored using binary encoding. The immutable flag and the non-deletable lock flag each occupy 1 bit of storage space, the compliance level and the audit level each occupy 4 bits of storage space, and the compliance retention period is stored in timestamp format, occupying 8 bytes of storage space. The compliance rule templates are organized using a structured data format. Each template contains fixed-length header information and variable-length field content. The header information stores the template version number, template length, and number of fields. The field content stores the values ​​of each field in the order shown in Table 1. The generated compliance rule templates are associated with the file's unique identifier. The association relationship is stored in the association index table of the metadata node. The association index table uses a hash index structure, with the file's unique identifier as the key and the storage address of the compliance rule template as the value, enabling fast lookup of compliance rule templates.

[0036] Table 1. WORM Compliance Attribute Field Encoding and Inheritance Rules In some specific implementations, a version management mechanism for compliance rule templates is established. Each generated compliance rule template is assigned a unique version number, which is an incrementing integer. When an update to a compliance rule template is needed, a new version number is generated, and a new compliance rule template is created, while the original version remains unchanged. The binding relationship between files and compliance rule templates is recorded as a unique file identifier corresponding to the template version number. Each file can be bound to multiple versions of compliance rule templates, but only the latest version is active. When a derived file inherits a compliance rule template, it inherits the latest active version from the parent file. When a compliance rule template update needs to be rolled back, the active template version number bound to the file is switched to a historical version number, without modifying the template content itself. Simultaneously, all template version creation, update, and switching operations are recorded in the WORM compliance audit log, including the template version number, operation time, operation subject, and operation type, ensuring the traceability of the template version change process.

[0037] Step 2. File operation tracking and derivative file identification: Step 2.1. Deployment and Operation Capture of End-to-End Tracing Hooks: End-to-end tracing hooks are code snippets embedded in the software execution path that can be triggered when a specific operation is executed, capturing relevant information about that operation. In this embodiment, end-to-end tracing hooks are embedded at all operation entry points of the POSIX, Object, NFS, and CIFS protocols to capture all file operations in real time. The POSIX protocol is a portable operating system interface standard that defines the interface provided by the operating system to applications; the Object protocol is a protocol for accessing object storage, storing and managing data in the form of objects; the NFS and CIFS protocols are two commonly used network file system protocols used to achieve file sharing between different computers.

[0038] When capturing file operations, the system records the initiator, time, path, and data range of the operation. The initiator includes the user ID and process ID; the time uses a system timestamp format, accurate to milliseconds; the path includes the source file path and the target file path; and the data range includes the starting byte offset and the length of the operation in bytes.

[0039] In some embodiments, the end-to-end tracing hook can also be deployed at the client driver layer or storage gateway layer to enable more comprehensive file operation capture.

[0040] Step 2.2. Determining Derivative File Generation Behavior: Determining derivative file generation behavior based on operational characteristics. Derivative file generation behavior refers to the act of generating a new file based on all or part of the data content of the source file. Derivative file generation behavior includes file copying, file exporting, file format conversion, file content extraction, file copy generation, file fragment download and retransmission, and cross-protocol read / write to generate new files.

[0041] When an operation generates a new file's unique identifier based on the full or partial data content of the source file, and the new file shares data origin with the source file, the source file is determined to be the parent file, and the newly generated file is considered a derivative file. Data origin means that all or part of the data in the new file originates from the source file. Determining data origin involves comparing the hash values, data block identifiers, and byte offsets of the data content in the source and new files to confirm the data source of the new file.

[0042] Step 2.3. Lineage Identifier Generation and Association Information Binding: Generate a unique lineage identifier for each derived file. The lineage identifier is a string used to uniquely identify the lineage relationship of a file. Bind the parent file's unique file identifier, operation type, operation time, operation subject, and data source range. The data source range is used to identify which parts of the parent file the derived file's data originates from. Output the bound information to the directed acyclic graph (DAG) construction process for lineage relationships.

[0043] In some specific implementations, a hash concatenation algorithm is used to generate lineage identifiers. This involves concatenating the parent file's unique identifier, the derived file's creation timestamp, and the operation type string. The concatenated string is then hashed to generate a 128-bit hash value as the lineage identifier. To ensure the uniqueness of the lineage identifier, an incrementing sequence number is appended to the hash result. When a hash collision occurs, the sequence number automatically increments until a unique lineage identifier is generated. The generated lineage identifier is then bound to the derived file's unique identifier. This binding relationship is stored in the lineage index table of the metadata node. The lineage index table uses a B+ tree index structure, with the derived file's unique identifier as the key and the lineage identifier and related information as values. The related information is stored in a structured data format, containing five fields: parent file unique identifier, operation type, operation time, operation subject, and data source range. Each field occupies a fixed length of storage space for fast parsing and querying.

[0044] Step 2.4. Cross-Protocol Derivative File Tracking and Differentiated Inheritance Configuration: Derivative files generated across POSIX, Object, NFS, and CIFS protocols are included in the end-to-end lineage tracing scope. The data source range includes file byte offset, file data block identifier, and file content hash value. The file byte offset identifies the starting position and length of data within the file; the file data block identifier identifies the storage data block where the data resides; and the file content hash value is a string calculated from the file content using a hash algorithm, used to verify the integrity of the file content. For derivative files that only use a portion of the source file's data, differentiated inheritance rules can be configured based on the data source range.

[0045] For regulatory scenarios, it supports enabling permanent locking of WORM attributes for derived files. Permanent locking of WORM attributes is achieved by writing to the WORM compliance metadata pool using an append-only write method. After permanent locking takes effect, the WORM compliance attributes of derived files can only have their retention period extended; other compliance attributes cannot be changed.

[0046] In some specific implementations, differentiated judgment rules and data source range extraction methods are adopted for different types of derivative file generation behaviors. The judgment rules for derivative file generation behaviors are shown in Table 2. For file copying and file duplicate generation operations, the hash values ​​of the file content of the source file and the new file are directly compared. If the hash values ​​are the same, it is determined to be a derivative file, and the data source range is the entire data content of the source file. For file format conversion and file export operations, the content digest of the source file and the content digest of the new file are extracted and compared. If the matching degree of the content digest reaches a set threshold, it is determined to be a derivative file, and the data source range is the entire data content of the source file.

[0047] For file content extraction operations, the start and end offsets of the extraction operation are recorded. This offset range is considered the data source range, and the hash value of the data within this range is calculated for subsequent verification of data source consistency. For file fragment download and retransmission operations, the fragment number and data hash value of each fragment are recorded, and the information of all fragments is combined to form the data source range. For cross-protocol read / write operations to generate new files, the source and target file paths of the read / write operation are recorded. The data content of the read / write operation is compared; if data overlap exists, it is determined to be a derived file, and the data source range is the overlapping data interval.

[0048] Table 2. Rules for Determining Derivative File Generation Behavior Step 3. Construction and persistence of the directed acyclic graph of kinship: Step 3.1. Construction of Directed Acyclic Graph (DAG): A DAG is a graph structure composed of vertices and directed edges, in which there is no cycle starting from a vertex, traversing several edges, and returning to the vertex. In this embodiment, a DAG with full-link lineage relationships is constructed based on the lineage identifier and the unique file identifier of the parent file, using the WORM compliant file as the root node, derived files as child and grandchild nodes, and lineage relationships as directed edges.

[0049] Lineage relationships refer to the derivation relationships between parent and derived files, represented by directed edges pointing from the parent file to the derived file. When constructing a directed acyclic graph (DAG), a root node is first created, corresponding to the original WORM compliance file. Then, based on the lineage information, child and grandchild nodes are created sequentially, and directed edges are added between parent and child nodes. Finally, the constructed DAG is checked for cycles. If a cycle exists, the newly added directed edge is deleted, and the anomaly is recorded in the WORM compliance audit log.

[0050] In some embodiments, a directed acyclic graph can be stored using an adjacency list or an adjacency matrix. An adjacency list is a storage method that stores the adjacent vertices of each vertex in a linked list, which is suitable for sparse graphs. An adjacency matrix is ​​a storage method that uses a two-dimensional array to represent the adjacency relationship between vertices, which is suitable for dense graphs.

[0051] Step 3.2. Node Information Recording and Backtracking Support: Record the corresponding file's unique identifier, compliance rule template, operating entity, generation time, and lineage relationship in each node of the directed acyclic graph of lineage relationships. Support backtracking from any derived file to the root WORM compliance file, and also support traversing all derived files from the root WORM compliance file.

[0052] The backtracking function allows for quick location of all source compliance files for a derived file, revealing the origin of its compliance attributes. The traversal function allows for quick retrieval of all derived files from a root compliance file, facilitating batch compliance management. The backtracking operation uses a depth-first search algorithm, starting from the target derived file node and sequentially visiting parent nodes along the reverse direction of the directed edges until the root node is reached. The traversal operation uses a breadth-first search algorithm, starting from the root node and sequentially visiting all child and grandchild nodes along the directed edges.

[0053] Step 3.3. Directed Acyclic Graph Persistence and Audit Log Update: The lineage directed acyclic graph is persisted to the WORM compliance metadata pool using an append-only write method. The WORM compliance metadata pool is a dedicated storage area for storing WORM compliance-related metadata, protected using WORM storage technology to ensure the immutability of the metadata. Simultaneously, the WORM compliance audit log is updated synchronously, recording the generation time and the generating entity of the lineage directed acyclic graph.

[0054] In some implementations, incremental persistence is used to update the kinship-based directed acyclic graph (DAG). Each time a new node or directed edge is added, the new content is only appended to the WORM-compliant metadata pool, without rewriting the entire DAG. The structural information and node data of the kinship-based DAG are stored separately. The structural information stores the connection relationships of directed edges, while the node data stores detailed information for each node. The structural information is stored in adjacency list format, with each node corresponding to an adjacency list entry containing a node identifier and a list of pointers to its child nodes. The node data is stored in key-value pair format, with the node identifier as the key and the node's detailed information as the value. When the kinship-based DAG needs to be read, the structural information is first read to construct the graph's skeleton, and then the corresponding node data is read according to the pointer list to reconstruct the complete DAG. This storage method effectively reduces the amount of data written and improves the efficiency of persistence and retrieval.

[0055] Step 4. Atomic inheritance and synchronization of compliance attributes: Step 4.1. Parent File Identifier Acquisition and Template Verification: When a derived file is generated, the unique file identifier of the parent file corresponding to the derived file is obtained based on the directed acyclic graph of lineage relationships. The validity and completeness of the compliance rule template of the parent file are verified. Validity verification checks whether the compliance rule template is within its validity period and whether it complies with current regulatory requirements. Integrity verification checks whether the compliance rule template has been tampered with and whether it contains all necessary fields. During validity verification, the current system time is compared with the effective time and expiration time in the compliance rule template. If the current time is between the effective time and expiration time, the template is valid. During integrity verification, the hash value of the compliance rule template is calculated and compared with the template hash value stored in the metadata. If the hash values ​​match, the template is complete.

[0056] Step 4.2. Compliance Rule Matching and Attribute Generation: Based on the unique file identifier of the parent file, match the corresponding compliance rule template of the parent file, extract the mandatory inheritance fields and inheritable fields from the compliance rule template, and generate the WORM compliance attributes of the derived file. The generation of WORM compliance attributes of the derived file follows the set rules. Mandatory inheritance fields include compliance retention period, immutable flag, non-deletable lock flag, and compliance level. Mandatory inheritance fields must be fully inherited, the retention deadline of the derived file must not be earlier than the retention deadline of the parent file, and the lock flag must be consistent with the parent file. Inheritable fields include audit level, access control rules, storage media lock requirements, and data retention requirements. Inheritable fields are fully inherited by default and can be adjusted upwards based on regulatory requirements, but cannot be adjusted downwards.

[0057] When a derived file originates from two or more WORM parent files, the derived file generation scenarios include file merging, data splicing, and multi-source data aggregation. The inheritance rule adopts the principle of choosing the higher value over the lower value. The retention period is taken as the maximum value of the retention periods of all parent files, the compliance level is taken as the highest value of the compliance level of all parent files, the lock flag is taken as the most stringent value of the lock flag of all parent files, and the audit level is taken as the highest value of the audit level of all parent files.

[0058] Step 4.3. Atomicity Attribute Configuration and Permission Opening: Transactional atomic operations refer to a series of operations that either all succeed or all fail and roll back to the state before execution; there is no intermediate state where only some operations succeed. In this embodiment, transactional atomic operations are used to synchronize the creation of derived files and the configuration of WORM compliance attributes.

[0059] When creating the metadata of the derived file, the inherited WORM compliance attributes are written synchronously. The compliance attribute configuration is locked first, and the integrity of the attribute write is verified before granting data write permissions to the file. The entire process is a single atomic transaction without any intermediate states, avoiding any window of time when compliance attributes are not configured.

[0060] In some embodiments, a two-phase commit protocol can be used to implement transactional atomic operations. The two-phase commit protocol includes a preparation phase and a commit phase. In the preparation phase, all nodes participating in the transaction need to confirm that the transaction can be executed. In the commit phase, all nodes participating in the transaction execute the transaction simultaneously.

[0061] Step 4.4. Lineage Relationship and Audit Log Update: Synchronously update the lineage relationship directed acyclic graph and WORM compliance audit log, recording all inherited fields, inheritance time, and operation subject. The updated lineage relationship directed acyclic graph accurately reflects the compliance attribute inheritance of derived files, and the updated audit log provides a complete record for subsequent compliance audits. When updating the lineage relationship directed acyclic graph, add inherited compliance attribute information to the nodes corresponding to the derived files, and add attribute inheritance association edges between the parent node and the derived file node; when updating the audit log, record the file unique identifier of the derived file, the file unique identifier of the parent file, the inherited compliance attribute fields, inheritance time, and operation subject.

[0062] In some specific implementations, WORM compliance audit logs are structured and indexed, dividing them into two parts: an operation header and an operation body. The operation header stores fixed-length common information, including operation type, operation time, operation subject, and operation result; the operation body stores variable-length detailed operation information, with different operation types corresponding to different operation body structures. Multi-dimensional indexes are established for the audit logs, including a time index, an operation subject index, a file unique identifier index, and an operation type index. The time index uses a B+ tree structure, with the operation timestamp as the key and the audit log entry storage address as the value; the operation subject index uses a hash structure, with the user identifier as the key and a list of audit log entry addresses for all operations performed by that user as the value; the file unique identifier index uses a hash structure, with the file unique identifier as the key and a list of audit log entry addresses for all related operations performed by that file as the value; the operation type index uses a hash structure, with the operation type identifier as the key and a list of audit log entry addresses for all operations of that type as the value. By establishing multi-dimensional indexes, rapid querying and statistical analysis of audit logs can be achieved, improving the efficiency of compliance audits.

[0063] Step 4.5. End-to-End Synchronization of Compliance Attribute Changes: When a compliance attribute of the root WORM file undergoes a compliance change, such as extending the compliance retention period, upgrading the compliance level, adjusting the audit level, or changing storage media locking requirements, the derived files under the root node are traversed based on the lineage-based directed acyclic graph, and atomic compliance attribute synchronization changes are performed. The change operation adopts end-to-end transactional atomic commit, which includes four phases: pre-commit, verification, formal commit, and rollback.

[0064] During the pre-commit phase, change requests are sent to all derivative files that require modification. During the verification phase, all derivative files verify the legality and feasibility of the change requests. During the formal commit phase, all derivative files simultaneously undergo change operations. During the rollback phase, if any derivative file change fails, all derivative files are rolled back to their pre-change state. Simultaneously, an immutable WORM compliance audit log is generated, recording the attributes before the change, the attributes after the change, the entity initiating the change, and the change time.

[0065] In some specific implementations, each stage of the end-to-end transactional atomic commit executes a clear operational process. The execution content and failure handling methods for each stage are shown in Table 3. In the pre-commit stage, the metadata node sends a change pre-notification to all associated derived file nodes, including the changed fields and their new values. Each derived file node, upon receiving the pre-notification, checks if it is in a modifiable state; if it is in a locked state, it directly returns a change failure. In the verification stage, each derived file node verifies the legality of the change request, checking whether the changed attributes conform to inheritance rules, such as whether the compliance level is higher than the original level and the retention period is longer than the original period. After successful verification, it writes the changed content to the temporary storage area and returns a verification success response to the metadata node. In the formal commit stage, after receiving the verification success responses from all derived file nodes, the metadata node sends a formal commit instruction to all nodes. Each node writes the changed content from the temporary storage area to the formal metadata area and completes attribute locking. In the rollback stage, if any node returns a failure response in the pre-commit or verification stage, the metadata node sends a rollback instruction to all nodes that have received the pre-notification. Each node clears the changed content from the temporary storage area, restoring the state to its pre-change state. The execution status of all stages is recorded in the WORM compliance audit log. The audit log is stored in an append-only manner and the retention period is no less than the longest compliance retention period of the file.

[0066] Table 3 Execution Table for the End-to-End Transactional Atomic Commit Phase In some embodiments, incremental synchronization can be used to update the compliance attributes of derived files, synchronizing only the fields that have changed, thereby reducing the amount of data transmitted and the processing time.

[0067] Step 4.6. Real-time Compliance Attribute Verification and Abnormal Operation Interception: Real-time verification of file WORM compliance attributes is performed at all file operation entry points in the storage protocol layer, metadata operation interface, and data read / write interface. Real-time verification is triggered when a file is opened, written, its attributes are modified, or it is deleted. Operations that violate inheritance rules are intercepted. These violations include modifying locked files, deleting locked files, shortening the compliance retention period, and lowering the compliance level. For derived files that have not completed compliance attribute inheritance, an incomplete flag is set in the metadata, and all data write and modification operations are intercepted. The incomplete flag is cleared and the corresponding permissions are granted only after atomic inheritance is completed. For intercepted abnormal operations, alarms are triggered in real-time and recorded in the WORM compliance audit log.

[0068] In some specific implementations, a dynamic update mechanism for compliance verification rules is established. These rules are stored in an independent rule base. Each rule in the rule base consists of three parts: a rule identifier, rule conditions, and rule actions. Rule conditions describe the operation type and file attribute status that triggers the verification; rule actions describe the actions to be performed when the verification passes or fails, including allowing, blocking, and triggering alarms. When regulatory requirements change, the compliance verification logic is dynamically adjusted by updating the rules in the rule base without modifying the system code. The rule base is protected using WORM storage technology. All rule creation, update, and deletion operations are recorded in the WORM compliance audit log, ensuring the traceability of rule changes. The rule base is also backed up regularly, with backup data stored on an independent WORM storage medium to prevent data loss.

[0069] This embodiment constructs a directed acyclic graph of file lineage relationships across the entire file hierarchy, enabling automatic inheritance of WORM compliance attributes based on file lineage. This effectively reduces the possibility of missing compliance attributes in derived files, ensuring the continuity of the compliance cycle. Transactional atomic operations are used for configuring and changing compliance attributes, guaranteeing the integrity of compliance attribute operations and avoiding compliance gaps caused by partial changes. The fully automated compliance attribute management process reduces the workload of manual configuration and mitigates compliance risks caused by human error. Cross-protocol lineage tracing and compliance attribute inheritance capabilities adapt to the needs of multi-protocol storage environments, covering more file-derived scenarios. The end-to-end immutable audit log provides a complete record for compliance audits, helping to meet the regulatory compliance requirements of different industries. Furthermore, the differentiated inheritance rules and permanent locking function provided in this embodiment can adapt to the needs of different regulatory scenarios, offering users a more flexible compliance management approach. The version management mechanism for compliance rule templates and the dynamic update mechanism for compliance verification rules enable the system to quickly adapt to changes in regulatory requirements, improving system maintainability and scalability. The incremental persistence method of directed acyclic graphs based on bloodline relationships and the multi-dimensional index optimization of audit logs improve the system's operating efficiency and query performance, enabling it to support the compliance management needs of large-scale distributed storage systems.

[0070] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A WORM storage file bloodline compliance attribute automatic inheritance method, characterized in that, Includes the following steps: S1. Complete the WORM compliance semantic standardization definition, parse the WORM compliance attributes of the file, generate a compliance rule template, bind the compliance rule template with the file's unique identifier and persist it to the metadata; S2. Capture all operations on the file, identify the source file based on the file's unique identifier, identify the behavior of generating derived files, generate a unique lineage identifier for the derived file, and bind the derived file to the parent file's file unique identifier, operation type, operation time, operation subject, and data source range; S3. Using WORM compliance files as root nodes, derived files as child nodes, and lineage relationships as directed edges, construct a full-link lineage relationship directed acyclic graph based on lineage identifiers and the unique file identifiers of parent files, and persist the lineage relationship directed acyclic graph to the WORM compliance metadata pool. S4. Based on the directed acyclic graph of lineage, obtain the unique file identifier of the parent file corresponding to the derived file, match the compliance rule template corresponding to the parent file, generate the WORM compliance attribute of the derived file, and use transactional atomic operations to complete the creation of the derived file and the synchronization of the WORM compliance attribute configuration, and synchronously update the directed acyclic graph of lineage and the WORM compliance audit log.

2. The method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S1.

1. Decompose WORM attributes into inheritable fields, synchronizeable fields, and verifiable fields to complete the standardized definition of WORM core compliance semantics. The standardized fields include compliance retention period, unmodifiable flag, undeletable lock flag, audit level, compliance level, data retention requirements, storage media lock requirements, and access control rules. S1.

2. Based on the WORM compliance semantics of the parsing file, which is based on inheritable fields, synchronizeable fields, and verifiable fields, remove personalized non-compliant fields, generate a compliance rule template, and clearly mark inheritable fields, mandatory inheritance fields, and unchangeable fields in the compliance rule template; S1.

3. Bind the compliance rule template to the unique file identifier, persist it to the metadata using an append write method, and simultaneously write it to the WORM compliance audit log.

3. The method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S2.

1. Embed full-link tracing hooks at all operation entry points of POSIX, object, NFS and CIFS protocols to capture all file operations in real time and record the initiating entity, operation time, operation path and operation data range of the operation; S2.

2. Determine the derivative file generation behavior based on operation characteristics. The derivative file generation behavior includes file copying, file exporting, file format conversion, file content extraction, file copy generation, file fragment download and retransmission, and cross-protocol read and write to generate new files; S2.

3. Generate a unique lineage identifier for each derived file, bind the parent file's unique file identifier, operation type, operation time, operation subject, and data source range, and output it to the lineage relationship directed acyclic graph construction process.

4. The method according to claim 1, characterized in that, Step S3 includes the following sub-steps: S3.

1. Using WORM compliant files as the root node, derived files as child and grandchild nodes, and lineage relationships as directed edges, construct a directed acyclic graph of the entire lineage relationship based on the lineage identifier and the unique file identifier of the parent file; S3.

2. Record the corresponding file unique identifier, compliance rule template, operation subject, generation time and blood relationship in each node of the directed acyclic graph of bloodline relationship, and support backtracking from any derived file to the root WORM compliance file; S3.

3. The directed acyclic graph of lineage is persisted to the WORM compliance metadata pool by appending to the graph, and the WORM compliance audit log is updated simultaneously to record the generation time and the generating entity of the directed acyclic graph of lineage.

5. The method according to claim 1, characterized in that, Step S4 includes the following sub-steps: S4.

1. When a derived file is generated, obtain the unique file identifier of the parent file corresponding to the derived file based on the directed acyclic graph of lineage, and verify the validity and completeness of the compliance rule template of the parent file; S4.

2. Based on the unique file identifier of the parent file, match the corresponding compliance rule template of the parent file, extract the mandatory inheritance field and inheritable field in the compliance rule template, and generate the WORM compliance attribute of the derived file; S4.

3. When creating the metadata of the derived file, synchronously write the inherited WORM compliance attributes, first complete the compliance attribute configuration locking, verify the integrity of attribute writing, and then open the data writing permissions of the file; S4.

4. Synchronously update the directed acyclic graph of bloodline relationships and the WORM compliance audit log, recording all inherited fields, inheritance time, and operation subject.

6. The method according to claim 1, characterized in that, In step S2, derivative files generated across protocols such as POSIX, Object, NFS, and CIFS are included in the full-link lineage tracing scope. The data source scope includes file byte offset, file data block identifier, and file content hash value. For derivative files that only use part of the source file's data, it is supported to configure differentiated inheritance rules based on the data source scope. For the set regulatory scenarios, it is supported to enable permanent locking of the WORM attribute of the derivative file. Permanent locking of the WORM attribute is achieved by writing to the WORM compliance metadata pool and is completed by appending. After the permanent lock takes effect, the WORM compliance attribute of the derivative file can only have its retention period extended and other compliance attributes cannot be changed.

7. The method according to claim 1, characterized in that, In step S4, when the compliance attributes of the root WORM file undergo compliance changes, these changes include extending the compliance retention period, upgrading the compliance level, adjusting the audit level, and changing storage media locking requirements. Based on the lineage relationship, the derived files under the root node are traversed in a directed acyclic graph, and atomic compliance attribute synchronization changes are performed. The change operation adopts a full-chain transactional atomic commit, which includes four stages: pre-commit, verification, formal commit, and rollback. If any derived file change fails, all changes are rolled back. At the same time, an immutable WORM compliance audit log is generated, which records the attributes before the change, the attributes after the change, the entity that initiated the change, and the change time.

8. The method according to claim 1, characterized in that, In step S4, the WORM compliance attributes of files are verified in real time at all file operation entry points of the storage protocol layer, metadata operation interface, and data read / write interface. Real-time verification is triggered when a file is opened, written, its attributes are modified, or deleted. Operations that violate inheritance rules are blocked. Operations that violate inheritance rules include modifying locked files, deleting locked files, shortening the compliance retention period, and lowering the compliance level. For derived files that have not completed the inheritance of compliance attributes, an unready flag is set in the metadata, and all data write and modification operations are blocked. The unready flag is cleared and the corresponding permissions are granted only after atomic inheritance is completed.

9. The method according to claim 5, characterized in that, In step S4.2, the WORM compliance attributes of the derived file are generated according to the set rules. The mandatory inheritance fields include compliance retention period, unmodifiable flag, undeletable lock flag and compliance level. The mandatory inheritance fields must be fully inherited. The retention deadline of the derived file must not be earlier than the retention deadline of the parent file. The lock flag must be consistent with the parent file. The inheritable fields include audit level, access control rules, storage media lock requirements and data retention requirements. The inheritable fields are fully inherited by default. They can be adjusted upwards based on regulatory requirements, but cannot be adjusted downwards.

10. The method according to claim 5, characterized in that, In step S4.2, when a derived file originates from two or more WORM parent files, the derived file generation scenarios include file merging, data splicing, and multi-source data aggregation. The inheritance rule adopts the principle of choosing the higher value over the lower value. The retention period is taken as the maximum value of the retention periods of all parent files, the compliance level is taken as the highest value of the compliance level of all parent files, the locking flag is taken as the most stringent value of the locking flag of all parent files, and the audit level is taken as the highest value of the audit level of all parent files. The WORM compliance attributes of the generated derived file are written to the metadata of the derived file through transactional atomic operations.

Images