A method and system for separating physical storage and logical organization of mobile files based on content fingerprinting

Abstract

This invention discloses a method and system for separating the physical storage and logical organization of mobile files based on content fingerprinting. The method includes: generating a globally unique content fingerprint; constructing a hierarchical hash physical path based on the fingerprint's positional characters, and generating a globally unique identifier as the physical filename to store the physical entity, thus separating the physical layer from business information; establishing an association mapping between business metadata, content fingerprint, and the globally unique identifier, and storing it in a local database. This invention constructs a virtual organizational structure through database relationships, allowing file classification, movement, and renaming to only require modifying database records without physically moving the files. This invention has a significant deduplication effect, effectively reduces mobile flash memory I / O consumption, supports multi-dimensional classification of single files, and greatly improves management flexibility.

Description

Technical Field

[0001] This invention relates to the field of mobile terminal data processing technology, specifically to a method, system, and storage medium for separating the physical storage and logical organization of mobile terminal files based on content fingerprinting. Background Technology

[0002] Currently, file management systems on mobile devices (such as smartphones and tablets) primarily use the traditional tree-structured physical folder architecture. In this architecture, the physical storage path of a file is completely bound to its logical category; that is, a file is uniquely identified by its directory path and filename.

[0003] However, the traditional architecture described above has the following drawbacks in mobile applications: 1. Poor flexibility: A single file entity can only belong to one physical directory, making it impossible to achieve cross-category affiliation without increasing storage consumption. 2. Severe redundancy: It cannot perform full-domain deduplication based on content, resulting in a large accumulation of files with the same name but different characteristics, or vice versa, wasting limited flash memory space. 3. High IO overhead: Moving, classifying, or archiving files requires moving physical file entities, generating a large number of flash memory read and write operations, accelerating flash memory aging and easily causing storage fragmentation. 4. Low retrieval efficiency: Retrieval operations usually rely on traversing the directory tree, making it impossible to achieve efficient full-text search or multi-dimensional filtering when the file base is large. 5. Insufficient architectural adaptability: Although existing cloud object storage can achieve decoupling, its communication overhead and computational complexity are high, making it difficult to directly apply to resource-constrained mobile terminal standalone environments. Summary of the Invention

[0004] This invention provides a method and system for separating physical storage and logical organization of mobile files based on content fingerprinting, aiming to solve the problems of insufficient flexibility and storage performance loss caused by the strong binding of physical and logical elements in traditional mobile file management.

[0005] 1. Technical Solution A method for separating physical storage and logical organization of mobile files based on content fingerprints includes the following steps: 1. Content digital fingerprint generation: Obtaining file feature information of the file to be stored, selecting a corresponding content extraction strategy based on the file feature information, performing digest calculation on the file to be stored, and generating a globally unique content digital fingerprint; 2. Physical entity storage: Based on the content digital fingerprint, constructing a hierarchical hash physical storage path according to a preset hash mapping rule, and storing the physical entity of the file to be stored in the hierarchical hash physical storage path; 3. Metadata stripping and storage: Stripping the business metadata of the file to be stored, establishing an association mapping between the business metadata and the content digital fingerprint, and storing it in a local embedded database; 4. Logical organization construction: Establishing a virtual organizational structure in the local embedded database, the virtual organizational structure pointing to the corresponding physical entity through the content digital fingerprint, and managing the logical ownership of files by modifying the corresponding association records in the database.

[0006] Preferably, the content extraction strategy includes: 1. If the file size is lower than a preset threshold, then perform a summary calculation on the entire file content; 2. If the file size is not lower than the preset threshold, then extract multiple data segments from the file according to a preset sampling rule, and perform a combined summary calculation on the multiple data segments.

[0007] Preferably, the business metadata includes: file name, format suffix, creation time, modification time, remarks, feature extraction results of multimedia content, and user-defined tag information.

[0008] Preferably, in the logical organization construction step, multiple logical paths are supported to be associated with the same physical entity through the same content digital fingerprint.

[0009] A mobile file physical storage and logical organization separation management system based on content fingerprinting includes: 1. A fingerprint generation module: used to generate content digital fingerprints based on file feature information using a differentiated sampling strategy; 2. A physical storage module: used to construct a hierarchical hash storage structure decoupled from logical paths, and to perform file deduplication and persistent storage; 3. A metadata management module: used to manage file attributes and multi-dimensional tags stored in an embedded database; 4. A logical scheduling module: used to handle business logic related to virtual folders, coexistence of files with the same name, and zero-IO data transfer.

[0010] 2. Beneficial effects

[0011] 1) Lossless deduplication: By identifying the uniqueness of files through content fingerprinting, it effectively saves mobile flash memory space.

[0012] 2) Zero I / O management: File reclassification, moving and other operations only require modification of database records, without physical read and write operations, thus protecting the lifespan of flash memory.

[0013] 3) Multi-dimensional attribution: Enables parallel management of multiple classifications and tags for single files, breaking through the limitations of traditional tree structures.

[0014] 4) High-efficiency retrieval: Relying on database indexes, it enables millisecond-level full-text and attribute fuzzy retrieval. Attached Figure Description

[0015] Figure 1 The overall flowchart of the method provided by this invention; Figure 2 This is a schematic diagram of the decoupled architecture between the physical storage layer and the logical organization layer in this invention; Figure 3 This is a schematic diagram of the logic for differential fingerprint extraction in this invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Example 1: Differentiated Content Fingerprint Generation and Deduplication Storage In this embodiment, the mobile terminal receives a new file. As shown in Figure 3, the system executes the following fingerprint calculation logic: reads the file size and determines whether the file size is greater than 128KB. If the file size is not greater than 128KB, the SHA-256 algorithm is used to calculate a digest of the entire file content; if the file size is greater than 128KB, the first 64KB and last 64KB data segments of the file are read, sequentially concatenated, and the digest value is calculated as the globally unique digital fingerprint of the file.

[0018] As shown in Figure 1, after generating the fingerprint, the system performs conflict verification: it queries the local embedded database. If the same digital fingerprint already exists in the database, the file is considered duplicated. The system skips the physical write operation and only performs the reference count increment logic, adding a new logical association record to the database.

[0019] If the fingerprint does not conflict, it is determined to be a new file. The system performs path mapping: extracting the first character of the fingerprint string as the first-level directory name ( / e / as shown in Figure 2), and extracting the second character as the second-level directory name ( / 3 / as shown in Figure 2). Simultaneously, the system generates a globally unique identifier (UUID), names the file entity with this UUID (without retaining the original suffix), and stores it in the aforementioned second-level directory. This process achieves complete separation of physical storage from the original business path and filename.

[0020] Example 2: Logically Organized "Zero I / O" Movement and Multiple Attribution Suppose a user needs to categorize a video file "scheme.pdf" under the virtual folder " / My Documents / " into the folder " / Project Backup / ".

[0021] As shown in Figure 2, this invention does not move the physical file entity. The system only adds a new record (record 2) in the metadata association layer of the local embedded database, pointing the virtual path " / project backup / old version.pdf" to the content fingerprint and corresponding UUID physical entity of that file.

[0022] Technical effect: Users can see and access the file in both virtual folders, but only one copy occupies physical space. When a user deletes one of the logical associations, the system only deletes the corresponding metadata record in the database and reduces the reference count; only when the reference count is cleared (i.e., all logical associations are deleted) does the system destroy the UUID entity in the physical layer based on the fingerprint, thus significantly reducing the frequency of flash memory erase and write operations.

Claims

1. A method for separating the physical storage and logical organization of mobile files based on content fingerprinting, characterized in that, Includes the following steps: Step S1: Obtain the file characteristics of the file to be stored, select the corresponding content digest strategy based on the file characteristics, and generate a globally unique content digital fingerprint; Step S2: Construct a hierarchical hash physical storage path based on the position characters of the content digital fingerprint, and generate a globally unique identifier as the physical file name to store the physical entity, thereby achieving separation of the physical layer and business information; Step S3: Extract the business metadata of the file to be stored, establish an association mapping between the business metadata, the content digital fingerprint and the globally unique identifier, and store them in the local database; Step S4: By maintaining the association between files and virtual logical groups in the local database, the physical entity storage and logical organization structure are decoupled.

2. The method according to claim 1, characterized in that, The content summarization strategy mentioned in step S1 includes: Choose between full content summary calculation mode or partial sample summary calculation mode based on the size of the file to be stored.

3. The method according to claim 1, characterized in that, The physical storage path mentioned in step S2 is a hash-hierarchical path generated based on the content digital fingerprint, and the physical storage path does not contain the original path information in the business metadata.

4. The method according to claim 1, characterized in that, The business metadata mentioned in step S3 includes at least the file name, file extension, tag information, and text feature information extracted by the content recognition algorithm.

5. The method according to claim 1, characterized in that, The association described in step S4 supports a single content digital fingerprint corresponding to multiple different virtual logical groups.

6. The method according to claim 1, characterized in that, When performing logical moves, renaming, or reclassification operations on files, only the business metadata or relationships in the local database are updated, without changing the physical entity storage location.

7. A mobile file physical storage and logical organization separation management system based on content fingerprinting, characterized in that, include: 1) Fingerprint generation module, used to generate a digital fingerprint that reflects the uniqueness of the file content; 2) Physical storage module, used to construct a hierarchical hash storage path based on the content digital fingerprint, and to perform persistent storage of entities using a globally unique identifier; 3) Database management module, used to store the mapping relationship between the stripped metadata, fingerprints, and globally unique identifiers; 4) Logical interaction module, used to perform virtual organization and management of files without touching physical entities, based on the mapping relationship provided by the database management module.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images