Redundant file management methods, devices, electronic equipment and storage media

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for redundant file management. The method involves: acquiring a list of redundant files to be processed; converting redundant files stored in a first storage node from a first storage state to a second storage state; deleting redundant files from the first storage node if no access requests are detected within a first preset monitoring period; after deleting redundant files from the first storage node, converting redundant files stored in a second storage node from the first storage state to the second storage state; and deleting redundant files from the second storage node if no access requests are detected within a second preset monitoring period. This achieves automated and accurate identification of redundant files, while ensuring service availability through another storage node during operations on either storage node. It frees up storage space while achieving highly reliable decoupling between deletion operations and service quality, thus improving the efficiency of redundant file management.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for redundant file management. Background Technology

[0002] In existing file storage systems, a large number of redundant files are generated due to continuous operations such as program recoding and version updates. Traditional methods of managing redundant files mainly rely on manual deletion or batch cleanup.

[0003] Existing methods for handling redundant files heavily rely on manual labor, resulting in low efficiency and extremely high costs. Furthermore, they are prone to accidental deletion, which is irreversible once deleted, causing online playback failures and impacting user experience and business continuity. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for redundant file management, in order to solve the technical problem that existing methods of handling redundant files rely on manual labor, cannot accurately identify and process redundant files, and result in low efficiency and high cost in redundant file management.

[0005] Firstly, this application provides a method for managing redundant files, the method comprising: Obtain a list of redundant files to be processed; perform the following processing on the redundant files in the list of redundant files: The redundant file stored in the first storage node is converted from a first storage state to a second storage state, wherein the second storage state prevents the redundant file from being directly accessed. Access requests to the redundant file are monitored during a first preset monitoring period. If an access request to the redundant file is detected during the first preset monitoring period, a service response is provided based on the redundant file stored in the second storage node, and the redundant file is converted from the second storage state to the first storage state. If no access request to the redundant file is detected during the first preset monitoring period, the redundant file is deleted from the first storage node. After deleting the redundant file from the first storage node, the redundant file stored in the second storage node is changed from a first storage state to a second storage state; and during a second preset monitoring period, access requests to the redundant file are monitored; if no access requests to the redundant file are monitored during the second preset monitoring period, the redundant file is deleted from the second storage node.

[0006] In one possible implementation, obtaining the list of redundant files to be processed includes: Based on the metadata that records the mapping relationship between the first identifier and the second identifier of the referenced data file, update the reverse mapping relationship from the second identifier to the first identifier of the data file; The data files corresponding to the second identifier that are not associated with the first identifier in the updated reverse mapping relationship are identified as redundant files, thus obtaining the list of redundant files.

[0007] In one possible implementation, the reverse mapping relationship from the second identifier to the first identifier of the updated data file includes: A reconstruction operation is periodically performed on the reverse mapping relationship, wherein the reconstruction operation includes: Clear the first identifier from the existing mapping records in the reverse mapping relationship; Based on the current metadata information, a new mapping record from the second identifier to the first identifier is established in the reverse mapping relationship.

[0008] In one possible implementation, prior to transitioning the redundant files stored in the first storage node from the first storage state to the second storage state, the process includes: The redundant files are marked as an observation state, and the start time of the observation state is recorded; If a first preset time period elapses from the start time and no access request to the redundant file is detected during this period, then the step of converting the redundant file stored in the first storage node from the first storage state to the second storage state is triggered.

[0009] In one possible implementation, the method further includes: If an access request to the redundant file is detected before the duration of the observed state reaches the first preset duration, the redundant file is removed from the redundant file list.

[0010] In one possible implementation, the method further includes: If an access request for the redundant file is detected during the second preset monitoring period, the redundant file in the second storage node will be changed from the second storage state to the first storage state.

[0011] In one possible implementation, monitoring access requests to the redundant files includes: Continuously monitor access events in the file access log and convert redundant file identifiers associated with the access events into corresponding target binary values ​​according to preset rules; A Bloom filter is found based on the target binary value; wherein the Bloom filter contains a list of binary values ​​corresponding to the list of redundant files calculated based on the preset rules; If the target binary value is found in the Bloom filter, the redundant file identifier is further searched in the redundant file list. If the redundant file identifier is found, it is determined that an access request to the redundant file has been monitored. If the target binary value is not found in the Bloom filter, it is determined that no access request to the redundant file was detected.

[0012] Secondly, this application provides a redundant file management device, the device comprising: A redundant file list acquisition module is used to acquire a list of redundant files to be processed; and to perform the following processing on the redundant files in the redundant file list: A first storage node processing module is configured to convert the redundant files stored in the first storage node from a first storage state to a second storage state, wherein the second storage state prevents the redundant files from being directly accessed, and monitor access requests to the redundant files during a first preset monitoring period; wherein, if an access request to the redundant files is detected during the first preset monitoring period, a service response is provided based on the redundant files stored in the second storage node, and the redundant files are converted from the second storage state to the first storage state; if no access request to the redundant files is detected during the first preset monitoring period, the redundant files are deleted from the first storage node; The second storage node processing module is configured to, after deleting the redundant file from the first storage node, change the redundant file stored in the second storage node from a first storage state to a second storage state; and monitor access requests to the redundant file during a second preset monitoring period; if no access requests to the redundant file are monitored during the second preset monitoring period, then delete the redundant file from the second storage node.

[0013] In one possible implementation, the redundant file list acquisition module includes: The reverse mapping relationship update unit is used to update the reverse mapping relationship from the second identifier to the first identifier of the data file based on the metadata that records the mapping relationship between the first identifier and the second identifier of the data file it references; The redundant file determination unit is used to determine the data files corresponding to the second identifier that are not associated with the first identifier in the updated reverse mapping relationship as redundant files, thereby obtaining the redundant file list.

[0014] In one possible implementation, the reverse mapping update unit is specifically used for: A reconstruction operation is periodically performed on the reverse mapping relationship, wherein the reconstruction operation includes: Clear the first identifier from the existing mapping records in the reverse mapping relationship; Based on the current metadata information, a new mapping record from the second identifier to the first identifier is established in the reverse mapping relationship.

[0015] In one possible implementation, prior to the first storage node processing module, the following is included: The file marking unit is used to mark the redundant file as an observation state and record the start time of the observation state; The storage state transition unit is configured to trigger the step of transitioning the redundant file stored in the first storage node from the first storage state to the second storage state if a first preset time period has elapsed since the start time and no access request to the redundant file has been detected during the period.

[0016] In one possible implementation, prior to the first storage node processing module, the following further includes: The access request monitoring unit is configured to remove the redundant file from the redundant file list if an access request for the redundant file is detected before the duration of the observed state reaches the first preset duration.

[0017] In one possible implementation, the device further includes: The storage state transition module is used to transition the redundant file in the second storage node from a second storage state to a first storage state if an access request for the redundant file is detected during the second preset monitoring period.

[0018] In one possible implementation, the first storage node processing module monitors access requests to the redundant files in the following manner: Continuously monitor access events in the file access log and convert redundant file identifiers associated with the access events into corresponding target binary values ​​according to preset rules; A Bloom filter is found based on the target binary value; wherein the Bloom filter contains a list of binary values ​​corresponding to the list of redundant files calculated based on the preset rules; If the target binary value is found in the Bloom filter, the redundant file identifier is further searched in the redundant file list. If the redundant file identifier is found, it is determined that an access request to the redundant file has been monitored. If the target binary value is not found in the Bloom filter, it is determined that no access request to the redundant file was detected.

[0019] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a redundant file management program stored in the memory to implement the redundant file management method described in any one of the first aspects.

[0020] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the redundant file management method described in the first aspect.

[0021] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application obtains a list of redundant files to be processed; and performs the following processing on the redundant files in the list: converting the redundant files stored in the first storage node from a first storage state to a second storage state, and monitoring access requests to the redundant files during a first preset monitoring period; if no access requests to the redundant files are monitored during the first preset monitoring period, the redundant files are deleted from the first storage node; after deleting the redundant files from the first storage node, converting the redundant files stored in the second storage node from the first storage state to the second storage state; and monitoring access requests to the redundant files during a second preset monitoring period; if no access requests to the redundant files are monitored during the second preset monitoring period, the redundant files are deleted from the second storage node. This achieves automated and accurate identification of redundant files, completely replacing inefficient and error-prone manual auditing, and greatly improving processing efficiency and accuracy. Furthermore, during operation on any storage node, service availability can be guaranteed by another storage node, thereby freeing up storage space while achieving highly reliable decoupling between deletion operations and service quality, reducing the risk of accidental deletion to near zero; thus improving the efficiency of redundant file management. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0025] Figure 1 A flowchart illustrating an embodiment of a redundant file management method provided in this application; Figure 2 A flowchart illustrating an embodiment of another redundant file management method provided in this application; Figure 3 A flowchart illustrating another embodiment of a redundant file management method provided in this application; Figure 4 A structural block diagram of a redundant file management device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0028] To address the technical problems of existing redundant file processing methods that rely on manual intervention, making it difficult to accurately identify and process redundant files, resulting in low efficiency and high costs in redundant file management, this application provides a redundant file management method, apparatus, electronic device, and storage medium. This method enables automated and accurate identification of redundant files, completely replacing inefficient and error-prone manual auditing, and significantly improving processing efficiency and accuracy. Furthermore, during operation on any storage node, service availability can be guaranteed by another storage node. This frees up storage space while achieving highly reliable decoupling between deletion operations and service quality, reducing the risk of accidental deletion to near zero; thus improving the efficiency of redundant file management.

[0029] Figure 1 A flowchart illustrating an embodiment of a redundant file management method provided in this application includes the following steps: Step 101: Obtain the list of redundant files to be processed.

[0030] A redundant file list can refer to a list containing file identifiers (such as file address identifiers URIs) of all redundant files that the system determines may no longer be needed. These redundant files are usually old files replaced by new versions of files generated due to business changes (such as video re-encoding, content updates).

[0031] In one embodiment, all in-use file address identifiers (URIs) in the full index database are periodically synchronized to the reverse index database. Then, the historical data in the reverse index database is compared to identify file address identifiers that existed previously but were not synchronized this time. These file address identifiers are the old files that have expired. These old files are then compiled into a list of redundant files.

[0032] A full index database allows you to search for files by program. Knowing a program's unique identifier (such as a program ID), you can find all the video file addresses (file identifiers) currently used by that program, along with information such as the program name and director. A reverse index database allows you to search for programs by file. Knowing a video file's address, you can find out which program that file belongs to.

[0033] For example, the metadata corresponding to program video data is generally stored in an index database. This index database contains the program video name identifier index and the relevant attribute information or metadata information of the program, such as: the program's video file address identifier (URI), director, actors, program synopsis, etc.

[0034] Furthermore, based on the aforementioned full index database, a new reverse index database can be created, indexed by file address identifiers (URIs), storing program video name identifiers mapped to those URIs, as well as other metadata information for those programs. Periodically, such as daily, the reverse index database can be emptied, with an old snapshot created before emptying. Then, the metadata information from the current full index database is populated into the reverse index database, and a new snapshot is created again. By comparing the old and new snapshots, it can be determined which file address identifiers are absent in the new snapshot but exist in the old snapshot. This indicates that the file address identifier has been discarded, and no program needs to reference the file address corresponding to that file address identifier; therefore, the file address identifier is considered redundant. This is merely an example using snapshots; this application embodiment does not limit this approach, and redundant files can also be determined based on the current program metadata information using other methods. Specific details will be described in the relevant embodiments below.

[0035] For example, suppose the first episode of the TV series "Science and Technology" initially had files in both 360p and 720p bitrates, with corresponding file address URIs of file_360_v1 and file_720_v1. Later, the platform performed a quality enhancement and re-encoding of the first episode, generating new version file addresses file_360_v2 and file_720_v2. The metadata index was updated accordingly, and now the program video name identifier for the first episode of "Science and Technology" only associates with the latest version of the file. The next time the system retrieves the list of redundant files, it will find that the file address identifiers file_360_v1 and file_720_v1 are no longer referenced in any program's reference list, so they will be added to the list of redundant files to be deleted.

[0036] Step 102: Perform the following processing on the redundant files in the redundant file list: Change the redundant files stored in the first storage node from the first storage state to the second storage state, wherein the second storage state makes the redundant files inaccessible directly, and monitor access requests to the redundant files during the first preset monitoring period; wherein, if an access request to the redundant files is detected during the first preset monitoring period, provide a service response based on the redundant files stored in the second storage node, and change the redundant files from the second storage state to the first storage state; if no access request to the redundant files is detected during the first preset monitoring period, delete the redundant files from the first storage node.

[0037] The first and second storage nodes can refer to two object storage services that serve as backups for each other, used to store specific data files. Each data file has a unique file address identifier, such as the file address URI mentioned in the previous steps. These file address identifiers are stored in the aforementioned index database. Furthermore, the first and second storage nodes store identical data file content to ensure high availability and prevent single points of failure. This application embodiment only uses two storage nodes as an example for explanation; in addition, there can be three or more storage nodes.

[0038] The first storage state can refer to standard storage (or hot storage). Files in the first storage state can be accessed instantly in milliseconds, but the storage cost is high.

[0039] The second storage state can refer to archive storage or cold storage, which has extremely low storage costs. Files in this storage state are essentially frozen, and they need to be unfrozen before they can be accessed (which may take several hours). They cannot be read directly before they are unfrozen.

[0040] The first preset monitoring period can refer to the duration (e.g., 6 months) during which the system monitors access to a file after it has been converted to an archive state, that is, after the file is in the second storage state.

[0041] In one embodiment, access requests to redundant files can be monitored in the following way: continuously detecting access events in the file access log, converting the redundant file identifier associated with the access event into a corresponding target binary value according to a preset rule; searching a Bloom filter based on the target binary value; wherein the Bloom filter contains a list of binary values ​​corresponding to the list of redundant files calculated based on the preset rule; if the target binary value is found in the Bloom filter, then the redundant file identifier is further searched from the list of redundant files; if the redundant file identifier is found, it is determined that an access request to a redundant file has been monitored; if the target binary value is not found in the Bloom filter, it is determined that no access request to a redundant file has been monitored.

[0042] A Bloom filter can be considered a probabilistic data structure used to quickly determine whether an element is definitely not in a set. It uses very little memory and can improve search speed.

[0043] For example, for a redundant file in the redundancy list, if certain conditions are met, such as no access requests for the redundant file within a certain time period, the system will change the storage type of the redundant file in the first storage node from standard storage to archive storage. After the redundant file is archived, any request to directly access the redundant file will immediately return a 404 error; however, the system can respond to user requests using a backup file of the redundant file stored in the second storage node, although the response time will be slightly longer.

[0044] Furthermore, after changing the storage type of the redundant file from standard storage to archive storage, during the first preset monitoring period, access requests to the redundant file are monitored based on a Bloom filter. This allows for rapid detection of whether the redundant file has been accessed during the first preset monitoring period. If access is detected, the redundant file is switched from the second storage state (archive storage type) to the first storage state (standard storage type). If no access is detected, the redundant file is directly deleted from the first storage node.

[0045] For example, suppose the first preset monitoring period is 6 months, the first storage node is storage A, and the second storage node is storage B. The system archives the redundant file corresponding to the file identifier "file_360_v1" in storage A. During the next 6 months of monitoring, users do not access this redundant file for the vast majority of the time. However, one day, an app version using an old cached link suddenly requests this redundant file. The system instantly captures this access request through Bloom filters and log analysis, and confirms that the file is in the redundant list. The system immediately provides the redundant file stream from storage B to the user, and simultaneously initiates the unfreezing and restoration of the redundant file in storage A, changing the storage type of the redundant file in the first storage node from archived. If no access requests to the redundant file are detected within 6 months, the redundant file is directly deleted from the first storage node.

[0046] Step 103: After deleting the redundant file from the first storage node, the redundant file stored in the second storage node is changed from the first storage state to the second storage state; and during the second preset monitoring period, access requests to the redundant file are monitored; if no access requests to the redundant file are monitored during the second preset monitoring period, the redundant file is deleted from the second storage node.

[0047] The second preset period can refer to a preset time period after the redundant files are deleted from the first storage node, such as a preset period of 3 months, used to monitor whether anyone still accesses the deleted redundant files within this preset period.

[0048] In one embodiment, the redundant file has already been deleted from the first storage node in the above steps. At this point, the redundant file is still stored in the second storage node and is in a standard storage state. The system can then continuously monitor for a period of time (e.g., three weeks) to see if there are any further accesses to the redundant file. If not, the redundant file in the second storage node can be changed from standard storage to archive storage, meaning it cannot be accessed. Then, the system monitors for access requests to the redundant file for another period (a second preset monitoring period). Specifically, if no access requests are detected within the second preset monitoring period (e.g., three months), the redundant file is deleted from the second storage node. If access requests are detected within the second preset monitoring period, the second storage state of the file in the second storage node is changed to the first storage state.

[0049] For example, after a redundant file is deleted from the first storage node, the system first keeps the file in a standard storage state on the second storage node and monitors it continuously for a short period (e.g., 3 weeks). If there is no access during this period, the file is converted to archived status and enters a longer monitoring period, namely the second preset monitoring period (e.g., 3 months). Throughout the monitoring process, the system detects any access requests by analyzing access logs in real time (using a Bloom filter for quick filtering). If no access is detected throughout the long monitoring period, the file is permanently deleted from the second storage node. If access is detected at any stage, the unfreezing and cross-node recovery process is immediately triggered, restoring the file to a normal service state and removing it from the deletion queue.

[0050] For example, suppose the redundant file movie_1080p_v1 has been deleted from storage A and remains only in storage B. The system first observes whether it has been accessed for 3 weeks: if not, it is archived in storage B and monitored for another 3 months; if there are still no requests during this period, it is permanently deleted from storage B. Conversely, if a user suddenly requests the file in the second month after archiving, the system will detect this 404 access in real time, immediately unfreeze the file in storage B, copy it back to storage A to restore the dual backup, restore the file status to normal, and remove it from the deletion list, thus achieving safe cleanup without service interruption.

[0051] The key to ensuring high service availability in the above embodiments is the process of sequential processing of nodes. At any given time, at least one storage node is guaranteed to be in (or can be quickly restored to) a serviceable state, thereby reducing the risk of service interruption due to deletion operations to almost zero. At the same time, redundant files can be accurately deleted and storage resources can be intelligently reclaimed without noticing users or causing failures.

[0052] The method provided in this application embodiment obtains a list of redundant files to be processed and performs the following processing on the redundant files in the list: Redundant files stored in a first storage node are converted from a first storage state to a second storage state, and access requests to the redundant files are monitored during a first preset monitoring period; if no access requests to the redundant files are monitored during the first preset monitoring period, the redundant files are deleted from the first storage node; after deleting the redundant files from the first storage node, redundant files stored in a second storage node are converted from a first storage state to a second storage state; and access requests to the redundant files are monitored during a second preset monitoring period; if no access requests to the redundant files are monitored during the second preset monitoring period, the redundant files are deleted from the second storage node. This achieves automated and accurate identification of redundant files, completely replacing inefficient and error-prone manual auditing, and greatly improving processing efficiency and accuracy. Furthermore, during operation on any storage node, service availability can be guaranteed by another storage node, thereby freeing up storage space while achieving highly reliable decoupling of deletion operations and service quality, reducing the risk of accidental deletion to near zero.

[0053] Figure 2 A flowchart illustrating another embodiment of the redundant file management method provided in this application is shown below. Figure 1 Based on the illustrated process, this section mainly describes how to obtain a list of redundant files to be processed, including the following steps: Step 201: Based on the metadata that records the mapping relationship between the first identifier and the second identifier of the referenced data file, update the reverse mapping relationship between the second identifier and the first identifier of the data file.

[0054] Metadata describes the attribute information corresponding to the first identifier. Essentially, it describes the first identifier and its corresponding attribute information stored in the full index. Assuming the first identifier is the program name, its corresponding attribute information could be the file address information of the program, the program's actors, director, and other attribute information. Metadata can be understood as a dataset (such as a database table) that stores the mapping relationship between the first identifier and the second identifier. Its core function is to find the second identifier through the first identifier (for example, to find the corresponding video file address through the program name identifier).

[0055] The first identifier can refer to a program identifier, which can be a unique ID, such as a TVID. Essentially, it represents a program content entity, such as a movie or an episode of a TV series. This is merely an example; the second identifier can also be a content entity in other scenarios, and this application does not impose limitations on this.

[0056] The second identifier can refer to a data file identifier, typically the file's access path or a unique URI. For example, the address of a video segment file: movie_1080p_v1. This is merely an example, and the embodiments of this application do not impose limitations.

[0057] A mapping relationship refers to the correspondence between a program identifier recorded in metadata and the data file it references. Essentially, it's a logical link between a program and the data files that constitute that program.

[0058] The reverse mapping relationship can refer to the mapping direction opposite to the above metadata (full index library). Its core function is to find the first identifier through the second identifier (for example, to find the program ID to which it belongs through the video file address), and it is stored in a dedicated reverse index library.

[0059] In one embodiment, the reverse mapping relationship between the second identifier and the first identifier of the data file is updated by periodically performing a reconstruction operation on the reverse mapping relationship, wherein the reconstruction operation includes: clearing the first identifier in the existing mapping record in the reverse mapping relationship; and re-establishing the mapping record between the second identifier and the first identifier in the reverse mapping relationship based on the current metadata information.

[0060] The rebuild operation can refer to rewriting the contents of the entire index into the reverse index. For example, the old records in the reverse mapping are cleared, and based on the latest metadata, all mappings from the second identifier to the first identifier are re-established to ensure that the reverse mapping is fully synchronized with the metadata.

[0061] The second identifier that is not associated with the first identifier can refer to the second identifier that cannot be found in the reverse mapping relationship (meaning that this data file no longer belongs to any first identifier and has become invalid).

[0062] For example, first, clarify that metadata is a dataset that records the correspondence between program identifiers (first identifier, such as TVID) and data file identifiers (second identifier, such as URI), primarily used to find files by program; the reverse mapping relationship is stored in a reverse index, primarily used to find programs by file. A rebuild operation is performed periodically. First, the program identifiers in the existing mapping records in the reverse index are cleared. Then, based on the latest metadata, the mapping between all data file identifiers and their corresponding program identifiers is re-established, ensuring that the reverse mapping relationship is completely consistent with the metadata.

[0063] For example, suppose the metadata (full index) records a program identifier TVID=1001 (corresponding to the movie "Yuan XX"), and the associated data file identifiers are URI1, URI2, and URI3. The initial reverse mapping relationship of the reverse index is URI1→1001, URI2→1001, URI3→1001. After the movie "Yuan XX" is re-encoded, the metadata is updated to TVID=1001 associated with URI3, URI6, and URI7 (URI1 and URI2 are deleted, and URI6 and URI7 are added). At the set period, all old mapping records of TVID in the reverse index are first cleared, resulting in URI1→empty, URI2→empty, and URI3→empty. Then, based on the updated metadata, the mapping relationship is re-established, that is, the corresponding program identifier is re-filled for each file identifier, and the mapping relationship between the newly added file identifiers and program identifiers is written into the reverse index, resulting in URI1→empty, URI2→empty, URI3→1001, URI6→1001, and URI7→1001, completing the update of the reverse mapping relationship.

[0064] Step 202: Identify the data files corresponding to the second identifier that are not associated with the first identifier in the updated reverse mapping relationship as redundant files, and obtain a list of redundant files.

[0065] The second identifier, which is not associated with the first identifier, can refer to a file identifier (URI) in the reverse index database updated in step 201, where the first identifier (TVID) field is still empty. This indicates that the file corresponding to that file identifier has been discarded, and no program identifier references this discarded file.

[0066] In one embodiment, after the reconstruction operation in step 201 is completed, the system immediately performs a query on the reverse index database to filter out records where the first identifier field (program identifier) ​​is empty. All records with empty second identifier fields (file identifiers) are then collected to obtain the final list of redundant files.

[0067] For example, the system immediately performs a targeted query on the updated reverse index database. The core logic is to filter out all records where the first identifier (i.e., program identifier) ​​field is empty. The second identifier (i.e., file identifier) ​​corresponding to these records with empty fields is deemed by the system logic to be an invalid and redundant file identifier that is no longer used by any business entity because no first identifier (such as program identifier TVID) can be found in the latest metadata on which this reconstruction is based. Subsequently, the system collects and organizes these filtered file identifiers into a structured data set, namely the redundant file list. This list not only contains the identifier of the target file, but also usually adds metadata such as the initial state flag to be observed and the identification timestamp to each entry, thereby providing a clear and reliable target input for subsequent monitoring and secure deletion processes.

[0068] For example, suppose that after a periodic rebuild, the mapping in the reverse index is updated to URI1→empty, URI2→empty, URI3→1001, URI6→1001, URI7→1001. When the system executes a query, it iterates through all records and checks their first identifier field. URI1 and URI2 have empty values, indicating that no program (TVID) references these two files in the latest program metadata; therefore, they are identified as redundant files and included in the list of redundant files to be processed. Conversely, URI3, URI6, and URI7 all have valid program identifiers (1001), indicating that they are still referenced by currently active program content, and therefore are not considered redundant and will not be included in the deletion process. This mechanism ensures accurate identification, marking only truly ownerless files as objects to be cleaned up.

[0069] Step 203: Perform the following processing on the redundant files in the redundant file list: change the redundant files stored in the first storage node from the first storage state to the second storage state, wherein the second storage state makes the redundant files inaccessible directly, and monitor access requests to the redundant files during the first preset monitoring period; wherein, if an access request to the redundant files is detected during the first preset monitoring period, a service response is provided based on the redundant files stored in the second storage node, and the redundant files are changed from the second storage state to the first storage state; if no access request to the redundant files is detected during the first preset monitoring period, the redundant files are deleted from the first storage node.

[0070] Step 204: After deleting the redundant file from the first storage node, the redundant file stored in the second storage node is changed from the first storage state to the second storage state; and during the second preset monitoring period, access requests to the redundant file are monitored; if no access requests to the redundant file are monitored during the second preset monitoring period, the redundant file is deleted from the second storage node.

[0071] For steps 203-204 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0072] pass Figure 2 The description of the embodiment shown illustrates that by periodically reconstructing the reverse mapping relationship, the system can automatically and accurately identify redundant files that are no longer referenced due to program recoding, version updates, etc., completely replacing the inefficient and error-prone manual auditing. This achieves intelligent and batch processing of the redundant file discovery process, enabling more efficient and accurate acquisition of a redundant file list. Subsequently, based on the redundant file list, corresponding hierarchical deletion operations are performed on the relevant redundant files stored in the storage nodes.

[0073] Figure 3 A flowchart illustrating another embodiment of the redundant file management method provided in this application is shown below. Figure 1 Based on the illustrated process, this section mainly describes how to convert redundant files stored in the first storage node from the first storage state to the second storage state, including the following steps: Step 301: Obtain the list of redundant files to be processed.

[0074] For step 301 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0075] Step 302: Perform the following processing on the redundant files in the redundant file list: Mark the redundant files as being under observation and record the start time of the observation state. Continuously monitor access requests to the redundant files for a first preset duration; if no access requests to the redundant files are monitored during the first preset duration from the start time, proceed to step 303; if an access request to the redundant files is monitored before the duration of the observation state reaches the first preset duration, proceed to step 304.

[0076] Step 303: Trigger the execution of the step of converting the redundant files stored in the first storage node from the first storage state to the second storage state.

[0077] Step 304: Remove redundant files from the list of redundant files.

[0078] The following is a unified explanation of steps 302-304 above: The "observation state" refers to an intermediate buffer state after redundant files have been identified but before any storage state changes (such as archiving) have begun. Files in this state still maintain a normal, directly accessible storage state on the storage node (such as standard storage), but the system closely monitors their access behavior.

[0079] The first preset duration can refer to the length of time the file remains in a state of observation, such as 3 weeks. This is a buffer period for secondary verification, designed to capture access requests that may arise shortly after identification.

[0080] The start time can refer to the precise timestamp at which a file is marked as being in an observation state, used to calculate how long it has been in that state.

[0081] In one embodiment, when a file is added to the redundant file list, the system immediately sets its status field to "Under Observation" in the database record of the list and records the current time as the start time. The system continuously monitors access requests to the redundant file until the duration of the file's "Under Observation" status reaches a first preset duration (e.g., 3 weeks). If no access requests to the file are monitored during this period, the system determines that the file has passed the buffer period verification and is a genuine redundant file. Subsequently, the system triggers the subsequent archiving process (i.e., step 305), converting the file from standard storage on the first storage node to archive storage, and deleting the file's corresponding metadata information (i.e., the mapping relationship between the file identifier and its attribute information) from the full index database.

[0082] Furthermore, if the system detects an access request for the file before the first preset time period ends, it will immediately terminate its observation state. The system will remove the file from the redundant file list and write its corresponding mapping relationship back to the normal indexing system, thus removing the file from the entire deletion process. This means that the file still has business value and is not a redundant file.

[0083] For example, suppose the system identifies the file video_legacy.v1 as redundant on June 1st and marks it as pending observation, with a preset buffer period of 3 weeks. Scenario 1 (No Access): From June 1st to June 22nd (within 3 weeks), the system detects no access to video_legacy.v1. On June 22nd, the system determines that the conditions are met and automatically triggers the process to archive it in storage node A. Scenario 2 (Access Scenario): On June 15th, an existing user attempts to play video_legacy.v1 through historical records. The system monitors this access in real time, immediately removes video_legacy.v1 from the list of redundant files to be deleted, and restores it to the full index database so that the file is retained and will not be subsequently cleaned up.

[0084] By setting a safety buffer period in the above steps, it is equivalent to setting a safety valve, which effectively prevents misjudgments that may be caused by the instantaneous nature of the identification logic. In particular, it protects files that are accessed very infrequently but are occasionally needed, further improving the security of the solution.

[0085] Step 305: Convert the redundant files stored in the first storage node from the first storage state to the second storage state. The second storage state prevents the redundant files from being directly accessed. During the first preset monitoring period, monitor access requests to the redundant files. If an access request to the redundant files is detected during the first preset monitoring period, provide a service response based on the redundant files stored in the second storage node and convert the redundant files from the second storage state to the first storage state. If no access request to the redundant files is detected during the first preset monitoring period, delete the redundant files from the first storage node.

[0086] For step 305 above, please refer to the above. Figure 1 Detailed description of the relevant embodiments.

[0087] Step 306: After deleting the redundant file from the first storage node, change the redundant file stored in the second storage node from the first storage state to the second storage state; and monitor access requests to the redundant file during the second preset monitoring period; if no access requests to the redundant file are monitored during the second preset monitoring period, proceed to step 307; if access requests to the redundant file are monitored during the second preset monitoring period, proceed to step 308.

[0088] Step 307: Delete redundant files from the second storage node.

[0089] Step 308: Convert the redundant files in the second storage node from the second storage state to the first storage state.

[0090] The following is a unified explanation of steps 306-308 above: In one embodiment, after redundant files in the first storage node have been deleted, the file exists only in the second storage node and is in an accessible state (first storage state). The system first converts the file in the second storage node to an archived state that is not directly accessible (second storage state) and initiates a second preset monitoring period (e.g., 3 months). During this period, access logs are continuously monitored using techniques such as Bloom filters. If no access requests for the file are found throughout the entire monitoring period, the system ultimately confirms that there is no longer any access requirement and permanently deletes it from the second storage node after the monitoring period ends, completing the entire cleanup process.

[0091] If an access request is detected during the monitoring period (in which case the request will return a 404 error), the system immediately triggers the recovery mechanism: an unfreezing operation is initiated to the second storage node, restoring the file from the second storage state (archived state) to the first storage state (accessible state). Simultaneously, to rebuild the service capability of the redundant file, the system copies the redundant file back to the first storage node and removes it from the deletion list, thus ensuring that file service is fully restored and exiting the cleanup process. This stage achieves the final secure disposal of redundant files under the dual-backup architecture, maximizing storage space release while providing a final line of defense for data security.

[0092] For example, suppose the redundant file 1080p_v1.mp4 is deleted from the first storage node and only exists on the second storage node. The system archives it and starts monitoring for 3 months: if there is no access during the entire period, it will be permanently deleted from the second node after the expiration; if a user accesses the file during the monitoring period (such as the second month) and triggers a 404 error, the system will immediately unfreeze the file and restore it to an accessible state, while copying it back to the first node to rebuild a dual backup, and removing the file from the deletion list, thereby ensuring complete service recovery.

[0093] pass Figure 3 The illustrated embodiment, by adding an observation period and a preset duration of buffer monitoring after redundant files are identified, provides a crucial security verification phase for the system. This effectively filters out pseudo-redundant files that are occasionally accessed, significantly reducing the risk of accidental deletion at the source. Combined with subsequent phased and reversible closed-loop processes across multiple storage nodes, a multi-layered, progressive security protection system is constructed, ensuring that any access request detected at any stage triggers immediate recovery. This achieves automated and refined control over the storage costs of redundant files while maximizing service continuity and data reliability.

[0094] Figure 4 A structural block diagram of a redundant file management device provided in this application embodiment, the device comprising: The redundant file list acquisition module 41 is used to acquire a list of redundant files to be processed; and to perform the following processing on the redundant files in the redundant file list: The first storage node processing module 42 is used to convert the redundant file stored in the first storage node from a first storage state to a second storage state, wherein the second storage state prevents the redundant file from being directly accessed, and monitors access requests to the redundant file during a first preset monitoring period; wherein, if an access request to the redundant file is detected during the first preset monitoring period, a service response is performed based on the redundant file stored in the second storage node, and the redundant file is converted from the second storage state to the first storage state; if no access request to the redundant file is detected during the first preset monitoring period, the redundant file is deleted from the first storage node; The second storage node processing module 43 is used to, after deleting the redundant file from the first storage node, change the redundant file stored in the second storage node from a first storage state to a second storage state; and monitor access requests to the redundant file during a second preset monitoring period; if no access requests to the redundant file are monitored during the second preset monitoring period, the redundant file is deleted from the second storage node.

[0095] In one possible implementation, the redundant file list acquisition module 41 includes: The reverse mapping relationship update unit is used to update the reverse mapping relationship from the second identifier to the first identifier of the data file based on the metadata that records the mapping relationship between the first identifier and the second identifier of the data file it references; The redundant file determination unit is used to determine the data files corresponding to the second identifier that are not associated with the first identifier in the updated reverse mapping relationship as redundant files, thereby obtaining the redundant file list.

[0096] In one possible implementation, the reverse mapping update unit is specifically used for: A reconstruction operation is periodically performed on the reverse mapping relationship, wherein the reconstruction operation includes: Clear the first identifier from the existing mapping records in the reverse mapping relationship; Based on the current metadata information, a new mapping record from the second identifier to the first identifier is established in the reverse mapping relationship.

[0097] In one possible implementation, prior to the first storage node processing module, the following is included: The file marking unit is used to mark the redundant file as an observation state and record the start time of the observation state; The storage state transition unit is configured to trigger the step of transitioning the redundant file stored in the first storage node from the first storage state to the second storage state if a first preset time period has elapsed since the start time and no access request to the redundant file has been detected during the period.

[0098] In one possible implementation, prior to the first storage node processing module, the following further includes: The access request monitoring unit is configured to remove the redundant file from the redundant file list if an access request for the redundant file is detected before the duration of the observed state reaches the first preset duration.

[0099] In one possible implementation, the device further includes: The storage state transition module is used to transition the redundant file in the second storage node from a second storage state to a first storage state if an access request for the redundant file is detected during the second preset monitoring period.

[0100] In one possible implementation, the first storage node processing module monitors access requests to the redundant files in the following manner: Continuously monitor access events in the file access log and convert redundant file identifiers associated with the access events into corresponding target binary values ​​according to preset rules; A Bloom filter is found based on the target binary value; wherein the Bloom filter contains a list of binary values ​​corresponding to the list of redundant files calculated based on the preset rules; If the target binary value is found in the Bloom filter, the redundant file identifier is further searched in the redundant file list. If the redundant file identifier is found, it is determined that an access request to the redundant file has been monitored. If the target binary value is not found in the Bloom filter, it is determined that no access request to the redundant file was detected.

[0101] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the redundant file management method provided in any of the foregoing method embodiments, including: Obtain a list of redundant files to be processed; perform the following processing on the redundant files in the list: Redundant files stored in the first storage node are converted from a first storage state to a second storage state, whereby the second storage state prevents the redundant files from being directly accessed. Access requests to the redundant files are monitored during a first preset monitoring period. If an access request to the redundant files is detected during the first preset monitoring period, a service response is provided based on the redundant files stored in the second storage node, and the redundant files are converted from the second storage state to the first storage state. If no access request to the redundant files is detected during the first preset monitoring period, the redundant files are deleted from the first storage node. After deleting redundant files from the first storage node, the redundant files stored in the second storage node are switched from the first storage state to the second storage state; and during the second preset monitoring period, access requests to the redundant files are monitored; if no access requests to the redundant files are monitored during the second preset monitoring period, the redundant files are deleted from the second storage node.

[0102] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the redundant file management method provided in any of the foregoing method embodiments.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0106] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for managing redundant files, characterized in that, The method includes: Obtain a list of redundant files to be processed; perform the following processing on the redundant files in the list of redundant files: The redundant file stored in the first storage node is converted from a first storage state to a second storage state, wherein the second storage state prevents the redundant file from being directly accessed. Access requests to the redundant file are monitored during a first preset monitoring period. If an access request to the redundant file is detected during the first preset monitoring period, a service response is provided based on the redundant file stored in the second storage node, and the redundant file is converted from the second storage state to the first storage state. If no access request to the redundant file is detected during the first preset monitoring period, the redundant file is deleted from the first storage node. After deleting the redundant file from the first storage node, the redundant file stored in the second storage node is changed from a first storage state to a second storage state; and during a second preset monitoring period, access requests to the redundant file are monitored; if no access requests to the redundant file are monitored during the second preset monitoring period, the redundant file is deleted from the second storage node.

2. The method according to claim 1, characterized in that, The process of obtaining the list of redundant files to be processed includes: Based on the metadata that records the mapping relationship between the first identifier and the second identifier of the referenced data file, update the reverse mapping relationship from the second identifier to the first identifier of the data file; The data files corresponding to the second identifier that are not associated with the first identifier in the updated reverse mapping relationship are identified as redundant files, thus obtaining the list of redundant files.

3. The method according to claim 2, characterized in that, The reverse mapping relationship from the second identifier to the first identifier in the updated data file includes: A reconstruction operation is periodically performed on the reverse mapping relationship, wherein the reconstruction operation includes: Clear the first identifier from the existing mapping records in the reverse mapping relationship; Based on the current metadata information, a new mapping record from the second identifier to the first identifier is established in the reverse mapping relationship.

4. The method according to claim 1, characterized in that, Before the redundant files stored in the first storage node are converted from the first storage state to the second storage state, the following steps are included: The redundant files are marked as an observation state, and the start time of the observation state is recorded; If a first preset time period elapses from the start time and no access request to the redundant file is detected during this period, then the step of converting the redundant file stored in the first storage node from the first storage state to the second storage state is triggered.

5. The method according to claim 4, characterized in that, The method further includes: If an access request to the redundant file is detected before the duration of the observed state reaches the first preset duration, the redundant file is removed from the redundant file list.

6. The method according to claim 1, characterized in that, The method further includes: If an access request for the redundant file is detected during the second preset monitoring period, the redundant file in the second storage node will be changed from the second storage state to the first storage state.

7. The method according to claim 1, characterized in that, The monitoring requests for access to the redundant files include: Continuously monitor access events in the file access log and convert redundant file identifiers associated with the access events into corresponding target binary values ​​according to preset rules; A Bloom filter is found based on the target binary value; wherein the Bloom filter contains a list of binary values ​​corresponding to the list of redundant files calculated based on the preset rules; If the target binary value is found in the Bloom filter, the redundant file identifier is further searched in the redundant file list. If the redundant file identifier is found, it is determined that an access request to the redundant file has been monitored. If the target binary value is not found in the Bloom filter, it is determined that no access request to the redundant file was detected.

8. A redundant file management device, characterized in that, The device includes: A redundant file list acquisition module is used to acquire a list of redundant files to be processed; and to perform the following processing on the redundant files in the redundant file list: A first storage node processing module is configured to convert the redundant files stored in the first storage node from a first storage state to a second storage state, wherein the second storage state prevents the redundant files from being directly accessed, and monitor access requests to the redundant files during a first preset monitoring period; wherein, if an access request to the redundant files is detected during the first preset monitoring period, a service response is provided based on the redundant files stored in the second storage node, and the redundant files are converted from the second storage state to the first storage state; if no access request to the redundant files is detected during the first preset monitoring period, the redundant files are deleted from the first storage node; The second storage node processing module is configured to, after deleting the redundant file from the first storage node, change the redundant file stored in the second storage node from a first storage state to a second storage state; and monitor access requests to the redundant file during a second preset monitoring period; if no access requests to the redundant file are monitored during the second preset monitoring period, then delete the redundant file from the second storage node.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a redundant file management program stored in the memory to implement the redundant file management method according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the redundant file management method according to any one of claims 1-7.

Images