Redundant file identification method, device, equipment, medium and computer program product

Abstract

The application discloses a kind of redundancy file identification method, device, equipment, medium and computer program product, the method includes: obtaining seed file and the metadata information and file label of each file in file library specified by user;According to file label and file type, file is divided into corresponding fine classification set;For seed file and file in file library, respectively extract semantic feature vector and metadata feature vector and carry out multimodal fusion, obtain the comprehensive feature vector of each file;The similarity between the comprehensive feature vector of seed file and other files in the same fine classification set is calculated;The file with similarity greater than preset threshold is filtered as candidate redundant file, and a redundant file list is generated.The application fully utilizes different modal information to identify similar redundant files with the seed file selected by the user by fusing metadata information, picture encoding information and text encoding information, thereby effectively improving the identification accuracy and efficiency of redundant files.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, medium, and computer program product for redundant file identification. Background Technology

[0002] Cloud storage is a cloud storage system that provides online storage, access, backup, and sharing capabilities. Users typically store large amounts of files on cloud storage; however, much of this content is duplicated or expired. For cloud storage users, since cloud storage capacity is usually limited and paid, regular checks and timely cleanup of redundant files are necessary to reduce storage space usage. For cloud storage providers, redundant files consume server storage space, increasing operating costs.

[0003] Currently, methods for identifying redundant files include those based on metadata comparison (low computational complexity but prone to false positives), binary data comparison (high accuracy but extremely high computational complexity, making them unsuitable for real-time scenarios), and file feature value comparison (achieving a good balance between accuracy and efficiency, but only completely identical files are identified and offered to the user for deletion). These methods fail to identify useless files with different metadata or similar but not identical content, requiring users to manually filter them, severely impacting user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, medium, and computer program product for identifying redundant files. By integrating metadata information, image encoding information, and text encoding information, it can fully utilize information from different modalities to identify a list of redundant files similar to a seed file selected by the user, thereby effectively improving the accuracy and efficiency of redundant file identification.

[0005] To achieve the above objectives, embodiments of the present invention provide a method for identifying redundant files, including: Retrieve at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; Based on the file tags and file types, the files in the file library are divided into corresponding subcategories; wherein, the file types include text files and image files; For the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain the comprehensive feature vector of each file; Based on the file type, calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-class set; Files with a similarity greater than a preset threshold are selected as candidate redundant files, and a list of redundant files is generated.

[0006] As an improvement to the above solution, the step of dividing the files in the file library into corresponding subcategories based on the file tags and file types includes: Based on the file extension, files are roughly classified into text files and image files to determine the file type. For the files after coarse classification, fine classification is performed based on the file tags, and files with the same file tags are grouped into the same fine classification set; wherein, the file tags are pre-generated using a text classification model, an image classification model, or a clustering model.

[0007] As an improvement to the above scheme, for the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain a comprehensive feature vector for each file, including: For the seed file and the files in the file library, different encoding models are used to extract semantic feature vectors according to the different file types; Extract metadata feature vectors based on the file path, file name, and file size in the metadata information; According to the file type, the extracted semantic feature vector and the metadata feature vector are concatenated to obtain the comprehensive feature vector of each file.

[0008] As an improvement to the above scheme, the step of extracting semantic feature vectors using different encoding models based on different file types includes: For the text file, a large language model is used to encode the text content and extract the text semantic feature vector; For the image file, a pre-trained convolutional neural network model is used to encode the image content and extract the semantic feature vector of the image.

[0009] As an improvement to the above scheme, the step of concatenating the extracted semantic feature vector and the metadata feature vector according to the file type to obtain a comprehensive feature vector for each file includes: For the text file, the extracted text semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the text file; For the image file, the extracted image semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the image file.

[0010] As an improvement to the above scheme, after filtering files with a similarity greater than a preset threshold as candidate redundant files and generating a list of redundant files, the method further includes: The candidate redundant files are sorted in ascending order based on their historical access frequency, with the candidate redundant files having a lower historical access frequency ranking higher. When the historical access frequency of the candidate redundant files is the same, the candidate redundant files are sorted in ascending order according to their last update time, and the candidate redundant files with the oldest last update time are ranked higher. After reordering, a list of target redundant files is generated.

[0011] This invention also provides a redundant file identification device, comprising: The data acquisition module is used to acquire at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; The file classification module is used to divide the files in the file library into corresponding sub-categories based on the file tags and file types; wherein, the file types include text files and image files; The feature extraction module is used to extract semantic feature vectors and metadata feature vectors from the seed file and the files in the file library, respectively, and perform multimodal fusion to obtain a comprehensive feature vector for each file. The similarity calculation module is used to calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-category set, according to the file type. The redundancy filtering module is used to filter files with a similarity greater than a preset threshold as candidate redundant files and generate a list of redundant files.

[0012] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the redundant file identification method described above.

[0013] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the redundant file identification method described above.

[0014] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the redundant file identification method described above.

[0015] Compared to existing technologies, the beneficial effects of the redundant file identification method, apparatus, device, medium, and computer program product provided by this invention are as follows: By acquiring at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; based on the file tags and file types, the files in the file library are divided into corresponding sub-categories; wherein, the file types include text files and image files; for the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain a comprehensive feature vector for each file; according to the file type, the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vectors of other files in the same sub-categorie is calculated; files with similarity greater than a preset threshold are selected as candidate redundant files, generating a redundant file list. This invention, by fusing metadata information, image encoding information, and text encoding information, can fully utilize information from different modalities to identify a list of redundant files similar to the seed file selected by the user, thereby effectively improving the accuracy and efficiency of redundant file identification. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a preferred embodiment of a redundant file identification method provided by the present invention; Figure 2 This is a schematic diagram of the system architecture in a redundant file identification method provided by the present invention; Figure 3 This is a schematic diagram of file information in a redundant file identification method provided by the present invention; Figure 4 This is a schematic diagram of file classification in a redundant file identification method provided by the present invention; Figure 5 This is a schematic diagram of a preferred embodiment of a redundant file recognition device provided by the present invention; Figure 6 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 , Figure 1This is a flowchart illustrating a preferred embodiment of a redundant file identification method provided by the present invention. The redundant file identification method includes: S1, obtain at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; S2, based on the file tags and file types, divide the files in the file library into corresponding subcategories; wherein, the file types include text files and image files; S3, For the seed file and the files in the file library, extract semantic feature vectors and metadata feature vectors respectively, and perform multimodal fusion to obtain the comprehensive feature vector of each file; S4. According to the file type, calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-classification set; S5: Filter files with similarity greater than a preset threshold as candidate redundant files and generate a list of redundant files.

[0019] It should be noted that this invention provides a redundant file recognition method based on multimodal learning. A modality refers to a way of expressing or perceiving things; each source or form of information can be called a modality, such as speech, images, and text. Multimodal learning is the ability to process and understand multi-source modal information by learning from data of multiple modalities through machine learning methods. Cloud storage contains data of multiple modalities, such as images, text, audio, and numerical information. The redundant file recognition method based on multimodal learning can compensate for the shortcomings of insufficient information in a single modality, complementing information from different modalities and improving the predictive ability and accuracy of the model.

[0020] Specifically, the redundant file identification method provided in this embodiment of the invention is applied to a redundant file identification system. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the system architecture of a redundant file identification method provided by the present invention. The redundant file identification system includes a file classification module, a semantic encoding module, a metadata encoding module, a multimodal fusion module, a similarity calculation module, and a re-ranking module. In this embodiment of the invention, firstly, at least one seed file to be deleted specified by the user is obtained, along with metadata information and pre-configured file tags for each file in the file library. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of file information in a redundant file identification method provided by the present invention. The files in the file library include text files and image files. Metadata information includes file path, file name, file size, file usage frequency, and last update time. File tags include animals, landscapes, etc. Please refer to... Figure 4 , Figure 4This is a schematic diagram of file classification in a redundant file identification method provided by the present invention. The file classification module divides files in the file library into corresponding sub-category sets based on file tags and file types. File types include text files and image files, and each file has a pre-defined file tag, such as portrait, landscape, architecture, news, notes, etc. Subsequently, for seed files and files in the file library, the semantic encoding module extracts semantic feature vectors, and the metadata encoding module extracts metadata feature vectors. After extracting the semantic and metadata feature vectors, the multimodal fusion module performs multimodal fusion on the semantic and metadata feature vectors to obtain the comprehensive feature vector of each file. Then, the similarity calculation module calculates the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files within the same sub-category set, according to file type. For example, the similarity is calculated between each seed file and other files in the same sub-category set according to file type (image file or text file). This embodiment of the invention uses cosine similarity to calculate the similarity between files. For files... and documents Comprehensive feature vector and The file is calculated using the following formula. and documents Similarity between them: .

[0021] After the similarity calculation is completed, files with a similarity greater than a preset threshold are selected as candidate redundant files, generating a redundant file list. For example, in this embodiment of the invention, the preset threshold is set to 0.9, and files with a similarity greater than 0.9 will be added to the redundant file list. Furthermore, the list of redundant files is sorted from highest to lowest similarity.

[0022] This invention differs from existing redundant file identification schemes. It extracts semantic features of files (text or images) through a deep learning model and combines them with file metadata such as filename, file path, file size, file type, and file tags. This allows for the full utilization of information from different modalities to identify a list of redundant files similar to the seed file selected by the user. It can identify not only redundant files that are completely identical to the seed file, but also redundant files that are similar to the seed file, covering more useless files. This effectively improves the accuracy and efficiency of redundant file identification, reduces the storage space occupied by useless files in the cloud drive, improves the utilization rate of the cloud drive, and enhances the user experience.

[0023] In another preferred embodiment, step S2, which involves dividing the files in the file library into corresponding subcategories based on the file tags and file types, includes: S21. Based on the file extension, files are roughly classified into text files and image files to obtain the file type. S22, for the files after coarse classification, further classification is performed according to the file tags, and files with the same file tags are grouped into the same sub-classification set; wherein, the file tags are pre-generated by a text classification model, an image classification model, or a clustering model.

[0024] Specifically, in this embodiment of the invention, files are coarsely classified based on their file extensions, resulting in text files and image files, thus determining file types. Then, for the text and image files obtained from the coarse classification, finer classification is performed based on file tags, grouping files with the same tags into the same fine category set. This way, in the subsequent similarity calculation module, the seed file only needs to be compared with files belonging to the same fine category set, significantly reducing the number of files to be compared and improving comparison efficiency. The file tags can be pre-generated using a text classification model, an image classification model, or a clustering model.

[0025] The embodiments of the present invention use file type and file tag to perform preliminary screening of files to be compared, which can reduce the number of files to be compared for seed files, thereby effectively reducing computational complexity and improving file comparison efficiency.

[0026] In another preferred embodiment, step S3 involves extracting semantic feature vectors and metadata feature vectors from the seed file and the files in the file library, respectively, and performing multimodal fusion to obtain a comprehensive feature vector for each file, including: S31, For the seed file and the files in the file library, different encoding models are used to extract semantic feature vectors according to the different file types; S32, extract the metadata feature vector based on the file path, file name, and file size in the metadata information; S33, according to the file type, the extracted semantic feature vector and the metadata feature vector are concatenated to obtain the comprehensive feature vector of each file.

[0027] Specifically, in this embodiment of the invention, different encoding models are used to extract semantic feature vectors for seed files and files in the file library, depending on the file type. Simultaneously, metadata feature vectors are extracted based on the file path, filename, and file size in the file's metadata information. For example, the relative path of the file in a cloud drive or other file system is obtained. For example, for the path " / documents / artificial intelligence / natural language processing", the open-source large language model LLama3 is used to encode the path. The output of the last layer is taken and passed through a fully connected layer to finally obtain the file path vector representation. Similarly, for filenames File size Each filename was encoded using the LLama3 model, and then passed through a fully connected layer to obtain the final filename vector representation. and file size vector representation .Right now: ; ; ; In the above formula, The output of the fully connected layer.

[0028] Then, according to the file type, the extracted semantic feature vector and metadata feature vector are concatenated to obtain the comprehensive feature vector of each file.

[0029] This invention extracts metadata feature vectors, fully utilizing information such as file storage path, filename, and file size. This enables the identification of redundant files similar to the user-selected seed file path and name, and file size can be used as an auxiliary factor in the judgment, resulting in more accurate identification. By fusing metadata, image encoding, and text encoding information, it can fully utilize information from different modalities to identify redundant files similar to the user-selected seed file, thereby effectively improving the accuracy and efficiency of redundant file identification and reducing the storage space occupied by useless files in the cloud drive.

[0030] In yet another preferred embodiment, the step of extracting semantic feature vectors using different encoding models based on the different file types includes: For the text file, a large language model is used to encode the text content and extract the text semantic feature vector; For the image file, a pre-trained convolutional neural network model is used to encode the image content and extract the semantic feature vector of the image.

[0031] Specifically, in this embodiment of the invention, for text files, a large language model is used to encode the text content and extract text semantic feature vectors. For example, the text file undergoes format conversion, data cleaning, stop word removal, and word segmentation preprocessing to obtain the original text. The original text is then segmented with overlap according to a preset length, resulting in multiple text segments. The overlap length is dynamically adjusted based on the segment length and preset hyperparameters. Each text segment is input into the large language model, and the output of the hidden state of the last layer of the model is used as the vector representation of each text segment. Pooling is performed on the vector representations of each text segment, and the pooled vector sequence is input into a recurrent neural network layer to output the final text semantic feature vector.

[0032] For example, embodiments of the present invention use the open-source large language model LLama3 to encode the text content, assuming the length of the original text is... Each segment segment The length is denoted as , No. The text segment is denoted as : ; in, Indicates the first in the text segment Each word element. Two adjacent text segments. and The overlap length between them is denoted as , The calculation method is as follows: ; in, The segment length, These are preset hyperparameters used to adjust the length of text segments when the total length exceeds the length of the overlapping interval. C This is the upper limit of a constant. In the embodiments of the present invention... Set to 4000, Set to 0.2, C Set to 100. The total number of text segments is recorded as follows: The length of the text, the length of each segment, and the overlap length are calculated using a piecewise function. The calculation formula is as follows: ; in, This represents the floor function. It rounds the text segment up. segment Each element is fed into the LLama3 model, and the output of the last layer of the LLama3 model is taken as the vector representation of the text segment, i.e.: ; in, For the first indivual The vector representation of , The dimension is denoted as . This indicates that the LLama3 model is used for encoding, and the last_hidden_state of the last layer is taken as the output. Then, it passes through a max-pooling layer to obtain... : ; Then, the representations of each text segment are concatenated into a new text segment sequence.

[0033] }; Text segment sequence As input to the LSTM layer, the output of the LSTM layer This is the vector representation of the original text of the file, i.e.: .

[0034] For image files, a pre-trained convolutional neural network model is used to encode the image content and extract semantic feature vectors. For example, for image files... The pre-trained network EfficientNetV2 is used to extract image features, which are then encoded to obtain the image semantic feature vector. ,Right now: .

[0035] This invention encodes the semantic information of text and images to capture their rich semantic meaning, thereby enabling the identification of similar redundant files based on semantic similarity and improving the accuracy of redundant file recognition.

[0036] In another preferred embodiment, the step of concatenating the extracted semantic feature vector and the metadata feature vector according to the file type to obtain a comprehensive feature vector for each file includes: For the text file, the extracted text semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the text file; For the image file, the extracted image semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the image file.

[0037] Specifically, in this embodiment of the invention, for text files... The extracted text semantic feature vector Metadata feature vector , , By concatenating the features, we obtain the final comprehensive feature vector representing the text file. :

[0038] For image files The extracted image semantic feature vector Metadata feature vector 、 , The features are concatenated to obtain the final comprehensive feature vector representing the image file. :

[0039] In yet another preferred embodiment, after filtering files with a similarity greater than a preset threshold as candidate redundant files and generating a list of redundant files, the method further includes: The candidate redundant files are sorted in ascending order based on their historical access frequency, with the candidate redundant files having a lower historical access frequency ranking higher. When the historical access frequency of the candidate redundant files is the same, the candidate redundant files are sorted in ascending order according to their last update time, and the candidate redundant files with the oldest last update time are ranked higher. After reordering, a list of target redundant files is generated.

[0040] Specifically, after generating a list of redundant files based on similarity, this embodiment of the invention can further reorder the list of redundant files according to their historical access frequency and last update time, recommending the reordered list to the user to assist in quick decision-making. For example, during reordering, the candidate redundant files are sorted in ascending order based on their historical access frequency, with lower historical access frequencies ranking higher. This is because a higher historical access frequency indicates higher file importance, and a lower historical access frequency indicates lower file importance; therefore, files with lower historical access frequencies rank higher, making it easier for users to select and delete them. When candidate redundant files have the same historical access frequency, they are sorted in ascending order based on their last update time, with older last update times ranking higher. This is because recently accessed and updated files are highly likely to be used again, while files that haven't been accessed or updated recently are highly unlikely to be accessed again; therefore, files that haven't been accessed or updated for a longer period of time rank higher. After reordering, a final target redundant file list is generated. .

[0041] This invention proposes a reordering strategy based on the frequency of file usage and update time to reorder redundant file lists, facilitating quick decision-making for users.

[0042] In addition, this embodiment of the invention also includes a model training method. Model training first requires adding training labels to the dataset. It should be noted that these training labels are different from the file labels required by the file classification module. The file labels in the file classification module are category labels, used to identify which sub-category the file belongs to. These training labels are used to mark whether files are similar. Similar or identical files have the same training label, while dissimilar files have different training labels. Sample pairs with the same training label are denoted as the set of positive sample pairs. Sample pairs with different training labels can be denoted as the set of negative sample pairs. In this embodiment of the invention, the CoSENT loss function is used as the loss function for model training.

[0043] The model proposed in this invention is based on multimodal learning, integrating file semantic features and metadata information for judgment. It also fully considers user usage frequency and update time to generate a list of redundant files for the user, assisting them in processing useless redundant files more quickly. Compared to existing methods, the method proposed in this invention has higher coverage, can identify redundant files that existing solutions cannot detect, significantly improves the efficiency of deleting redundant files, and greatly enhances the user experience.

[0044] Accordingly, the present invention also provides a redundant file identification device, which can implement all the processes of the redundant file identification method in the above embodiments.

[0045] Please see Figure 5 , Figure 5 This is a schematic diagram of a preferred embodiment of a redundant file identification device provided by the present invention. The redundant file identification device includes: The data acquisition module 501 is used to acquire at least one seed file specified by the user, as well as metadata information and pre-configured file tags of each file in the file library; The file classification module 502 is used to divide the files in the file library into corresponding sub-categories based on the file tags and file types; wherein, the file types include text files and image files; The feature extraction module 503 is used to extract semantic feature vectors and metadata feature vectors from the seed file and the files in the file library, respectively, and perform multimodal fusion to obtain a comprehensive feature vector for each file. The similarity calculation module 504 is used to calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-classification set according to the file type. The redundancy filtering module 505 is used to filter files with a similarity greater than a preset threshold as candidate redundant files and generate a list of redundant files.

[0046] Preferably, the file classification module 502 is specifically used for: Based on the file extension, files are roughly classified into text files and image files to determine the file type. For the files after coarse classification, fine classification is performed based on the file tags, and files with the same file tags are grouped into the same fine classification set; wherein, the file tags are pre-generated using a text classification model, an image classification model, or a clustering model.

[0047] Preferably, the feature extraction module 503 is specifically used for: For the seed file and the files in the file library, different encoding models are used to extract semantic feature vectors according to the different file types; Extract metadata feature vectors based on the file path, file name, and file size in the metadata information; According to the file type, the extracted semantic feature vector and the metadata feature vector are concatenated to obtain the comprehensive feature vector of each file.

[0048] Preferably, the step of extracting semantic feature vectors using different encoding models according to different file types includes: For the text file, a large language model is used to encode the text content and extract the text semantic feature vector; For the image file, a pre-trained convolutional neural network model is used to encode the image content and extract the semantic feature vector of the image.

[0049] Preferably, the step of concatenating the extracted semantic feature vector and the metadata feature vector according to the file type to obtain a comprehensive feature vector for each file includes: For the text file, the extracted text semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the text file; For the image file, the extracted image semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the image file.

[0050] Preferably, the device further includes a reordering module for: The candidate redundant files are sorted in ascending order based on their historical access frequency, with the candidate redundant files having a lower historical access frequency ranking higher. When the historical access frequency of the candidate redundant files is the same, the candidate redundant files are sorted in ascending order according to their last update time, and the candidate redundant files with the oldest last update time are ranked higher. After reordering, a list of target redundant files is generated.

[0051] In specific implementation, the working principle, control process and technical effects of the redundant file identification device provided in the embodiments of the present invention are the same as those of the redundant file identification method in the above embodiments, and will not be repeated here.

[0052] Please see Figure 6 , Figure 6This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 601, a memory 602, and a computer program stored in the memory 602 and configured to be executed by the processor 601. When the processor 601 executes the computer program, it implements the redundant file identification method described in any of the above embodiments.

[0053] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 602 and executed by the processor 601 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0054] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor 601 may be any conventional processor. The processor 601 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0055] The memory 602 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory 602 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard drive, a smart media card (SMC), a secure digital card (SD), and a flash card, or it can be other volatile solid-state storage devices.

[0056] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 6The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.

[0057] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the redundant file identification method described in any of the above embodiments.

[0058] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the redundant file identification method described in any of the above embodiments.

[0059] This invention provides a method, apparatus, device, medium, and computer program product for identifying redundant files. It acquires at least one seed file specified by a user, along with metadata information and pre-configured file tags for each file in a file library. Based on the file tags and file types, the files in the file library are divided into corresponding sub-categories. The file types include text files and image files. For the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain a comprehensive feature vector for each file. According to the file types, the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vectors of other files in the same sub-categorie is calculated. Files with similarity greater than a preset threshold are selected as candidate redundant files, generating a list of redundant files. This invention, by fusing metadata information, image encoding information, and text encoding information, can fully utilize information from different modalities to identify a list of redundant files similar to the user-selected seed file, thereby effectively improving the accuracy and efficiency of redundant file identification.

[0060] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0061] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying redundant files, characterized in that, include: Retrieve at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; Based on the file tags and file types, the files in the file library are divided into corresponding subcategories; wherein, the file types include text files and image files; For the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain the comprehensive feature vector of each file; Based on the file type, calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-class set; Files with a similarity greater than a preset threshold are selected as candidate redundant files, and a list of redundant files is generated.

2. The redundant file identification method as described in claim 1, characterized in that, The step of classifying files in the file library into corresponding subcategories based on the file tags and file types includes: Based on the file extension, files are roughly classified into text files and image files to determine the file type. For the files after coarse classification, fine classification is performed based on the file tags, and files with the same file tags are grouped into the same fine classification set; wherein, the file tags are pre-generated using a text classification model, an image classification model, or a clustering model.

3. The redundant file identification method as described in claim 1, characterized in that, For the seed file and the files in the file library, semantic feature vectors and metadata feature vectors are extracted respectively, and multimodal fusion is performed to obtain a comprehensive feature vector for each file, including: For the seed file and the files in the file library, different encoding models are used to extract semantic feature vectors according to the different file types; Extract metadata feature vectors based on the file path, file name, and file size in the metadata information; According to the file type, the extracted semantic feature vector and the metadata feature vector are concatenated to obtain the comprehensive feature vector of each file.

4. The redundant file identification method as described in claim 3, characterized in that, The step of extracting semantic feature vectors using different encoding models based on the file type includes: For the text file, a large language model is used to encode the text content and extract the text semantic feature vector; For the image file, a pre-trained convolutional neural network model is used to encode the image content and extract the semantic feature vector of the image.

5. The redundant file identification method as described in claim 4, characterized in that, The step involves concatenating the extracted semantic feature vector and the metadata feature vector according to the file type to obtain a comprehensive feature vector for each file, including: For the text file, the extracted text semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the text file; For the image file, the extracted image semantic feature vector is concatenated with the metadata feature vector to obtain the comprehensive feature vector of the image file.

6. The redundant file identification method as described in claim 1, characterized in that, After filtering files with a similarity greater than a preset threshold as candidate redundant files and generating a list of redundant files, the process further includes: The candidate redundant files are sorted in ascending order based on their historical access frequency, with the candidate redundant files having a lower historical access frequency ranking higher. When the historical access frequency of the candidate redundant files is the same, the candidate redundant files are sorted in ascending order according to their last update time, and the candidate redundant files with the older last update time are ranked higher. After reordering, a list of target redundant files is generated.

7. A redundant document identification device, characterized in that, include: The data acquisition module is used to acquire at least one seed file specified by the user, as well as metadata information and pre-configured file tags for each file in the file library; The file classification module is used to divide the files in the file library into corresponding sub-category sets according to the file tags and file types; wherein, the file types include text files and image files; The feature extraction module is used to extract semantic feature vectors and metadata feature vectors from the seed file and the files in the file library, respectively, and perform multimodal fusion to obtain a comprehensive feature vector for each file. The similarity calculation module is used to calculate the similarity between the comprehensive feature vector of the seed file and the comprehensive feature vector of other files in the same sub-category set, according to the file type. The redundancy filtering module is used to filter files with a similarity greater than a preset threshold as candidate redundant files and generate a list of redundant files.

8. A terminal device, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the redundant file identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the redundant file identification method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the redundant file identification method as described in any one of claims 1 to 6.

Images