Data deduplication management method and device based on deep learning and electronic equipment

Abstract

This application relates to the field of intelligent data deduplication, specifically disclosing a data deduplication management method, apparatus, and electronic device based on deep learning. It acquires image information from storage space, preprocesses the image information to obtain preprocessed image information, and utilizes deep learning-based image analysis and processing algorithms to extract salient features and perform image semantic association on the preprocessed image information, thereby identifying duplicate images in the storage space and providing this information to the user. In this way, by identifying and deleting duplicate images, storage space can be freed up, improving the utilization rate of storage resources. Simultaneously, the automated deduplication process reduces the workload of manual checking and deleting of duplicate files, improving the efficiency of data management.

Description

Technical Field

[0001] This application relates to the field of intelligent data deduplication, and more specifically, to a data deduplication management method, apparatus, and electronic device based on deep learning. Background Technology

[0002] With the development of information technology, the amount of data generated by individuals and organizations has increased dramatically, leading to a large amount of duplicate or redundant data. Storage devices have limited cost and space. Duplicate data occupies valuable storage resources and increases storage costs. Therefore, data deduplication has become particularly important. Removing duplicate data can save storage space, optimize the use of storage resources, and reduce hardware costs. This is especially important for rail transit control systems, which generate a large amount of duplicate data during operation. Examples include images taken by inspection personnel at the same inspection point at different times, and images periodically collected by trackside cameras. These images all require deduplication to reduce the data storage pressure on the rail transit control system.

[0003] However, traditional deduplication methods often require a large amount of manual work, resulting in slow processing speeds, especially on large-scale datasets. Furthermore, manual comparison and identification of duplicate data are susceptible to fatigue and subjective judgment, leading to errors and omissions. Secondly, as data volumes increase, traditional deduplication methods struggle to scale to meet ever-growing processing demands.

[0004] Therefore, we look forward to an optimized data deduplication management solution. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a data deduplication management method, apparatus, and electronic device based on deep learning. This method acquires image information from storage space, preprocesses the image information to obtain preprocessed image information, and utilizes deep learning-based image analysis and processing algorithms to extract salient features and perform image semantic association on the preprocessed image information. This identifies duplicate images in the storage space and provides this information to the user. In this way, by identifying and deleting duplicate images, storage space can be freed up, improving the utilization rate of storage resources. Simultaneously, the automated deduplication process reduces the workload of manual checking and deleting of duplicate files, improving the efficiency of data management.

[0006] According to one aspect of this application, a deep learning-based data deduplication management method is provided, comprising:

[0007] Retrieve image information from storage space;

[0008] The image information is preprocessed to obtain preprocessed image information;

[0009] The preprocessed image information is input into the image salient feature capture module to obtain a sequence of image salient feature maps;

[0010] For each of the image saliency feature maps in the sequence of image saliency feature maps, image saliency enhancement and image saliency semantic association are performed respectively to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors;

[0011] The sequence of the image saliency enhancement feature vector and the sequence of the image saliency semantic association feature vector are fused to obtain a sequence of image saliency feature semantic fusion representation vectors;

[0012] Calculate the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information;

[0013] Based on the image similarity information, duplicate images in the storage space are obtained and fed back to the user.

[0014] According to another aspect of this application, a data deduplication management device based on deep learning is provided, comprising:

[0015] The image information acquisition module for storage space is used to acquire image information in the storage space;

[0016] The image information preprocessing module is used to preprocess the image information to obtain preprocessed image information;

[0017] The image saliency feature extraction module is used to input the preprocessed image information into the image saliency feature capture module to obtain a sequence of image saliency feature maps;

[0018] The image saliency feature enhancement and semantic association module is used to perform image saliency enhancement and image saliency semantic association on each image saliency feature map in the sequence of image saliency feature maps to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors;

[0019] The image feature fusion module is used to fuse the sequence of the image saliency enhancement feature vector and the sequence of the image saliency semantic association feature vector to obtain the sequence of image saliency feature semantic fusion representation vector;

[0020] The image similarity calculation module is used to calculate the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information;

[0021] The duplicate image acquisition module is used to obtain duplicate images in the storage space based on the image similarity information and then provide feedback to the user.

[0022] According to another aspect of this application, an electronic device is also provided, comprising: a processor and a memory, wherein a computer-executable program is stored in the memory, and when the computer-executable program is executed by the processor, it implements the data deduplication management method based on deep learning described above.

[0023] Compared with existing technologies, this application provides a data deduplication management method, apparatus, and electronic device based on deep learning. It acquires image information from storage space, preprocesses the image information to obtain preprocessed image information, and utilizes deep learning-based image analysis and processing algorithms to extract salient features and perform semantic association on the preprocessed image information. This identifies duplicate images in the storage space and provides this information to the user. In this way, by identifying and deleting duplicate images, storage space can be freed up, improving the utilization rate of storage resources. Simultaneously, the automated deduplication process reduces the workload of manual checking and deleting of duplicate files, improving the efficiency of data management. Attached Figure Description

[0024] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0025] Figure 1 This is a flowchart of a deep learning-based data deduplication management method according to an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of the architecture of a deep learning-based data deduplication management method according to an embodiment of this application.

[0027] Figure 3 This is a flowchart illustrating the process of performing image saliency enhancement and image saliency semantic association on each of the image saliency feature maps in the sequence of image saliency feature maps according to the data deduplication management method based on deep learning in this application, to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors.

[0028] Figure 4This is a flowchart illustrating how a deep learning-based data deduplication management method, according to an embodiment of this application, obtains duplicate images in a storage space based on image similarity information and feeds them back to the user.

[0029] Figure 5 This is a block diagram of a deep learning-based data deduplication management device according to an embodiment of this application.

[0030] Figure 6 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0033] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0034] It should be noted that the use of the terms "a" or "a plurality of" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0036] With the development of information technology, the amount of data generated by individuals and organizations has increased dramatically, leading to a large amount of duplicate or redundant data. Storage devices have limited cost and space. Duplicate data occupies valuable storage resources and increases storage costs. Therefore, data deduplication has become particularly important; removing duplicate data can save storage space, optimize the use of storage resources, and reduce hardware costs.

[0037] However, traditional deduplication methods often require a large amount of manual work, resulting in slow processing speeds, especially on large-scale datasets. Furthermore, manual comparison and identification of duplicate data are susceptible to fatigue and subjective judgment, leading to errors and omissions. Secondly, as data volumes increase, traditional deduplication methods struggle to scale to meet ever-growing processing demands.

[0038] Therefore, a deep learning-based data deduplication management method is desired. This method acquires image information from storage space, preprocesses the image information to obtain preprocessed image information, and uses deep learning-based image analysis and processing algorithms to extract salient features and perform semantic association on the preprocessed image information to identify duplicate images in storage space. These duplicate images are then fed back to the user. This approach frees up storage space and improves the utilization rate of storage resources by identifying and deleting duplicate images. Simultaneously, the automated deduplication process reduces the workload of manual checking and deleting of duplicate files, improving the efficiency of data management.

[0039] Figure 1 This is a flowchart of a deep learning-based data deduplication management method according to an embodiment of this application. Figure 2 This is a schematic diagram of the architecture of a deep learning-based data deduplication management method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the data deduplication management method based on deep learning according to an embodiment of this application includes:

[0040] S110, acquire image information from storage space; S120, preprocess the image information to obtain preprocessed image information; S130, input the preprocessed image information into the image salient feature capture module to obtain a sequence of image salient feature maps; S140, perform image salient enhancement and image salient semantic association on each image salient feature map in the sequence of image salient feature maps to obtain a sequence of image salient enhancement feature vectors and a sequence of image salient semantic association feature vectors; S150, fuse the sequence of image salient enhancement feature vectors and the sequence of image salient semantic association feature vectors to obtain a sequence of image salient feature semantic fusion representation vectors; S160, calculate the similarity information between any two image salient feature semantic fusion representation vectors in the sequence of image salient feature semantic fusion representation vectors to obtain image similarity information; S170, based on the image similarity information, obtain duplicate images in storage space and feed them back to the user.

[0041] In step S110, image information in the storage space is obtained. It should be understood that the image information in the storage space refers to a collection of image files stored on hard drives, solid-state drives, servers, cloud storage services, or other storage media. These images can be photographs, scanned documents, graphic design works, digital paintings, etc. Based on this, in the technical solution of this application, obtaining image information in the storage space and analyzing and processing it can accurately identify and delete duplicate image files, freeing up storage space. This can provide users with more efficient, secure, and personalized services, while also helping to improve the automation level and accuracy of data management. Specifically, image information in the storage space can be collected and obtained in various ways, depending on the type of storage medium, operating system, and the required level of automation. The following are some common ways to collect and obtain image information: using file management tools provided by the operating system or writing scripts (such as using Python's os module) to traverse a specified directory and its subdirectories to collect image files; manually browsing and selecting image files using graphical interface tools such as File Explorer (Windows) or Finder (Mac OS); and obtaining image information through API calls if the images are stored in cloud services or online storage. These methods can effectively collect and acquire image information from storage space, providing a data foundation for further processing and analysis.

[0042] In step S120, the image information is preprocessed to obtain preprocessed image information. Accordingly, considering that the image information contains different image sizes and complexities, in order to unify images from different sources and formats into a consistent format and size for easier subsequent processing and comparison, the technical solution of this application preprocesses the image information to obtain preprocessed image information. That is, by preprocessing the image information, the images in the storage space can be uniformly formatted, and through preprocessing techniques such as noise reduction, contrast enhancement, cropping, and rotation, the visual effect of the image can be improved, while highlighting key features in the image, thus facilitating subsequent feature extraction and recognition.

[0043] Specifically, in one feasible embodiment of this application, preprocessing the image information to obtain preprocessed image information can be achieved in the following way: First, adjust the image size using image processing software or libraries (such as Python's Pillow library). Next, remove unwanted parts from the image edges, retaining key content. Then, adjust the image orientation to ensure the correct orientation and adjust the image's brightness, contrast, saturation, and hue to improve the visual effect. Then, use filters to remove noise from the image, such as Gaussian blur, median filtering, etc., and enhance the image edges to improve clarity. This yields a preprocessed image, providing more accurate input for subsequent image feature extraction.

[0044] In step S130, the preprocessed image information is input into the image salient feature capture module to obtain a sequence of image salient feature maps. Specifically, in this embodiment, inputting the preprocessed image information into the image salient feature capture module to obtain a sequence of image salient feature maps includes: inputting the preprocessed image information into the image salient feature capture module based on a saliency network detector to obtain a sequence of image salient feature maps. It should be understood that, considering that each image in the preprocessed image information contains salient feature information of the entire image, this salient feature information plays an important role in identifying duplicate images. Based on this, in order to extract and capture the salient feature information contained in each image from the preprocessed image information, in the technical solution of this application, the preprocessed image information is input into the image salient feature capture module based on a saliency network detector to obtain a sequence of image salient feature maps. It is worth mentioning that the saliency network detector can identify the region in the image that is most attractive to the observer and extract the key content of the image. Secondly, the image salient feature capture module based on the salientity network detector can enhance the visual features in the image and reduce interference from the background or irrelevant areas, making subsequent image analysis and processing more accurate.

[0045] In step S140, each of the image saliency feature maps in the sequence of image saliency feature maps is subjected to image saliency enhancement and image saliency semantic association to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors.

[0046] Figure 3 This is a flowchart illustrating the process of performing image saliency enhancement and image saliency semantic association on each of the image saliency feature maps in the sequence of image saliency feature maps according to the data deduplication management method based on deep learning in this application, to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors.

[0047] Specifically, in the embodiments of this application, such as Figure 3 As shown, each of the image saliency feature maps in the sequence of image saliency feature maps undergoes image saliency enhancement and image saliency semantic association to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors, including:

[0048] S210, the sequence of the image saliency feature maps is input into an image saliency enhancement model based on a bidirectional attention mechanism to obtain a sequence of image saliency enhancement feature vectors.

[0049] S220, the sequence of the image saliency feature map is input into the image saliency semantic association extraction model based on the ViT model to obtain the sequence of image saliency semantic association feature vectors.

[0050] Specifically, in step S210, the sequence of the image saliency feature maps is input into an image saliency enhancement model based on a bidirectional attention mechanism to obtain a sequence of image saliency enhancement feature vectors. Correspondingly, considering that each image saliency feature map in the sequence expresses a key salient part of the image, in order to further improve the expressive power and recognition effect of salient features in the image, and to enhance the feature representation ability of salient regions in the image, in the technical solution of this application, the sequence of the image saliency feature maps is input into an image saliency enhancement model based on a bidirectional attention mechanism to obtain a sequence of image saliency enhancement feature vectors. Specifically, the bidirectional attention mechanism can enhance the features of salient regions in the image, making these regions more prominent in subsequent processing. That is, by focusing on the feature information of salient regions, the model can dynamically adjust the focus according to the image content, reduce the processing of non-salient regions, make the image clearer and easier to understand, and obtain a sequence of image saliency enhancement feature vectors containing rich salient features.

[0051] Specifically, in step S220, the sequence of the image saliency feature maps is input into an image saliency semantic association extraction model based on the ViT model to obtain a sequence of image saliency semantic association feature vectors. It should be understood that semantic relationships exist between local regions of the salient parts in each image saliency feature map within the sequence. The ViT model can deeply understand the semantic content of the image and extract richer semantic features. That is, ViT captures the global features of the entire image through a self-attention mechanism, enhancing the contextual relevance of the features. Therefore, in the technical solution of this application, the sequence of the image saliency feature maps is input into an image saliency semantic association extraction model based on the ViT model to capture and extract the semantic relevance between the contexts of local regions in each image saliency feature map, thereby obtaining a sequence of image saliency semantic association feature vectors.

[0052] In step S150, the sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors are fused to obtain a sequence of image saliency feature semantic fusion representation vectors. Specifically, in this embodiment, fusing the sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors to obtain a sequence of image saliency feature semantic fusion representation vectors includes: fusing each corresponding group of image saliency enhancement feature vectors and image saliency semantic association feature vectors from the sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors respectively to obtain a sequence of image saliency feature semantic fusion representation vectors. Accordingly, considering that each corresponding group of image saliency enhancement feature vectors and image saliency semantic association feature vectors respectively expresses the feature information of the region that is most visually attractive in an image and the correlation between different objects in the image, such as the relationship between people and objects in a scene. Therefore, in order to more completely and comprehensively represent the salient feature information and semantic association information expressed by each image, the technical solution of this application fuses each corresponding group of image salient enhancement feature vectors and image salient semantic association feature vectors from the sequence of image salient enhancement feature vectors and the sequence of image salient semantic association feature vectors to obtain a sequence of image salient feature semantic fusion representation vectors. This allows each image to combine salient enhancement features and semantic association features, providing a more comprehensive image feature representation, thereby improving the accuracy of object recognition in images, especially in complex scenes.

[0053] In step S160, the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors is calculated to obtain image similarity information. Specifically, in this embodiment, calculating the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information includes: calculating the cosine similarity between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain an image cosine similarity vector as the image similarity information. Further, to more precisely measure the similarity information between each pair of images so that duplicate images in the storage space can be more accurately identified subsequently, in the technical solution of this application, the cosine similarity between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors is calculated to obtain an image cosine similarity vector as the image similarity information. That is to say, cosine similarity is a widely used method for measuring the similarity of two vectors in direction, not just magnitude. By comparing the similarity of semantic fusion representation vectors of salient features of any two images, duplicate or very similar images in storage can be identified, which helps with deduplication and data cleaning.

[0054] In step S170, based on the image similarity information, duplicate images in the storage space are obtained and fed back to the user.

[0055] Figure 4 This is a flowchart illustrating how a deep learning-based data deduplication management method, according to an embodiment of this application, obtains duplicate images in a storage space based on image similarity information and feeds them back to the user.

[0056] Specifically, in the embodiments of this application, such as Figure 4 As shown, based on the image similarity information, duplicate images in the storage space are obtained and fed back to the user, including:

[0057] S310, apply an acceptable group distribution constraint based on intrinsic latent association descriptors to the image cosine similarity vector to obtain an optimized image cosine similarity vector; S320, input the optimized image cosine similarity vector into a generator-based image duplication discriminator to obtain duplicate images in the storage space; S330, provide the duplicate images in the storage space to the user.

[0058] Specifically, in step S310, the image cosine similarity vector is subjected to a population distribution acceptable constraint based on intrinsic latent association descriptors to obtain an optimized image cosine similarity vector. In particular, in the above technical solution, a neural network model is applied to multiple key stages to achieve salient feature capture, enhancement, semantic association extraction, and duplicate image discrimination of image information. Neural network models play a crucial role in feature mining because they can automatically learn complex feature representations from large amounts of data. However, to further improve the model's performance and robustness, it is necessary to consider the intrinsic latent association between the network output features and the network parameter matrix. By considering the intrinsic latent association between the network output features and the network parameter matrix, the prior information of the neural network model can be used to modulate the feature manifold of the input feature vector to reduce the possibility of special instance perturbations in its feature values. Based on this, in one embodiment of this application, the image cosine similarity vector is subjected to a population distribution acceptable constraint based on intrinsic latent association descriptors to obtain an optimized image cosine similarity vector.

[0059] Specifically, in this embodiment, the optimization of the image cosine similarity vector by applying a population distribution acceptable constraint based on the intrinsic latent association descriptor includes: calculating the mean and variance of the image cosine similarity vector; calculating the difference between the feature value at a predetermined position of the image cosine similarity vector and the mean of the image cosine similarity vector to obtain a first intermediate calculated value; multiplying the network parameter matrix by the image cosine similarity vector and then subtracting it from the image cosine similarity vector positionally to obtain a first intermediate vector, and calculating the Frobenius norm of the first intermediate vector to obtain a first norm value; adding the first norm value to the image... The variance of the cosine similarity vector is squared to obtain a second intermediate value; the first intermediate value is divided by the second intermediate value to obtain a third intermediate value; the squares of each position in the image cosine similarity vector are calculated and then summed to obtain a first sum; the first sum is squared and divided by the scale of the network parameter matrix to calculate the natural exponential function value to obtain a fourth intermediate value, wherein the scale of the network parameter matrix is ​​equal to the product of the width and height of the network parameter matrix; the fourth intermediate value is multiplied by the weighted hyperparameter and then added to the third intermediate value to obtain the feature value at the predetermined position of the optimized image cosine similarity vector.

[0060] In this embodiment of the application, specifically, the image cosine similarity vector is optimized by applying an acceptable population distribution constraint based on intrinsic latent association descriptors to the following optimization formula:

[0061]

[0062] Where V is the image cosine similarity vector, M is the network parameter matrix, β represents the weighted hyperparameter, and v i Let be the eigenvalue of the i-th element in the image cosine similarity vector, N be the scale of the network parameter matrix (equal to the product of its width and height), and L be the length of the image cosine similarity vector. F Let σ represent the Frobenius norm of the image cosine similarity vector, μ represent the mean of the feature set consisting of all eigenvalues ​​of the image cosine similarity vector, and σ represent the mean of the feature set consisting of all eigenvalues ​​of the image cosine similarity vector. 2 v' represents the variance of the feature set consisting of all eigenvalues ​​of the image cosine similarity vector. i Let be the feature value of the i-th element in the optimized image cosine similarity vector.

[0063] In other words, in the above embodiments, the image cosine similarity vector is subject to a population distribution acceptance constraint based on an intrinsic latent association descriptor. First, the image cosine similarity vector is mapped to the high-dimensional feature space of the network parameter matrix. The compatibility perception factor between the mapping result and the original input is used as the intrinsic latent association descriptor between the network output features and the network parameter matrix. Then, the statistical features of the feature groups of the image cosine similarity vector are used as the inherent modal information of the image cosine similarity vector, and the intrinsic latent association descriptor is used as an instance perturbation differential adjustment variable to dynamically standardize the attribute values ​​at each position of the image cosine similarity vector. Simultaneously, the class-standardized modulus data of the image cosine similarity vector is used as a bias term to constrain the population distribution acceptance range of the image cosine similarity vector. In this way, the influence of abnormal sample interference distribution attributes is suppressed as much as possible, improving the accuracy of the image cosine similarity vector.

[0064] Specifically, in step S320, the optimized image cosine similarity vector is input into a generator-based image duplication detector to obtain duplicate images in the storage space; in step S330, the duplicate images in the storage space are fed back to the user. That is, the duplicate images in the storage space are intelligently obtained through the generation process using the optimized image cosine similarity vector. In this way, by identifying and deleting duplicate images, storage space can be freed up, improving the utilization rate of storage resources. At the same time, the automated deduplication process reduces the workload of manually checking and deleting duplicate files, improving the efficiency of data management.

[0065] In summary, the data deduplication management method based on deep learning, as described in the embodiments of this application, is explained. It obtains image information from storage space, preprocesses the image information to obtain preprocessed image information, and utilizes deep learning-based image analysis and processing algorithms to extract salient features and perform image semantic association on the preprocessed image information. This identifies duplicate images in the storage space and provides this information to the user. In this way, by identifying and deleting duplicate images, storage space can be freed up, improving the utilization rate of storage resources. Simultaneously, the automated deduplication process reduces the workload of manual checking and deleting of duplicate files, improving the efficiency of data management.

[0066] Figure 5 This is a block diagram of a deep learning-based data deduplication management device according to an embodiment of this application. Figure 5 As shown, the deep learning-based data deduplication management device 100 according to an embodiment of this application includes:

[0067] The storage space image information acquisition module 110 is used to acquire image information in the storage space; the image information preprocessing module 120 is used to preprocess the image information to obtain preprocessed image information; the image saliency feature extraction module 130 is used to input the preprocessed image information into the image saliency feature capture module to obtain a sequence of image saliency feature maps; the image saliency feature enhancement semantic association module 140 is used to perform image saliency enhancement and image saliency semantic association on each image saliency feature map in the sequence of image saliency feature maps to obtain a sequence of image saliency enhancement feature vectors. The system includes: a sequence of image saliency semantic association feature vectors; an image feature fusion module 150, used to fuse the sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors to obtain a sequence of image saliency feature semantic fusion representation vectors; an image similarity calculation module 160, used to calculate the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information; and a duplicate image acquisition module 170, used to obtain duplicate images in the storage space based on the image similarity information and provide feedback to the user.

[0068] Here, those skilled in the art will understand that the specific operations of each step in the aforementioned deep learning-based data deduplication management device have been referenced above. Figures 1 to 4 The description of the deep learning-based data deduplication management method is detailed here, and therefore, its repeated description will be omitted.

[0069] Figure 6 A block diagram of an electronic device according to an embodiment of this application, such as Figure 6 As shown, it includes: at least one processor 601; a memory 602 communicatively connected to the at least one processor; wherein the memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 to perform the above method.

[0070] The memory 602 and processor 601 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 601 and memory 602 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other systems over a transmission medium. Information processed by processor 601 is transmitted over a wireless medium via an antenna, which further receives information and transmits it to processor 601.

[0071] Processor 601 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 602 can be used to store information used by the processor during operation.

[0072] Embodiments of the present invention also relate to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0073] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0074] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data deduplication management method based on deep learning, characterized in that, include: Retrieve image information from storage space; The image information is preprocessed to obtain preprocessed image information; The preprocessed image information is input into the image salient feature capture module to obtain a sequence of image salient feature maps; For each of the image saliency feature maps in the sequence of image saliency feature maps, image saliency enhancement and image saliency semantic association are performed respectively to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors; The sequence of the image saliency enhancement feature vector and the sequence of the image saliency semantic association feature vector are fused to obtain a sequence of image saliency feature semantic fusion representation vectors; Calculate the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information; Based on the image similarity information, duplicate images in the storage space are obtained and fed back to the user.

2. The data deduplication management method based on deep learning according to claim 1, characterized in that, The preprocessed image information is input into the image salient feature capture module to obtain a sequence of image salient feature maps, including: The preprocessed image information is input into the image salient feature capture module based on the salientity network detector to obtain a sequence of image salient feature maps.

3. The data deduplication management method based on deep learning according to claim 2, characterized in that, For each image saliency feature map in the sequence of image saliency feature maps, image saliency enhancement and image saliency semantic association are performed respectively to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors, including: The sequence of the image saliency feature maps is input into an image saliency enhancement model based on a bidirectional attention mechanism to obtain a sequence of image saliency enhancement feature vectors. The sequence of the image saliency feature maps is input into the image saliency semantic association extraction model based on the ViT model to obtain the sequence of image saliency semantic association feature vectors.

4. The data deduplication management method based on deep learning according to claim 3, characterized in that, The sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors are fused to obtain a sequence of image saliency feature semantic fusion representation vectors, including: The corresponding image saliency enhancement feature vectors and image saliency semantic association feature vectors in each group are fused from the sequence of image saliency enhancement feature vectors and the sequence of image saliency semantic association feature vectors to obtain a sequence of image saliency feature semantic fusion representation vectors.

5. The data deduplication management method based on deep learning according to claim 4, characterized in that, Calculating the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information includes: Calculate the cosine similarity between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain the image cosine similarity vector as the image similarity information.

6. The data deduplication management method based on deep learning according to claim 5, characterized in that, Based on the image similarity information, duplicate images in the storage space are obtained and fed back to the user, including: An optimized image cosine similarity vector is obtained by applying a population distribution acceptable constraint based on intrinsic latent association descriptors to the image cosine similarity vector. The optimized image cosine similarity vector is input into a generator-based image duplication detector to obtain duplicate images in the storage space; The system will send duplicate images from the storage space back to the user.

7. The data deduplication management method based on deep learning according to claim 6, characterized in that, To obtain an optimized image cosine similarity vector, the following steps are taken: The image cosine similarity vector is subjected to an acceptable population distribution constraint based on intrinsic latent association descriptors. Calculate the mean and variance of the cosine similarity vector of the images; The difference between the feature value at a predetermined position of the image cosine similarity vector and the mean of the image cosine similarity vector is calculated to obtain a first intermediate calculated value; The network parameter matrix is ​​multiplied by the image cosine similarity vector and then subtracted from the image cosine similarity vector by position to obtain the first intermediate vector. The Frobenius norm of the first intermediate vector is calculated to obtain the first norm value. Add the variance of the image cosine similarity vector to the first norm value and then take the square root to obtain the second intermediate calculated value; Divide the first intermediate calculated value by the second intermediate calculated value to obtain the third intermediate calculated value; The squares of each position in the cosine similarity vector of the image are calculated and then summed to obtain the first total value; The natural exponential function value is calculated by taking the square root of the first sum and dividing it by the scale of the network parameter matrix to obtain the fourth intermediate value, wherein the scale of the network parameter matrix is ​​equal to the product of the width and height of the network parameter matrix. The fourth intermediate calculated value is multiplied by the weighted hyperparameter and then added to the third intermediate calculated value to obtain the feature value at the predetermined position of the optimized image cosine similarity vector.

8. A data deduplication management device based on deep learning, characterized in that, include: The image information acquisition module for storage space is used to acquire image information in the storage space; The image information preprocessing module is used to preprocess the image information to obtain preprocessed image information; The image saliency feature extraction module is used to input the preprocessed image information into the image saliency feature capture module to obtain a sequence of image saliency feature maps; The image saliency feature enhancement and semantic association module is used to perform image saliency enhancement and image saliency semantic association on each image saliency feature map in the sequence of image saliency feature maps to obtain a sequence of image saliency enhancement feature vectors and a sequence of image saliency semantic association feature vectors; The image feature fusion module is used to fuse the sequence of the image saliency enhancement feature vector and the sequence of the image saliency semantic association feature vector to obtain the sequence of image saliency feature semantic fusion representation vector; The image similarity calculation module is used to calculate the similarity information between any two image saliency feature semantic fusion representation vectors in the sequence of image saliency feature semantic fusion representation vectors to obtain image similarity information; The duplicate image acquisition module is used to obtain duplicate images in the storage space based on the image similarity information and then provide feedback to the user.

9. The data deduplication management device based on deep learning according to claim 8, characterized in that, The image saliency feature extraction module is used for: The preprocessed image information is input into the image salient feature capture module based on the salientity network detector to obtain a sequence of image salient feature maps.

10. An electronic device, comprising: Processor and memory, The memory stores a computer-executable program, which, when executed by the processor, implements the data deduplication management method based on deep learning as described in any one of claims 1 to 7.

Images