Image deduplication method, terminal device, and computer storage medium

Abstract

The application discloses an image deduplication method, a terminal device and a computer storage medium. The image deduplication method comprises the following steps: acquiring a first image set; reading a plurality of second image sets according to an image index table; traversing each second image set, and performing a first deduplication operation on the first image set to obtain a third image set formed after deduplication of the first image set; the first deduplication operation comprises the following steps: traversing each second image of the second image set, and performing an image set update operation on the first image set by using each second image; the image set update operation comprises the following steps: acquiring a first similarity between the second image and all first images in the first image set, and deleting a first image corresponding to the first similarity higher than a preset threshold from the first image set when the first similarity is higher than the preset threshold, so as to update the first image set. The image deduplication method can improve the efficiency of image deduplication through a segmentation task, reduce the matching times of each deduplication, and realize the processing of a large number of images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image deduplication method, a terminal device, and a computer storage medium. Background Technology

[0002] Image databases often contain a large number of identical or similar images, resulting in wasted storage resources. Displaying too many duplicate images in a presentation also negatively impacts the user experience. For some data-driven algorithms (such as neural networks), excessive duplicate images increase training time and can bias the final training results. Traditional image deduplication methods involve pairwise similarity matching of images to be deduplicated; if two images are deemed similar, one is deleted. This process is time-consuming, and its time and memory usage increase significantly with the size of the data. Summary of the Invention

[0003] This application provides an image deduplication method, a terminal device, and a computer storage medium.

[0004] One technical solution adopted in this application is to provide an image deduplication method, the image deduplication method comprising:

[0005] Obtain the first image set;

[0006] Several second image sets are read according to the image index table, wherein each second image in the image index table exists in one of the second image sets;

[0007] Traverse each second image set and perform a first deduplication operation on the first image set to obtain a third image set formed by deduplicating the first image set;

[0008] The first deduplication operation includes:

[0009] Iterate through each second image in the second image set, and perform an image set update operation on the first image set using each second image; the image set update operation includes:

[0010] Obtain the first similarity between the second image and all first images in the first image set. If there is a first similarity higher than a preset threshold, delete the first image corresponding to the first similarity higher than the preset threshold from the first image set to update the first image set.

[0011] The acquisition of the first image set includes:

[0012] The first image set is divided into several first image subsets, and any first image in each first image subset does not exist in any other first image subset;

[0013] Each pair of first image subsets is determined as a group of image subsets. A second deduplication operation is performed on each group of image subsets to obtain a third image subset formed by each group of image subsets. The similarity between any two first images contained in each third image subset is lower than a second preset threshold.

[0014] The second deduplication operation includes:

[0015] Select the first target image of the first target image subset in the image subset, and perform an image subset update operation on the first target image subset;

[0016] The image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset of the image subset, deleting first target images whose second similarity is higher than the preset threshold from the first target image subset, and updating the first target image subset;

[0017] Continue to select the first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed.

[0018] The image deduplication method further includes, after obtaining each set of image subsets and forming the third image subset, the method also includes:

[0019] Obtain the number of the third image subset;

[0020] When the number of the third image subset is 1, output the third image subset;

[0021] When the number of the third image subsets is greater than 1, every two third image subsets are determined as a group of image subsets, and the second deduplication operation is performed on each group of image subsets until the number of the third image subsets formed by merging each group of image subsets is 1.

[0022] Wherein, after dividing the first image set into several first image subsets, the image deduplication method further includes:

[0023] Perform a third deduplication operation on each first image subset to update each first image subset;

[0024] The third deduplication operation includes:

[0025] Select a first sub-image from the first image subset and perform inter-set deduplication;

[0026] The inter-set deduplication operation includes: obtaining the third similarity between the first sub-image and other first sub-images, and deleting the first sub-images whose third similarity is higher than the preset threshold from the first image subset;

[0027] Continue to select one unselected first sub-image from the first image subset and perform inter-set deduplication operation until the selection of first sub-images within the first image subset is completed. The unselected first sub-images include first sub-images within the image subset whose third similarity is lower than the preset threshold.

[0028] The image deduplication method further includes:

[0029] The image index table is updated using the third image set.

[0030] Another technical solution adopted in this application is to provide another image deduplication method. 6. The image deduplication method includes:

[0031] Obtain a first image set, divide the first image set into several first image subsets, and ensure that any first image in each first image subset does not exist in any other first image subset;

[0032] Perform a first deduplication operation on each first image subset to update each first image subset;

[0033] The first deduplication operation includes:

[0034] Select a first sub-image from the first image subset and perform inter-set deduplication;

[0035] The inter-set deduplication operation includes: obtaining the first similarity between the first sub-image and other first sub-images, and deleting the first sub-images with the first similarity higher than the preset threshold from the first image subset;

[0036] Continue to select a first sub-image that has not been selected in the first image subset and perform inter-set deduplication operation until the selection of the first sub-images in the first image subset is completed. The unselected first sub-images include the first sub-images in the image subset whose first similarity is lower than the preset threshold.

[0037] Wherein, after performing the first deduplication operation on each first image subset to update each first image subset, the image deduplication method further includes:

[0038] Each pair of first image subsets is determined as a group of image subsets. A second deduplication operation is performed on each group of image subsets to obtain a third image subset formed by each group of image subsets. The similarity between any two first images contained in each third image subset is lower than a second preset threshold.

[0039] The second deduplication operation includes:

[0040] Select the first target image of the first target image subset in the image subset, and perform an image subset update operation on the first target image subset;

[0041] The image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset of the image subset, deleting first target images whose second similarity is higher than the preset threshold from the first target image subset, and updating the first target image subset;

[0042] Continue to select the first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed.

[0043] The image deduplication method further includes, after obtaining each set of image subsets and forming the third image subset, the method also includes:

[0044] Several second image sets are read according to the image index table, wherein each second image in the image index table exists in one of the second image sets;

[0045] Traverse each second image set and perform a third deduplication operation on the third image subset to obtain the third image set formed after deduplication of the third image subset;

[0046] The third deduplication operation includes:

[0047] Iterate through each second image in the second image set, and perform an image set update operation on the third image subset using each second image; the image set update operation includes:

[0048] Obtain the third similarity between the second image and all first images in the third image subset. If there is a third similarity higher than a preset threshold, delete the first image corresponding to the third similarity higher than the preset threshold from the third image subset to update the third image subset.

[0049] Another technical solution adopted in this application is to provide a terminal device, the terminal device including a memory and a processor coupled to the memory;

[0050] The memory is used to store program data, and the processor is used to execute the program data to implement the image deduplication method described above.

[0051] Another technical solution adopted in this application is to provide a computer storage medium for storing program data, which, when executed by a computer, is used to implement the image deduplication method described above.

[0052] The beneficial effects of this application are as follows: The terminal device reads several second image sets according to an image index table, wherein each second image in the image index table exists in one of the second image sets; it iterates through each second image set and performs a first deduplication operation on the first image set to obtain a third image set formed after deduplication of the first image set; the first deduplication operation includes: iterating through each second image in the second image set and performing an image set update operation on the first image set using each second image; the image set update operation includes: obtaining the first similarity between the second image and all first images in the first image set; when there is a first similarity higher than a preset threshold, deleting the first image corresponding to the first similarity higher than the preset threshold from the first image set to update the first image set. The image deduplication method of this application can improve the efficiency of image deduplication through segmentation tasks, reduce the number of matching operations for each deduplication, and realize the processing of massive images. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the structure of an embodiment of the image deduplication system provided in this application;

[0055] Figure 2 This is a schematic diagram of another embodiment of the image deduplication system provided in this application;

[0056] Figure 3 This is a schematic diagram of the data reading module provided in this application;

[0057] Figure 4 This is a flowchart illustrating an embodiment of the image deduplication method provided in this application;

[0058] Figure 5 This is a schematic diagram illustrating the working principle of the image deduplication method provided in this application;

[0059] Figure 6 yes Figure 4 The diagram shows the specific process flow of step S31 in the image deduplication method.

[0060] Figure 7This is a functional diagram of the image set deduplication module and the image set deduplication module provided in this application;

[0061] Figure 8 This is a flowchart illustrating another embodiment of the image deduplication method provided in this application;

[0062] Figure 9 This is a schematic diagram of the structure of an embodiment of the terminal device provided in this application;

[0063] Figure 10 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0065] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of an embodiment of the image deduplication system provided in this application. Figure 2 This is a schematic diagram of another embodiment of the image deduplication system provided in this application.

[0066] in, Figure 1 The image deduplication system 10 shown includes a first data reading module 11, an image processing module 12, and an image set deduplication module 13.

[0067] The first data reading module 11 is used to obtain a first image set from the image database, wherein the first image set includes a first number of first images.

[0068] The image processing module 12 is used to divide the first image set into several first image subsets.

[0069] The image set deduplication module 13 is used to perform the following for each first image subset: take a first image in the first image subset and match it with the remaining first images in the first image subset, delete the first images in the remaining first images with a similarity higher than a preset threshold, until all the first images in the first image subset are traversed to obtain the final second image subset.

[0070] The image set deduplication module 13 is further configured to divide the two second image subsets into a group, and perform the following for each group of second image subsets: take a first image in one second image subset and match it with all first images in another second image subset, delete the first images in the other second image subset whose similarity is higher than the preset threshold, until all first images in a group of second image subsets are traversed, and the group of second image subsets is merged to form a third image subset.

[0071] The image set deduplication module 13 is further configured to, when the number of the third image subset is one, use the third image subset as the second image set formed after deduplication of the first image set.

[0072] The image deduplication system 10 also includes a second data reading module 14 and an image set deduplication module 15.

[0073] The second data reading module 14 is used to obtain a third image set from the image database, wherein the third image set includes a second number of second images.

[0074] The image set deduplication module 15 is used to take a second image from the third image subset and match it with all the first images in the second image set, delete the first images in the second image set whose similarity is higher than a preset threshold, until all the first images in the third image subset are traversed to obtain the fourth image set formed after the second image set is deduplicated.

[0075] The first data reading module 11 is also used to acquire the sixth image set to be stored in the image database.

[0076] Specifically, such as Figure 2 As shown, the image deduplication system 20 of this application includes an image database 21, a first data reading module 22, an image set preprocessing module 23, an image set intra-deduplication module 24, an image set inter-deduplication module 25, a second data reading module 26, and a post-processing module 27.

[0077] Among them, the image database 21 is a database used to store image data. The image data stored in the image database 21 can be in the following forms: raw image data; image feature data extracted according to corresponding algorithms (such as neural networks, image filtering, etc.); image resource locators (URLs, uniform resource locators), etc.

[0078] like Figure 3As shown, the first data reading module 22 and the second data reading module 26 contain an image index table. The image index table stores the image's identity document (ID), which can be: the image's serial number, URL, storage path, etc. The corresponding image data can be found and read from the image database based on the image's ID. The image index table is provided by the user according to the specific scenario.

[0079] The first data reading module 22 and the second data reading module 26 are used to read data from the image database 21 according to the image IDs in the image index table. Due to limitations in real-world computing resources (such as memory and bandwidth), it is not possible to read and process all data in the database at once. Therefore, the first data reading module 22 and the second data reading module 26 read one image set from their image index table at a time (processing images in one image set at a time) and input it to the next module. It is specified that the first data reading module 22 reads B1 images at a time, and the second data reading module 26 reads B2 images at a time. It should be noted that the image IDs in the image index table are not read repeatedly (each image ID is read only once).

[0080] The image set preprocessing module 23 is used to receive the image set read by the first data reading module 22. It preprocesses the images in the image set according to the requirements of the corresponding deduplication algorithm, including image scaling, grayscale conversion, image encoding, image transformation, and feature extraction.

[0081] The image set deduplication module 24 is used to deduplicat images within an image set obtained from the image set preprocessing module 23, resulting in multiple image subsets, where the images in each subset are dissimilar. The image set deduplication module 24 deduplicates images within an input image set, filtering out dissimilar images and deleting similar images.

[0082] The image set deduplication module 25 is used to match and deduplicate all images in the image index table of the second data reading module 26 with the image set obtained by the image set deduplication module 24 within the image set. The image set read from the second data reading module 26 is then used for inter-set image deduplication with the image set obtained by the aforementioned image set deduplication module 24 within the image set.

[0083] The post-processing module 27 is used to obtain the image index table from the second data reading module 26, and perform corresponding operations on the images according to the image IDs in the image index table. These operations include, but are not limited to, deleting images from the image database whose image IDs are not in the image index table, reading images from the image index table and writing them into the image database 21, etc.

[0084] The following is combined Figures 1 to 3The image deduplication system shown further describes the image deduplication method applied to this system; please refer to [link / reference]. Figure 4 and Figure 5 , Figure 4 This is a flowchart illustrating an embodiment of the image deduplication method provided in this application. Figure 5 This is a schematic diagram illustrating the working principle of the image deduplication method provided in this application.

[0085] like Figure 4 As shown, the image deduplication method in this application embodiment may specifically include the following steps:

[0086] Step S31: Obtain the first image set.

[0087] In this embodiment of the application, a first image set is obtained from an image database, wherein the first image set includes a first number of first images.

[0088] In this embodiment, several modules of the image deduplication system are initialized. The structure of the image deduplication system can be found in [reference needed]. Figures 1 to 3 This will not be elaborated upon here.

[0089] Initialize the image index tables in the first data reading module and the second data reading module, set the number of images in the image set read by the first data reading module each time to B1 (the first quantity), and the number of images in the image set read by the second data reading module each time to B2. Set the number of subset groups N in the deduplication module within the image set, and select or define a deduplication algorithm.

[0090] It should be noted that parameters B1, B2, and N can be manually set according to the actual available computing resources of the system, which can effectively improve the deduplication efficiency of the image deduplication system.

[0091] Specifically, if there are unread images in the image index table of the first data reading module, the first data reading module reads an image set 0 containing B1 images from the image database. If there are no unread images in the image index table of the first data reading module, the second data reading module reads the images according to the image index table and performs post-processing operations.

[0092] For details on how to obtain the first image set, please refer to [link / reference]. Figure 6 , Figure 6 yes Figure 4 The diagram shows the specific process flow of step S31 in the image deduplication method.

[0093] like Figure 6 As shown, the image deduplication method in this application embodiment may specifically include the following steps:

[0094] Step S311: Divide the first image set into several first image subsets, wherein any first image in each first image subset does not exist in any other first image subset.

[0095] In this embodiment of the application, before performing inter-image deduplication on the first image set, the terminal device may first perform intra-image deduplication on the first image set. The specific process is as follows:

[0096] The terminal device performs a third deduplication operation on each first image subset to update each first image subset; the third deduplication operation includes:

[0097] Select a first sub-image from the first image subset and perform inter-set deduplication operation; the inter-set deduplication operation includes: obtaining the third similarity between the first sub-image and other first sub-images, deleting the first sub-images whose third similarity is higher than the preset threshold from the first image subset; continuing to select a first sub-image that has not been selected from the first image subset and perform inter-set deduplication operation until the selection of first sub-images in the first image subset is completed, wherein the unselected first sub-images include first sub-images in the image subset whose third similarity is lower than the preset threshold.

[0098] Specifically, after image set 0 undergoes image preprocessing by the image set preprocessing module, it is input into the image set deduplication module to perform the image set deduplication process.

[0099] For details, please continue reading Figure 7 , Figure 7 This is a functional diagram illustrating the intra-image set deduplication module and the inter-image set deduplication module provided in this application. The intra-image set deduplication module divides image set 0 into N mutually exclusive image subsets, as shown below. Figure 7 Subsets 1 to N in the set.

[0100] It should be noted that when N=1, image set 0 is not partitioned.

[0101] In one possible embodiment, the number N of the image subsets can be set according to the number of computing units, that is, the number of subsets can be set according to the amount of parallel processing that the image deduplication system can perform, thereby maximizing the processing capacity of the image deduplication system.

[0102] In this embodiment of the application, the image deduplication module within the image set performs image deduplication on each image subset to obtain N deduplicated image subsets.

[0103] by Figure 7Taking image deduplication in image subset 1 as an example, the image deduplication module within the image set extracts image 1 from image subset 1 and performs pairwise matching with all other images in image subset 1. If there is an image x whose similarity to image 1 is higher than a preset threshold, then image x is deleted. Then, the image deduplication module extracts image 2 from image subset 1 and performs pairwise matching with all other images in image subset 1 except for images 1 and 2. If there is an image y whose similarity to image 2 is higher than a preset threshold, then image y is deleted. The above steps are repeated until all duplicate images in image subset 1 are deleted, resulting in image subset 1'.

[0104] It should be noted that the specific deduplication algorithm is selected or defined by the user, including but not limited to neural network-based deduplication algorithms, hash-based image deduplication algorithms, and image deduplication algorithms based on structural similarity (SSIM). To improve deduplication efficiency, the deduplication of N subsets of images is performed in parallel on N computing units (CPU, central processing unit, GPU, graphics processing unit, FPGA, FieldProgrammableGate Array, embedded devices, etc.).

[0105] Since most computing units have additional optimizations for matrix calculations, matrix operations can be used to remove duplicate images within an image set, replacing pairwise cyclic matching, to further improve deduplication efficiency. For example, if image subset 1 contains m images, a similarity score matrix M of shape m×m can be obtained according to the deduplication algorithm, where the i-th row and j-th column of M represents the similarity score between image i and image j in image subset 1. Duplicate images are then removed based on the similarity score.

[0106] The image deduplication module within the image set traverses image subsets 1 to N, resulting in image subsets 1' to N'.

[0107] Step S312: Determine every two first image subsets as a group of image subsets, perform a second deduplication operation on each group of image subsets, obtain the third image subset formed by each group of image subsets, and the similarity between any two first images contained in each third image subset is lower than a second preset threshold.

[0108] In this embodiment of the application, the second deduplication operation specifically includes:

[0109] Select a first target image from the first target image subset in the image subset, and perform an image subset update operation on the first target image subset; the image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset in the image subset, delete first target images whose second similarity is higher than the preset threshold from the first target image subset, and update the first target image subset; continue to select first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed.

[0110] Specifically, the image deduplication module within the image set divides the N image subsets obtained in step S311 into pairs and performs image deduplication between the two image subsets in the same group.

[0111] Specifically, the image deduplication module within the image set extracts image 1 from image subset 1' and matches it one by one with all images in image subset 2': if there is an image x in image subset 2' that is similar to image 1, then image x is deleted. The above matching process is repeated for all images in image subset 1', and all images in image subset 2' that are similar to image subset 1' are deleted.

[0112] The intra-image deduplication module further merges the images retained in image subset 2' with all images in image subset 1' to obtain a new image subset 1'2'. After inter-image subset deduplication for all groups, the following is obtained: A new subset, in which The symbol represents the rounding up sign. The new image subset is also divided into pairs, and the deduplication process between image sets is performed on each group to obtain the following results. Create a new subset of images; repeat this process until all the image subsets are combined into a single image set 1.

[0113] In the embodiments of this application, image deduplication between each group of image sets can be performed in parallel by N computing units, which effectively improves the image deduplication efficiency of the image deduplication system.

[0114] Step S32: Read several second image sets according to the image index table, wherein each second image in the image index table exists in one of the second image sets.

[0115] like Figure 7 As shown, when there are unread images in the image index table of the second data reading module, the second data reading module reads an image set 2 containing B2 images from the image database and inputs it into the image set deduplication module to perform the image set deduplication process.

[0116] Step S33: Traverse each second image set and perform the first deduplication operation on the first image set to obtain the third image set formed after deduplication of the first image set.

[0117] In this embodiment of the application, the first deduplication operation includes:

[0118] Traverse each second image in the second image set and perform an image set update operation on the first image set using each second image; the image set update operation includes: obtaining the first similarity between the second image and all first images in the first image set; when there is a first similarity higher than a preset threshold, delete the first image corresponding to the first similarity higher than the preset threshold from the first image set to update the first image set.

[0119] Specifically, the image set deduplication module matches and deduplicates the image set 1 obtained by the image set deduplication module with all the images in the image index table of the second data reading module.

[0120] Specifically, the second data reading module reads B2 images at a time, for example, image set 2_1. It then matches image 1 in image set 2_1 with all images in image set 1 one by one: if an image x in image set 1 is similar to image 1, then image x is deleted. The module then iterates through image set 2_1, deleting all images in image set 1 that are similar to images in image set 2_1.

[0121] The above operations are performed sequentially on the image sets (image set 2_2, ..., image set 2_n) read by the second data reading module until the second data reading module has read all image IDs in its image index table, resulting in image set 3 after filtering out duplicate images from image set 1. The IDs of all images in image set 3 are then added to the image index table in the second data reading module to update the image IDs in the image index table.

[0122] Since image set 3, used to update the image index table in the second data reading module, has undergone deduplication both within and between image sets, it has been determined that there are no duplicate images in image set 3. Therefore, image set 3 can be stored in the image index table in the second data reading module for further deduplication of subsequent image sets.

[0123] Then, the post-processing module obtains the image index table from the second data reading module and performs corresponding operations on the images based on the image IDs in the image index table. These operations include, but are not limited to, deleting images from the image database whose image IDs are not in the image index table, reading images from the image index table, and writing them into the image database.

[0124] In summary, the image deduplication system and method of this application can realize two common image deduplication scenarios, including but not limited to: deduplicating existing images in the image database, i.e., scenario 1; and deduplicating images to be stored in the image database, i.e., scenario 2.

[0125] Specifically, for scenario 1, during the initialization phase, the image index table in the first data reading module is initialized with the IDs of all images in the image database, while the image index table in the second data reading module is initialized to be empty. After deduplication by the image deduplication system of this application, the images corresponding to the image IDs finally contained in the image index table of the second data reading module are dissimilar. The difference between this image index table and the image index table in the first data reading module is calculated to obtain all the deleted image IDs. Finally, the corresponding images in the image database are deleted based on the deleted image IDs.

[0126] For scenario 2, the image index table in the first data reading module is initialized with the image IDs to be stored in the image database, and the image index table in the second data reading module is initialized with the IDs of all images in the image database. After deduplication by the image deduplication system of this application, the image index table in the second data reading module finally contains the image IDs from the original database and the deduplicated image IDs to be stored. Finally, the image data corresponding to the image IDs to be stored is read and added to the image database.

[0127] In this embodiment, the terminal device reads several second image sets according to an image index table, wherein each second image in the image index table exists in one of the second image sets; it iterates through each second image set and performs a first deduplication operation on the first image set to obtain a third image set after deduplication of the first image set; the first deduplication operation includes: iterating through each second image in the second image set and performing an image set update operation on the first image set using each second image; the image set update operation includes: obtaining a first similarity between the second image and all first images in the first image set; when there is a first similarity higher than a preset threshold, deleting the first image corresponding to the first similarity higher than the preset threshold from the first image set to update the first image set. The image deduplication method of this application can improve the efficiency of image deduplication through segmentation tasks, reduce the number of matching operations for each deduplication, and realize the processing of massive images.

[0128] The massive data fast deduplication system proposed in this application can process any number of massive image data. It improves deduplication efficiency by segmenting tasks and performing parallel computation. By adjusting the B1, B2, and N values ​​in the module, the system can maximize the use of computing resources under limited conditions, making it suitable for various computing environments. In this system, each retained image is matched with all other retained images, ensuring deduplication accuracy. The system does not restrict the use of specific deduplication algorithms, making it universally applicable. Furthermore, the system does not generate any intermediate products and does not occupy additional storage.

[0129] Please continue reading. Figure 8 , Figure 8 This is a flowchart illustrating another embodiment of the image deduplication method provided in this application.

[0130] like Figure 8 As shown, the image deduplication method in this application embodiment may specifically include the following steps:

[0131] Step S41: Obtain the first image set and divide the first image set into several first image subsets, wherein any first image in each first image subset does not exist in any other first image subset.

[0132] Step S42: Perform a first deduplication operation on each first image subset to update each first image subset.

[0133] The first deduplication operation includes:

[0134] Select a first sub-image from the first image subset and perform inter-set deduplication operation; the inter-set deduplication operation includes: obtaining the first similarity between the first sub-image and other first sub-images, deleting the first sub-images with the first similarity higher than the preset threshold from the first image subset; continuing to select a first sub-image that has not been selected from the first image subset and perform inter-set deduplication operation until the selection of first sub-images in the first image subset is completed, wherein the unselected first sub-images include the first sub-images in the image subset whose first similarity is lower than the preset threshold.

[0135] Furthermore, the terminal device can also determine every two first image subsets as a group of image subsets, perform a second deduplication operation on each group of image subsets, obtain a third image subset formed by each group of image subsets, and the similarity between any two first images contained in each third image subset is lower than a second preset threshold.

[0136] The second deduplication operation includes:

[0137] Select a first target image from the first target image subset in the image subset, and perform an image subset update operation on the first target image subset; the image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset in the image subset, delete first target images whose second similarity is higher than the preset threshold from the first target image subset, and update the first target image subset; continue to select first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed.

[0138] Furthermore, the terminal device can also read several second image sets according to the image index table, wherein each second image in the image index table exists in one of the second image sets; traverse each second image set and perform a third deduplication operation on the third image subset to obtain the third image set formed after deduplication of the third image subset.

[0139] The third deduplication operation includes:

[0140] Traverse each second image in the second image set and perform an image set update operation on the third image subset using each second image; the image set update operation includes: obtaining the third similarity between the second image and all first images in the third image subset; when there is a third similarity higher than a preset threshold, delete the first image corresponding to the third similarity higher than the preset threshold from the third image subset to update the third image subset.

[0141] It should be noted that the first deduplication operation, the second deduplication operation, and the third deduplication operation in the embodiments of this application are respectively related to... Figure 4 The third deduplication operation, the second deduplication operation, and the first deduplication operation of the image deduplication method in the illustrated embodiment are the same, and will not be described again here.

[0142] The above embodiments are merely one common example of this application and do not constitute any limitation on the technical scope of this application. Therefore, any minor modifications, equivalent changes, or alterations made to the above content based on the substance of the solution of this application shall still fall within the scope of the technical solution of this application.

[0143] Please continue reading Figure 9 , Figure 9 This is a schematic diagram of the structure of an embodiment of the terminal device provided in this application. The terminal device 500 of this application embodiment includes a processor 51, a memory 52, an input / output device 53, and a bus 54.

[0144] The processor 51, memory 52, and input / output device 53 are connected to the bus 54. The memory 52 stores program data, and the processor 51 is used to execute the program data to implement the image deduplication method described in the above embodiments.

[0145] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.

[0146] This application also provides a computer storage medium; please refer to the following: Figure 10 , Figure 10 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores program data 61, which is used to implement the image deduplication method of the above embodiment when executed by the processor.

[0147] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image deduplication method, characterized in that, The image deduplication method includes: Obtain the first image set; Several second image sets are read according to the image index table, wherein each second image in the image index table exists in one of the second image sets; Traverse each second image set and perform a first deduplication operation on the first image set to obtain a third image set formed by deduplicating the first image set; The first deduplication operation includes: Iterate through each second image in the second image set, and perform an image set update operation on the first image set using each second image; the image set update operation includes: Obtain the first similarity between the second image and all first images in the first image set. If there is a first similarity higher than a preset threshold, delete the first image corresponding to the first similarity higher than the preset threshold from the first image set to update the first image set. The acquisition of the first image set includes: The first image set is divided into several first image subsets, and any first image in each first image subset does not exist in any other first image subset; Each pair of first image subsets is determined as a group of image subsets. A second deduplication operation is performed on each group of image subsets to obtain a third image subset formed by each group of image subsets. The similarity between any two first images contained in each third image subset is lower than a second preset threshold. The second deduplication operation includes: Select the first target image of the first target image subset in the image subset, and perform an image subset update operation on the first target image subset; The image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset of the image subset, deleting first target images whose second similarity is higher than the preset threshold from the first target image subset, and updating the first target image subset; Continue to select the first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed; After obtaining each set of image subsets and forming the third image subset, the image deduplication method further includes: Obtain the number of the third image subset; When the number of the third image subset is 1, output the third image subset; When the number of the third image subsets is greater than 1, every two third image subsets are determined as a group of image subsets, and the second deduplication operation is performed on each group of image subsets until the number of the third image subsets formed by merging each group of image subsets is 1.

2. The image deduplication method according to claim 1, characterized in that, After dividing the first image set into several first image subsets, the image deduplication method further includes: Perform a third deduplication operation on each first image subset to update each first image subset; The third deduplication operation includes: Select a first sub-image from the first image subset and perform inter-set deduplication; The inter-set deduplication operation includes: obtaining the third similarity between the first sub-image and other first sub-images, and deleting the first sub-images whose third similarity is higher than the preset threshold from the first image subset; Continue to select one unselected first sub-image from the first image subset and perform inter-set deduplication operation until the selection of first sub-images within the first image subset is completed. The unselected first sub-images include first sub-images within the image subset whose third similarity is lower than the preset threshold.

3. The image deduplication method according to claim 1, characterized in that, The image deduplication method further includes: The image index table is updated using the third image set.

4. An image deduplication method, characterized in that, The image deduplication method includes: Obtain a first image set, divide the first image set into several first image subsets, and ensure that any first image in each first image subset does not exist in any other first image subset; Perform a first deduplication operation on each first image subset to update each first image subset; The first deduplication operation includes: Select a first sub-image from the first image subset and perform inter-set deduplication; The inter-set deduplication operation includes: obtaining the first similarity between the first sub-image and other first sub-images, and deleting the first sub-images with the first similarity higher than a preset threshold from the first image subset; Continue to select a first sub-image that has not been selected in the first image subset and perform inter-set deduplication operation until the first sub-images in the first image subset are selected. The unselected first sub-images include the first sub-images in the image subset whose first similarity is lower than the preset threshold. After performing a first deduplication operation on each first image subset to update each first image subset, the image deduplication method further includes: Each pair of first image subsets is determined as a group of image subsets. A second deduplication operation is performed on each group of image subsets to obtain a third image subset formed by each group of image subsets. The similarity between any two first images contained in each third image subset is lower than a second preset threshold. The second deduplication operation includes: Select the first target image of the first target image subset in the image subset, and perform an image subset update operation on the first target image subset; The image subset update operation includes: based on the second similarity between the first target image and all first non-target images in the first non-target image subset of the image subset, deleting first target images whose second similarity is higher than the preset threshold from the first target image subset, and updating the first target image subset; Continue to select the first target images that have not been selected in the image subset and perform the image subset update operation until all first target images in the first target image subset have been traversed.

5. The image deduplication method according to claim 4, characterized in that, After obtaining each set of image subsets and forming the third image subset, the image deduplication method further includes: Several second image sets are read according to the image index table, wherein each second image in the image index table exists in one of the second image sets; Traverse each second image set and perform a third deduplication operation on the third image subset to obtain the third image set formed after deduplication of the third image subset; The third deduplication operation includes: Iterate through each second image in the second image set, and perform an image set update operation on the third image subset using each second image; the image set update operation includes: Obtain the third similarity between the second image and all first images in the third image subset. If there is a third similarity higher than a preset threshold, delete the first image corresponding to the third similarity higher than the preset threshold from the third image subset to update the third image subset.

6. A terminal device, characterized in that, The terminal device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image deduplication method as described in any one of claims 1 to 5.

7. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the image deduplication method as described in any one of claims 1 to 5.

Images