An image clustering method and apparatus

Abstract

Embodiments of the present application relate to the technical field of computer vision, and particularly relate to an image clustering method and device. The method comprises: clustering each image in a first image domain and each image in a second image domain respectively to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains in a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing collected images according to a space-time condition; when the first clustering result and the second clustering result satisfy an expansion condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result; and based on the third clustering result, adjusting the first clustering result to obtain a final clustering result of the first image domain. The final clustering result reduces the interference between images, considers the connection between images, and improves the clustering effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to an image clustering method, apparatus, computing device, and computer-readable storage medium. Background Art

[0002] With the widespread adoption of intelligent surveillance devices like cameras, massive amounts of image data are generated daily. In the era of big data security, how to utilize this massive amount of image data and accurately cluster and archive it remains a major challenge.

[0003] However, these massive image data are extremely complex. For example, there are massive portrait image data in the system. These portrait images are taken in different environments, have different behaviors, and different image qualities, which often results in poor results when directly clustering these portrait images.

[0004] In summary, the embodiments of the present invention provide a possible clustering method to reduce interference between image data while taking into account the connections between image data, thereby improving the image clustering effect. Summary of the Invention

[0005] The embodiment of the present invention provides a possible clustering method for reducing interference between image data while taking into account the connections between image data, thereby improving the image clustering effect.

[0006] In a first aspect, an embodiment of the present invention provides an image clustering method, comprising:

[0007] Clustering each image in a first image domain and each image in a second image domain respectively to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains among a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing the collected images according to spatiotemporal conditions;

[0008] When the first clustering result and the second clustering result meet an extension condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result;

[0009] Based on the third clustering result, the first clustering result is adjusted to obtain a final clustering result of the first image domain.

[0010] By dividing an image into multiple image domains based on spatiotemporal conditions, clustering each domain separately can eliminate interference between images with different spatiotemporal conditions, resulting in more accurate clustering. However, different image domains may also contain highly similar images. Therefore, clustering only one domain will ignore the connections between images in different domains, reducing clustering effectiveness. Therefore, a first clustering result can be obtained by clustering the first image domain, and a second clustering result can be obtained by clustering the second image domain that meets the first dimension's proximity condition. If the first and second clustering results meet the extension condition, the images in the first and second image domains are considered to be connected. Clustering is then performed based on the images in the first and second image domains to obtain a third clustering result. Because the third clustering result takes into account more information, such as the connections between images, the first clustering result can be adjusted based on the third clustering result to obtain the final clustering result. The final clustering result both reduces interference between images and considers the connections between images, improving clustering effectiveness.

[0011] Optionally, determining whether the first clustering result and the second clustering result meet an expansion condition in the following manner includes:

[0012] According to the similarity between the plurality of first cluster centers in the first clustering result and the plurality of second cluster centers in the second clustering result, it is determined whether the first clustering result and the second clustering result meet an expansion condition.

[0013] It is more reasonable and accurate to determine whether the expansion condition is met based on the cluster centers of the multiple first clustering results and the second clustering results.

[0014] Optionally, determining whether the first clustering result and the second clustering result meet an expansion condition based on similarities between multiple first cluster centers in the first clustering result and multiple second cluster centers in the second clustering result includes:

[0015] Determining each first cluster center in the first clustering result and each second cluster center in the second clustering result;

[0016] For any first cluster center, respectively calculate the similarity between the first cluster center and each second cluster center;

[0017] According to the highest similarity corresponding to each first cluster center in each second cluster center, it is determined whether the first clustering result and the second clustering result meet the expansion condition.

[0018] According to the highest similarity corresponding to each first cluster center in each second cluster center, whether the first clustering result and the second clustering result meet the expansion condition, if so, clustering can be performed based on more related images to obtain the third clustering result. In this way, the connection between images in different image domains is taken into account, and the clustering effect is improved.

[0019] Optionally, before obtaining the third clustering result, the method further includes:

[0020] determining a third image domain that satisfies an adjacent condition with the first image domain in a second dimension;

[0021] determining whether the first clustering result and the fourth clustering result of the third image domain meet an expansion condition;

[0022] Clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result includes:

[0023] Clustering is performed on each image in the first image domain, each image in the second image domain, and each image in the third image domain to obtain the third clustering result.

[0024] The first image domain can be expanded not only in the first dimension, but also in the second dimension. Specifically, the third image domain is determined based on the first image domain, and then whether the target image domain can be expanded in the second dimension is determined based on whether the first clustering result and the fourth clustering result meet the expansion conditions. The third clustering result obtained in this way takes into account richer image information, better considers the connection between images in different image domains, and improves the clustering effect.

[0025] Optionally, when the first clustering result and the second clustering result meet an extension condition, the extension domain of the first image domain includes the second image domain;

[0026] Adjusting the first clustering result based on the third clustering result to obtain a final clustering result of the first image domain includes:

[0027] Obtaining a fifth clustering result; the fifth clustering result is obtained by clustering each image in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain;

[0028] Based on the third clustering result and the fifth clustering result, the first clustering result is adjusted in a preset manner to obtain a final clustering result of the first image domain.

[0029] When determining the final clustering result, not only the third clustering result but also the fifth clustering result should be considered. Since the extended domain of the fourth image domain includes the first image domain, the fifth clustering result can also be used to guide the final image clustering result. In this way, the final image clustering result is more accurate.

[0030] Optionally, based on the third clustering result and the fifth clustering result, adjusting the first clustering result in a preset manner to obtain a final clustering result of the first image domain includes:

[0031] For any two images in the first clustering result, determining a probability that the two images belong to the same class based on the clustering of the two images in the first clustering result, the third clustering result, and the fifth clustering result, respectively; if the probability is greater than a preset threshold, the two images belong to the same class;

[0032] All category results are traversed, and categories containing at least one identical image are merged to obtain a final clustering result of the first image domain.

[0033] By comprehensively considering the first clustering result, the third clustering result, and the fifth clustering result, the probability that any two images in the first clustering result belong to the same class is determined, thereby determining whether the two images belong to the same class. In this way, each image in the first clustering result can be accurately clustered.

[0034] Optionally, the first dimension is a time dimension, and the second dimension is a space dimension; or

[0035] The first dimension is a spatial dimension, and the second dimension is a temporal dimension.

[0036] Since images taken at different times are more likely to be independent of each other, and images taken in different spaces are also more likely to be independent of each other, the images are divided using time and / or space as the first and / or second dimensions. The resulting image domains can reduce the influence between images taken at different times and / or spaces, and improve the clustering effect.

[0037] Optionally, when the first dimension is a time dimension, the adjacent condition refers to a time interval with the first image domain being less than a preset interval threshold;

[0038] When the first dimension is a spatial dimension, the adjacent condition means that the distance to the region boundary of the first image domain is less than a preset distance threshold.

[0039] The second image domain is not determined arbitrarily, but is determined based on satisfying the adjacent conditions. Different first dimensions correspond to different adjacent conditions. In this way, the images in the second image domain obtained are more likely to be related to the images in the first image domain, which can enable the subsequent third clustering results to take into account the images in different image domains that are related to each other as much as possible, thereby improving the clustering effect.

[0040] In a second aspect, an embodiment of the present invention further provides an image clustering device, comprising:

[0041] Clustering units, used to:

[0042] Clustering each image in a first image domain and each image in a second image domain respectively to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains among a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing the collected images according to spatiotemporal conditions;

[0043] When the first clustering result and the second clustering result meet an extension condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result;

[0044] A processing unit is configured to adjust the first clustering result based on the third clustering result to obtain a final clustering result of the first image domain.

[0045] Optionally, the clustering unit is specifically configured to:

[0046] According to the similarity between the plurality of first cluster centers in the first clustering result and the plurality of second cluster centers in the second clustering result, it is determined whether the first clustering result and the second clustering result meet an expansion condition.

[0047] Optionally, the clustering unit is specifically configured to:

[0048] Determining each first cluster center in the first clustering result and each second cluster center in the second clustering result;

[0049] For any first cluster center, respectively calculate the similarity between the first cluster center and each second cluster center;

[0050] According to the highest similarity corresponding to each first cluster center in each second cluster center, it is determined whether the first clustering result and the second clustering result meet the expansion condition.

[0051] Optionally, the clustering unit is further configured to:

[0052] determining a third image domain that satisfies an adjacent condition with the first image domain in a second dimension;

[0053] determining whether the first clustering result and the fourth clustering result of the third image domain meet an expansion condition;

[0054] The clustering unit is specifically used for:

[0055] Clustering is performed on each image in the first image domain, each image in the second image domain, and each image in the third image domain to obtain the third clustering result.

[0056] Optionally, when the first clustering result and the second clustering result meet an extension condition, the extension domain of the first image domain includes the second image domain;

[0057] The processing unit is specifically configured to:

[0058] Obtaining a fifth clustering result; the fifth clustering result is obtained by clustering each image in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain;

[0059] Based on the third clustering result and the fifth clustering result, the first clustering result is adjusted in a preset manner to obtain a final clustering result of the first image domain.

[0060] Optionally, the processing unit is specifically configured to:

[0061] For any two images in the first clustering result, determining a probability that the two images belong to the same class based on the clustering of the two images in the first clustering result, the third clustering result, and the fifth clustering result, respectively; if the probability is greater than a preset threshold, the two images belong to the same class;

[0062] All category results are traversed, and categories containing at least one identical image are merged to obtain a final clustering result of the first image domain.

[0063] Optionally, the first dimension is a time dimension, and the second dimension is a space dimension; or

[0064] The first dimension is a spatial dimension, and the second dimension is a temporal dimension.

[0065] Optionally, when the first dimension is a time dimension, the adjacent condition refers to a time interval with the first image domain being less than a preset interval threshold;

[0066] When the first dimension is a spatial dimension, the adjacent condition means that the distance to the region boundary of the first image domain is less than a preset distance threshold.

[0067] In a third aspect, an embodiment of the present invention further provides a computing device, including:

[0068] memory for storing computer programs;

[0069] The processor is configured to call the computer program stored in the memory and execute the image clustering method listed in any one of the above-mentioned ways according to the obtained program.

[0070] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the image clustering method listed in any of the above-mentioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 A schematic diagram of a system architecture provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of a process flow of an image clustering method provided by an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of a divided spatial domain provided by an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of a possible method of determining whether an expansion condition is satisfied for two different image domains by using cluster centers provided by an embodiment of the present invention;

[0076] Figure 5 A schematic diagram of a possible first clustering result and a third clustering result provided by an embodiment of the present invention;

[0077] Figure 6 A schematic diagram of a possible expansion scenario provided by an embodiment of the present invention;

[0078] Figure 7 A schematic diagram of a possible first clustering result, a third clustering result, and a fifth clustering result provided by an embodiment of the present invention;

[0079] Figure 8 A schematic diagram of a possible image clustering method provided by an embodiment of the present invention;

[0080] Figure 9 A schematic structural diagram of a possible image clustering device provided by an embodiment of the present invention;

[0081] Figure 10 A schematic diagram of the structure of a possible computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0083] All other embodiments derived by persons of ordinary skill in the art based on the exemplary embodiments described herein without inventive effort are within the scope of protection of the claims appended hereto. Furthermore, although the disclosure herein is presented based on one or more exemplary embodiments, it should be understood that each aspect of the disclosure may constitute a complete embodiment on its own.

[0084] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0085] In the specification and claims of this application and the drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or precedence, unless otherwise indicated. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances, for example, the embodiments of this application can be implemented in an order other than those shown or described in the drawings.

[0086] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0087] Figure 1 A system architecture applicable to an embodiment of the present invention is exemplarily shown. The system architecture may be a server 100 , including a processor 110 , a communication interface 120 , and a memory 130 .

[0088] The communication interface 120 is used to communicate with the terminal device, send and receive information transmitted by the terminal device, and realize communication.

[0089] The processor 110 is the control center of the server 100. It connects various parts of the server 100 using various interfaces and routes. It executes various functions of the server 100 and processes data by running or executing software programs and / or modules stored in the memory 130 and calling data stored in the memory 130. Optionally, the processor 110 may include one or more processing units.

[0090] Memory 130 can be used to store software programs and modules. Processor 110 executes various functional applications and data processing by running the software programs and modules stored in memory 130. Memory 130 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the data storage area may store data generated based on business processing. Memory 130 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state memory device.

[0091] It should be noted that the above Figure 1 The structure shown is only an example and is not limited in the embodiment of the present invention.

[0092] above Figure 1 The server shown can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0093] Figure 2 An image clustering method provided by an embodiment of the present invention is shown, comprising:

[0094] Step 201: Clustering each image in a first image domain and each image in a second image domain to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains among a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing the collected images according to spatiotemporal conditions;

[0095] Step 202: When the first clustering result and the second clustering result meet an extension condition, clustering the images in the first image domain and the images in the second image domain to obtain a third clustering result.

[0096] Step 203: Based on the third clustering result, the first clustering result is adjusted to obtain a final clustering result of the first image domain.

[0097] In step 201 , in order to reduce interference between image data, we can divide the image data according to spatiotemporal conditions to obtain multiple image domains, and cluster each image domain separately.

[0098] This paper describes a method for partitioning image domains, using the example of partitioning massive amounts of image data collected at all intersections in a city on a single day according to spatiotemporal conditions. By obtaining the location of each surveillance camera in the city, the images captured by each camera, and the time at which the images were captured, each image will have its own spatial and temporal information. For example, the spatial information for an image may be the camera's location coordinates (100, 100), and the temporal information may be the capture time 13:05. The location coordinates here can be latitude and longitude, or other coordinates, and are not limited in this embodiment of the present invention.

[0099] Based on the collected image data, the temporal and spatial boundaries are determined, and fine-grained image domain segmentation is achieved in the temporal and spatial dimensions. For example, for all collected images, the temporal boundary is from 0:00 to 23:59, and the spatial boundary is from (100, 100) to (500, 500). Then we can divide the time into two-hour segments, namely 0:00-1:59, 2:00-3:59, 4:00-5:59, 6:00-7:59, 8:00-9:59, 10:00-11:59, 12:00-13:59, 14:00-15:59, 16:00-17:59, 18:00-19:59, 20:00-21:59, and 22:00-23:59. The time information of each image can be discretized. For example, if the time information of an image is 1:05, the corresponding time domain is t0; if the time information of an image is 2:05, the corresponding time domain is t1, and so on. According to the spatial boundaries, it is divided into 4 areas, namely area 1 (100, 100)-(200, 200), area 2 (300, 100)-(500, 300), area 3 (100, 300)-(300, 400), area 4 (300, 400)-(500, 500), as shown in Figure 3As shown, the spatial information of each image is discretized. For example, if the spatial information of an image is (150, 200) and is located in area 1, the corresponding spatial domain is a0. The above is only an example and the embodiment of the present invention does not limit this.

[0100] In the above example, the time domain is divided into 12 and the spatial domain is divided into 4, so a total of 12*4 image domains can be divided, as shown in Table 1.

[0101] Table 1

[0102]

[0103]

[0104] Based on the temporal and spatial information of each image, each image is divided into image domains. Clustering is performed on each image domain to obtain a clustering result for each image domain. The specific clustering method is not limited in this embodiment of the present invention. For example, a deep convolutional neural network can be used to calculate the feature vectors of all images in the image domain. By calculating the similarity between the feature vectors, a k-means clustering method can be used to obtain a clustering result.

[0105] Clustering is performed on each image in a first image domain to obtain a first clustering result, and clustering is performed on each image in a second image domain to obtain a second clustering result. The first image domain and the second image domain are two different image domains from the plurality of image domains that satisfy an adjacency condition along a first dimension. The first dimension can be either a temporal dimension or a spatial dimension. For example, if the first image domain is a0t1 and the first dimension is a temporal dimension, then the second image domains can be a0t0, a0t2, a0t3, etc.; if the first dimension is a spatial dimension, then the second image domains can be a1t1, a2t1, etc.

[0106] Optionally, the user can customize the adjacent condition. If the first dimension is a temporal dimension, the adjacent condition refers to the time interval with the first image domain being less than a preset interval threshold; if the first dimension is a spatial dimension, the adjacent condition refers to the distance from the region boundary of the first image domain being less than a preset distance threshold. For example, if the first dimension is a temporal dimension and the adjacent condition is that the time interval with the first image domain is less than 2 hours, then the only second image domains with a time interval less than 2 hours from the first image domain a0t1 are a0t0 and a0t2. If the first dimension is a spatial dimension and the adjacent condition is that the distance from the region boundary of the first image domain is less than 200 meters, then the only second image domain with a distance less than 200 meters from the region boundary of the first image domain a0t1 is a1t1. There can be one or more second image domains, and this is not limited to this.

[0107] In step 202, it is determined whether the first image domain needs to be expanded based on the first clustering result and the second clustering result. If expansion is required, clustering is performed on the expanded image to obtain a third clustering result.

[0108] Specifically, determining whether the first clustering result and the second clustering result meet the expansion condition is performed in the following manner, including:

[0109] According to the similarity between the plurality of first cluster centers in the first clustering result and the plurality of second cluster centers in the second clustering result, it is determined whether the first clustering result and the second clustering result meet an expansion condition.

[0110] For example, in step 201, clustering is performed on the first image domain to obtain a first clustering result of two first cluster centers, namely M1 and M2. Clustering is performed on the second image domain to obtain a second clustering result of three second cluster centers, namely N1, N2 and N3. The similarities between M1 and N1, N2, N3, and the similarities between M2 and N1, N2, N3 are calculated respectively. Figure 4 A schematic diagram shows a possible method for determining whether two different image domains meet the expansion condition by clustering the centers. The six similarities obtained are averaged, median, or calculated using other methods to determine the final similarity. This final similarity is then compared with a similarity threshold to determine whether the first and second clustering results meet the expansion condition.

[0111] It is also possible to determine whether the first clustering result and the second clustering result meet the expansion condition in the following manner:

[0112] Determine each first cluster center in the first clustering result and each second cluster center in the second clustering result; for any first cluster center, calculate the similarity between the first cluster center and each second cluster center respectively; and determine whether the first clustering result and the second clustering result meet the expansion condition based on the highest similarity corresponding to each first cluster center in each second cluster center.

[0113] For example, in step 201, clustering the first image domain yields two first cluster centers, M1 and M2, and clustering the second image domain yields three second cluster centers, N1, N2, and N3. The highest similarity among the similarities between M1 and N1, N2, and N3 is determined to be s1; and the highest similarity between M2 and N1, N2, and N3 is determined to be s3. The final similarity S is determined by taking the average, median, or other calculation methods of s1 and s3. For example, the final similarity S is determined by the following formula:

[0114]

[0115] Optionally, when determining the similarity between two cluster centers, such as M1 and N1, various methods may be used, such as calculating cosine similarity, Euclidean distance, etc., which is not limited in this embodiment of the present invention.

[0116] The formula for calculating cosine similarity is as follows:

[0117]

[0118] The final similarity is compared with a similarity threshold to determine whether the first clustering result and the second clustering result meet the expansion condition. If the similarity is greater than the similarity threshold, it indicates that the first clustering result and the second clustering result meet the expansion condition, and the first image domain can be expanded in the first dimension. If the similarity is less than the similarity threshold, it indicates that the first clustering result and the second clustering result do not meet the expansion condition, and the first image domain cannot be expanded in the first dimension. The similarity threshold can be adjusted according to the data situation. This embodiment of the present invention is not limited to this.

[0119] If the expansion conditions are met, the first image domain can be expanded in the first dimension. Then, clustering is performed based on each image in the first image domain and each image in the second image domain to obtain a third clustering result. Originally, the first clustering result was obtained by clustering the first image domain, and now the third clustering result is obtained by clustering more images. Since new images have been added, the original first clustering result will change due to the addition of new images. For example, in the first clustering result, image A, image B, and image C are in the same category. In the third clustering result, image A and image B are in the same category, and image C, image H, and image G are in the same category. Since the third clustering result takes into account the connection between images in different image domains, while the first clustering result only clusters images in that image domain, the third clustering result is a further correction to the first clustering result.

[0120] Because the third clustering result includes images from other image domains, such as images H and G in the above example, when performing steps 201 and 202 on the other image domains, the third clustering result may also include images H and G. In this third clustering result, images H and G may not belong to the same cluster, which will cause conflicts. Therefore, it is inappropriate to directly use the third clustering result as the final clustering result. The first clustering result can only be further adjusted based on the third clustering result, that is, step 203. The specific adjustment method is detailed below. The following first introduces another method for obtaining the third clustering result.

[0121] We can expand not only in the first dimension, but also in the second dimension. Before obtaining the third clustering result, the method further includes: determining a third image domain that satisfies the adjacent condition with the first image domain in the second dimension; and determining whether a fourth clustering result of the first clustering result and the third image domain satisfies the expansion condition.

[0122] In the above, if the first dimension is the time dimension, then the second dimension is the space dimension. If the first image domain is a0t1, and the third image domain that satisfies the adjacency condition in the second dimension is a1t1, clustering the third image domain to obtain a fourth clustering result is performed. The method for determining whether the first and fourth clustering results meet the expansion condition is the same as the method for determining whether the first and second clustering results meet the expansion condition described above, and will not be repeated here.

[0123] If the expansion condition is met, the first image domain may be expanded in the second dimension.

[0124] Then, the third clustering result can be obtained in the following manner: clustering each image in the first image domain, each image in the second image domain, and each image in the third image domain to obtain the third clustering result.

[0125] If, in the above example, the first image domain a0t1 meets the extension conditions with the second image domains a0t0 and a0t2, and also meets the extension conditions with the third image domain a1t1, clustering can be performed based on the four image domains a0t1, a0t0, a0t2, and a1t1 to obtain a third clustering result. Because this third clustering result considers not only the connections between images in different image domains in the temporal dimension but also the connections between images in different image domains in the spatial dimension, it considers more information and is therefore more accurate.

[0126] In step 203, based on the third clustering result, the first clustering result is adjusted to obtain a final clustering result of the first image domain.

[0127] In steps 201 and 202, we expand the first image domain to obtain an extended domain. If the expansion is performed only along the first dimension, the extended domain includes the second image domain; if the expansion is also performed along the second dimension, the extended domain includes both the second and third image domains. Clustering is performed on multiple images in the extended domain and the first image domain to obtain a third clustering result.

[0128] An optional implementation is to directly adjust the first clustering result only by using the third clustering result to obtain the final clustering result.

[0129] For example, Figure 5A possible first clustering result and a possible third clustering result are shown. The first clustering result shows that images A, B, and C belong to one class, and images D and E belong to another class. The third clustering result shows that images A and B belong to one class, images C and F belong to one class, and image D, images E, G, and H belong to one class.

[0130] For any two images in the first clustering result, the probability that the two images belong to the same category is determined based on the clustering of the two images in the first clustering result, the third clustering result, and the fifth clustering result, respectively. If the probability is greater than a preset threshold, the two images belong to the same category; all category results are traversed, and the categories containing at least one identical image are merged to obtain the final clustering result of the first image domain.

[0131] In the above example, for any two images among images A, B, C, D, and E, for example, images A and B, in the first clustering result, AB is in the same class, recorded as 1, and in the third clustering result, AB is still in the same class, recorded as 1, then the probability that AB is in the same class is (1+1) / 2=1. If the preset threshold is 1 / 2, then AB belong to the same class. For images A and C, in the first clustering result, AC is in the same class, recorded as 1, and in the third clustering result, AC is not in the same class, recorded as 0, then the probability that AC is in the same class is (1+0) / 2=1 / 2. If it is not greater than the preset threshold, then AC does not belong to the same class. Then, in the above example, according to the above method, the category results that can be determined are: images A and B are in the same class, image C is in the same class, and images D and E are in the same class. Since none of these classification results contain at least one identical image, there is no need to merge them. The final clustering result for the first image domain is that images A and B belong to one class, image C belongs to another class, and images D and E belong to another class. If, in the above example, images A and B belong to one class, and images A and C belong to another class, then these two classes contain the same image A. Therefore, they can be merged, resulting in the final clustering result: images A, B, and C belong to one class, and images D and E belong to one class. The above is merely an example and is not a limitation of the present invention.

[0132] Another optional implementation is to adjust the first clustering result using the third and fifth clustering results to obtain a final clustering result. The fifth clustering result is obtained by clustering images in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain.

[0133] Since the method steps of step 201 and step 202 are performed on any image domain in each image domain, the extended domain obtained by performing steps 201 and 202 on another image domain, ie, the fourth image domain, may include the first image domain. Figure 6 A possible expansion scenario is shown. For example, the first image domain is a0t1, and it is expanded to obtain an extended domain, which includes the second image domains a0t2 and a0t0. When the fourth image domain a1t1 is expanded, the obtained extended domain includes the first image domain a0t1, because the image domain a0t1 and the image domain a1t1 meet the expansion conditions. When clustering the images in the fourth image domain and the extended domain of the fourth image domain, a fifth clustering result is obtained. In the above example, the second image domain and the fourth image domain are different. However, in actual implementation, the second image domain and the fourth image domain can be the same or different. For example, if the fourth image domain is a0t2 or a0t0, the obtained extended domain will also include the first image domain a0t1.

[0134] The manner of adjusting the first clustering result based on the third clustering result and the fifth clustering result is the same as the manner of adjusting the first clustering result based on the third clustering result.

[0135] Figure 7 The following table shows possible first, third, and fifth clustering results. The first clustering result shows that images A, B, and C belong to one class, and images D and E belong to one class. The third clustering result shows that images A and B belong to one class, images C and F belong to one class, and images D, E, G, and H belong to one class. The fifth clustering result shows that images A, B, C, and K belong to one class, images D and I belong to one class, and images E and J belong to one class.

[0136] In the above example, for any two images among images A, B, C, D, and E, for example, images A and B, if AB is in the same class in the first clustering result (denoted as 1), AB is still in the same class in the third clustering result (denoted as 1), and AB is in the same class in the fifth clustering result (denoted as 1), then the probability that AB is in the same class is (1+1+1) / 3=1. If the preset threshold is 1 / 2, then AB belongs to the same class. For images A and C, if AC is in the same class in the first clustering result (denoted as 1), AC is not in the same class in the third clustering result (denoted as 0), and AC is in the same class in the fifth clustering result (denoted as 1), then the probability that AC is in the same class is (1+0+1) / 3=2 / 3. If the threshold is greater than the preset threshold, then AC belongs to the same class. Similarly, in the above example, according to the above method, the following classification results can be determined: images A and B are in the same class, images A and C are in the same class, and images D and E are in the same class. After traversing these category results and merging them, the final clustering result is: image A, image B and image C are in one category, and image D and image E are in one category.

[0137] In actual applications, each clustering result contains a large number of images, and there may be more than one fifth clustering result because there may be many extended domains containing the first image domain. Therefore, when the first clustering result is corrected by the third and fifth clustering results, the final clustering result may differ significantly from the first clustering result, or it may be slightly different, depending on the actual situation.

[0138] In order to better explain the embodiment of the present invention, the above-mentioned image clustering process will be explained as a whole below. Figure 8 shown.

[0139] Step 801: Divide each image into different image domains according to the temporal and spatial conditions of the image.

[0140] Step 802: cluster the images in the first image domain to obtain a first clustering result.

[0141] For example, if the first image domain is a0t1, the first clustering result includes two first cluster centers.

[0142] Step 803: Determine a second image domain that meets the adjacent condition in the time dimension through the first image domain.

[0143] For example, the determined second image areas are a0t0 and a0t2.

[0144] Step 804: cluster the images in the second image domain to obtain a second clustering result.

[0145] For example, clustering is performed on the image domain a0t0, and the second clustering result obtained includes 3 second cluster centers; clustering is performed on the image domain a0t2, and the second clustering result obtained includes 4 second cluster centers.

[0146] Step 805 : determining whether the first image domain and the second image domain meet an expansion condition according to the first clustering result and the second clustering result; if so, performing expansion in the time dimension; otherwise, not performing expansion.

[0147] For example, after calculation and processing, it is obtained that the image domain a0t1 and the image domain a0t0 meet the extension condition; the image domain a0t1 and the image domain a0t2 do not meet the extension condition.

[0148] Step 806: Determine a third image domain that meets the adjacent condition in the spatial dimension through the first image domain.

[0149] For example, the determined third image domain is a1t1.

[0150] Step 807: cluster the images in the third image domain to obtain a fourth clustering result.

[0151] Step 808: Determine whether the first image domain and the third image domain meet the expansion condition according to the first clustering result and the fourth clustering result. If so, expand them in the spatial dimension; otherwise, do not expand them.

[0152] For example, after calculation, it is obtained that the image domain a0t1 and the image domain a1t1 meet the expansion condition.

[0153] Step 809 : clustering the images in the first image domain and the extended domain of the first image domain to obtain a third clustering result.

[0154] In the above example, the images in the image domains a0t1, a0t0, and a1t1 are clustered to obtain a third clustering result.

[0155] Step 810 : Obtain a fifth clustering result, where the fifth clustering result is obtained by clustering images in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain.

[0156] Step 811 : Based on the third clustering result and the fifth clustering result, the first clustering result is adjusted to obtain a final clustering result of the first image domain.

[0157] Based on the same technical concept, Figure 9 The structure of an image clustering device provided by an embodiment of the present invention is exemplarily shown, and the structure can execute the process of image clustering.

[0158] like Figure 9 As shown, the device specifically includes:

[0159] The clustering unit 901 is configured to:

[0160] Clustering each image in a first image domain and each image in a second image domain respectively to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains among a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing the collected images according to spatiotemporal conditions;

[0161] When the first clustering result and the second clustering result meet an extension condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result;

[0162] The processing unit 902 is configured to adjust the first clustering result based on the third clustering result to obtain a final clustering result of the first image domain.

[0163] Optionally, the clustering unit 901 is specifically configured to:

[0164] According to the similarity between the plurality of first cluster centers in the first clustering result and the plurality of second cluster centers in the second clustering result, it is determined whether the first clustering result and the second clustering result meet an expansion condition.

[0165] Optionally, the clustering unit 901 is specifically configured to:

[0166] Determining each first cluster center in the first clustering result and each second cluster center in the second clustering result;

[0167] For any first cluster center, respectively calculate the similarity between the first cluster center and each second cluster center;

[0168] According to the highest similarity corresponding to each first cluster center in each second cluster center, it is determined whether the first clustering result and the second clustering result meet the expansion condition.

[0169] Optionally, the clustering unit 901 is further configured to:

[0170] determining a third image domain that satisfies an adjacent condition with the first image domain in a second dimension;

[0171] determining whether the first clustering result and the fourth clustering result of the third image domain meet an expansion condition;

[0172] The clustering unit 901 is specifically configured to:

[0173] Clustering is performed on each image in the first image domain, each image in the second image domain, and each image in the third image domain to obtain the third clustering result.

[0174] Optionally, when the first clustering result and the second clustering result meet an extension condition, the extension domain of the first image domain includes the second image domain;

[0175] The processing unit 902 is specifically configured to:

[0176] Obtaining a fifth clustering result; the fifth clustering result is obtained by clustering each image in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain;

[0177] Based on the third clustering result and the fifth clustering result, the first clustering result is adjusted in a preset manner to obtain a final clustering result of the first image domain.

[0178] Optionally, the processing unit 902 is specifically configured to:

[0179] For any two images in the first clustering result, determining a probability that the two images belong to the same class based on the clustering of the two images in the first clustering result, the third clustering result, and the fifth clustering result, respectively; if the probability is greater than a preset threshold, the two images belong to the same class;

[0180] All category results are traversed, and categories containing at least one identical image are merged to obtain a final clustering result of the first image domain.

[0181] Optionally, the first dimension is a time dimension, and the second dimension is a space dimension; or

[0182] The first dimension is a spatial dimension, and the second dimension is a temporal dimension.

[0183] Optionally, when the first dimension is a time dimension, the adjacent condition refers to a time interval with the first image domain being less than a preset interval threshold;

[0184] When the first dimension is a spatial dimension, the adjacent condition means that the distance to the region boundary of the first image domain is less than a preset distance threshold.

[0185] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 10 As shown, it includes at least one processor 1001 and a memory 1002 connected to the at least one processor. The specific connection medium between the processor 1001 and the memory 1002 is not limited in the embodiment of the present application. Figure 10 For example, the processor 1001 and the memory 1002 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0186] In the embodiment of the present application, the memory 1002 stores instructions that can be executed by at least one processor 1001. The at least one processor 1001 can perform the steps of the above-mentioned image clustering method by executing the instructions stored in the memory 1002.

[0187] Among them, the processor 1001 is the control center of the computer device. It can use various interfaces and lines to connect various parts of the computer device. By running or executing instructions stored in the memory 1002 and calling data stored in the memory 1002, image clustering is performed. Optionally, the processor 1001 may include one or more processing units. The processor 1001 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1001. In some embodiments, the processor 1001 and the memory 1002 may be implemented on the same chip. In some embodiments, they may also be implemented separately on separate chips.

[0188] The processor 1001 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0189] Memory 1002 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1002 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Memory 1002 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1002 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0190] Based on the same technical concept, an embodiment of the present invention further provides a computer-readable storage medium storing a computer-executable program, which is used to enable a computer to execute the image clustering method listed in any of the above-mentioned ways.

[0191] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0192] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0195] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image clustering method, characterized in that: include: Clustering each image in the first image domain and each image in the second image domain respectively to obtain a first clustering result and a second clustering result; The first image domain and the second image domain are two different image domains among the multiple image domains that meet an adjacent condition in a first dimension; the multiple image domains are obtained by dividing the collected images according to spatiotemporal conditions; determining, based on similarities between a plurality of first cluster centers in the first clustering result and a plurality of second cluster centers in the second clustering result, whether the first clustering result and the second clustering result meet an expansion condition; When the first clustering result and the second clustering result meet an extension condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result; Based on the third clustering result, the first clustering result is adjusted to obtain a final clustering result of the first image domain.

2. The method according to claim 1, wherein Determining whether the first clustering result and the second clustering result meet an expansion condition according to similarities between the plurality of first cluster centers in the first clustering result and the plurality of second cluster centers in the second clustering result includes: Determining each first cluster center in the first clustering result and each second cluster center in the second clustering result; For any first cluster center, respectively calculate the similarity between the first cluster center and each second cluster center; According to the highest similarity corresponding to each first cluster center in each second cluster center, it is determined whether the first clustering result and the second clustering result meet the expansion condition.

3. The method according to claim 1, wherein Before obtaining the third clustering result, it also includes: determining a third image domain that satisfies an adjacent condition with the first image domain in a second dimension; determining whether the first clustering result and the fourth clustering result of the third image domain meet an expansion condition; Clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result includes: Clustering is performed on each image in the first image domain, each image in the second image domain, and each image in the third image domain to obtain the third clustering result.

4. The method according to claim 1, wherein When the first clustering result and the second clustering result meet an extension condition, the extension domain of the first image domain includes the second image domain; Adjusting the first clustering result based on the third clustering result to obtain a final clustering result of the first image domain includes: Obtaining a fifth clustering result; the fifth clustering result is obtained by clustering each image in a fourth image domain and an extended domain of the fourth image domain, and the extended domain of the fourth image domain includes the first image domain; Based on the third clustering result and the fifth clustering result, the first clustering result is adjusted in a preset manner to obtain a final clustering result of the first image domain.

5. The method according to claim 4, wherein Based on the third clustering result and the fifth clustering result, adjusting the first clustering result in a preset manner to obtain a final clustering result of the first image domain includes: For any two images in the first clustering result, determining a probability that the two images belong to the same class based on the clustering of the two images in the first clustering result, the third clustering result, and the fifth clustering result, respectively; if the probability is greater than a preset threshold, the two images belong to the same class; All category results are traversed, and categories containing at least one identical image are merged to obtain a final clustering result of the first image domain.

6. The method according to any one of claims 1 to 5, wherein: The first dimension is a time dimension, and the second dimension is a space dimension; or The first dimension is a spatial dimension, and the second dimension is a temporal dimension.

7. The method according to claim 6, wherein When the first dimension is a time dimension, the adjacent condition means that the time interval with the first image domain is less than a preset interval threshold; When the first dimension is a spatial dimension, the adjacent condition means that the distance to the region boundary of the first image domain is less than a preset distance threshold.

8. An image clustering device, characterized in that: include: Clustering units, used to: Clustering each image in a first image domain and each image in a second image domain respectively to obtain a first clustering result and a second clustering result; the first image domain and the second image domain are two different image domains among a plurality of image domains that satisfy an adjacent condition in a first dimension; the plurality of image domains are obtained by dividing the collected images according to spatiotemporal conditions; determining, based on similarities between a plurality of first cluster centers in the first clustering result and a plurality of second cluster centers in the second clustering result, whether the first clustering result and the second clustering result meet an expansion condition; When the first clustering result and the second clustering result meet an extension condition, clustering each image in the first image domain and each image in the second image domain to obtain a third clustering result; A processing unit is configured to adjust the first clustering result based on the third clustering result to obtain a final clustering result of the first image domain.

9. A computing device, characterized in that include: memory for storing computer programs; A processor, configured to call a computer program stored in the memory, and execute the method according to any one of claims 1 to 7 according to the obtained program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the method according to any one of claims 1 to 7.

Images