Image clustering method, device, electronic device and storage medium

Abstract

The disclosed embodiments relate to a method, device, electronic device and storage medium for image clustering, wherein the method for image clustering comprises: obtaining target images to be clustered, wherein the types of the target images include face images and body images; obtaining image feature information of the target images, wherein the image feature information includes vector feature information and spatiotemporal feature information; for an image combination consisting of any two target images, obtaining combined feature information of each image combination based on the image feature information of each target image; and performing image clustering based on the combined feature information of each image combination and a pre-trained classifier. When performing image clustering, the disclosed embodiments not only consider the vector feature information of the image, but also consider the spatiotemporal feature information of the image, so as to comprehensively judge whether the images in the image combination are of the same category, which is conducive to improving the accuracy of image clustering.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image clustering and visual big data, and in particular to an image clustering method, device, electronic device and storage medium. Background Art

[0002] Clustering massive multi-dimensional feature vectors is an important information processing method in the fields of computer vision, visual big data, data mining, etc. Image clustering can divide the feature vectors of multiple images in an image data set into several clusters, with the feature vector distance within the same cluster as small as possible and the feature vector distance between different clusters as large as possible.

[0003] In the currently commonly used image clustering methods, clustering is usually performed based on the vector distance features between images. However, due to shooting angles, lighting, the distance between the camera and the object being photographed, the pixels of the camera, etc., the quality of the captured images varies, resulting in that the vector distances of images of the same object may not be close, and thus they cannot be clustered into one category based on the vector distance alone, resulting in poor clustering accuracy. In addition, the currently commonly used image clustering methods can only cluster facial images, and cannot cluster images that do not capture faces but have bodies in actual scenes. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method, a device, an electronic device and a storage medium for image clustering.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for image clustering, comprising:

[0006] Acquire target images to be clustered, where types of the target images include face images and body images;

[0007] Acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0008] For an image combination consisting of any two target images, obtaining combination feature information of each image combination based on the image feature information of each target image;

[0009] Image clustering is performed based on the combined feature information of each image combination and a pre-trained classifier.

[0010] In a second aspect, the present disclosure also provides a method for training a classifier for image clustering, comprising:

[0011] Acquire a target image for training, where types of the target image include a face image and a body image;

[0012] Acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0013] For an image combination consisting of any two target images, obtaining combined feature information of each image combination based on the image feature information of each target image, and obtaining a clustering annotation result of each image combination;

[0014] The classifier is trained based on the combined feature information of each image combination and the clustering annotation result of each image combination.

[0015] In a third aspect, the present disclosure also provides a method for merging clustered data sets, including:

[0016] Acquire a first clustering data set and a second clustering data set, wherein the first clustering data set includes a plurality of first feature vectors and at least one first cluster identifier obtained by clustering the plurality of first feature vectors, and the second clustering data set includes the plurality of first feature vectors and at least one second feature vector, and at least one second cluster identifier obtained by clustering the plurality of first feature vectors and the at least one second feature vector;

[0017] Performing diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, wherein the cluster-affected eigenvector group includes at least one first eigenvector;

[0018] Based on the at least one second feature vector and the feature vector group affected by the cluster, the second cluster identifier in the second cluster data set is updated to obtain a merged third cluster data set.

[0019] In a fourth aspect, the present disclosure also provides an apparatus for image clustering, including:

[0020] An image acquisition module, used for acquiring target images to be clustered, wherein the types of the target images include face images and body images;

[0021] An image feature acquisition module, used to acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0022] A combination feature acquisition module, for acquiring, for an image combination consisting of any two target images, combination feature information of each image combination based on image feature information of each target image;

[0023] The clustering module is used to perform image clustering based on the combined feature information of each image combination and a pre-trained classifier.

[0024] In a fifth aspect, the present disclosure also provides a training device for a classifier for image clustering, comprising:

[0025] A training image acquisition module, used to acquire a target image for training, wherein the types of the target image include a face image and a body image;

[0026] A feature information acquisition module, used to acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0027] A combination annotation module is used to obtain, for an image combination consisting of any two target images, combination feature information of each image combination based on the image feature information of each target image, and to obtain a cluster annotation result of each image combination;

[0028] The training module is used to train the classifier based on the combined feature information of each image combination and the clustering annotation result of each image combination.

[0029] In a sixth aspect, an embodiment of the present disclosure further provides a device for merging clustered data sets, including:

[0030] A data set acquisition module, used to acquire a first clustering data set and a second clustering data set, wherein the first clustering data set includes a plurality of first feature vectors and at least one first clustering identifier obtained by clustering the plurality of first feature vectors, and the second clustering data set includes the plurality of first feature vectors and at least one second feature vector, and at least one second clustering identifier obtained by clustering the plurality of first feature vectors and the at least one second feature vector;

[0031] A diffusion processing module, configured to perform diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, wherein the cluster-affected eigenvector group includes at least one first eigenvector;

[0032] An updating module is used to update the second cluster identifier in the second cluster data set based on the at least one second feature vector and the feature vector group affected by the cluster, so as to obtain a merged third cluster data set.

[0033] In the seventh aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor for reading the executable instructions from the memory and executing the executable instructions to implement any of the image clustering methods provided in the embodiments of the present disclosure, or to implement any of the training methods for classifiers for image clustering provided in the embodiments of the present disclosure, or to implement any of the methods for merging clustering data sets provided in the embodiments of the present disclosure.

[0034] In an eighth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute any of the image clustering methods provided in the embodiments of the present disclosure, or to execute any of the training methods for classifiers for image clustering provided in the embodiments of the present disclosure, or to execute any of the methods for merging clustering data sets provided in the embodiments of the present disclosure.

[0035] In a ninth aspect, an embodiment of the present disclosure further provides a computer program product, which is used to execute any of the image clustering methods provided in the embodiments of the present disclosure, or to execute any of the training methods for classifiers for image clustering provided in the embodiments of the present disclosure, or to execute any of the methods for merging clustering data sets provided in the embodiments of the present disclosure.

[0036] Compared with the prior art, the technical solution provided by the disclosed embodiment has at least the following advantages: in the disclosed embodiment, target images to be clustered are obtained, including face images and body images, and image feature information of the target images is obtained, including vector feature information and spatiotemporal feature information, and then for an image combination consisting of any two target images, combined feature information of each image combination is obtained based on the image feature information of each target image, and image clustering is performed based on the combined feature information of each image combination and a pre-trained classifier, so that when performing image clustering, not only the vector feature information of the image is considered, but also the spatiotemporal feature information of the image is considered, so as to comprehensively judge whether the images in the image combination are of the same category, which is conducive to improving the accuracy of image clustering. In addition, by combining the spatiotemporal feature information of the image to perform image clustering, the face and body that are actually the same object can be associated, which can not only improve the aggregation degree of image clustering, but also improve the accuracy of target object analysis in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0038] Figure 1 A schematic diagram of a flow chart of an image clustering method provided by an embodiment of the present disclosure;

[0039] Figure 2 A schematic diagram of the prediction results of the classifier provided in the embodiment of the present disclosure;

[0040] Figure 3 A schematic diagram of a flow chart of a method for training a classifier for image clustering provided in an embodiment of the present disclosure;

[0041] Figure 4 This is a diagram of the architecture of a device supporting visual target analysis capabilities according to an embodiment of the present disclosure;

[0042] Figure 5 A schematic diagram of a process for merging cluster data sets provided in an embodiment of the present disclosure;

[0043] Figure 6 is a schematic diagram of a first clustering data set and a second clustering data set;

[0044] Figure 7 A schematic diagram of the result of merging the first clustering data set and the second clustering data set;

[0045] Figure 8 A schematic diagram of the structure of an image clustering device provided by an embodiment of the present disclosure;

[0046] Fig. 9 A schematic diagram of the structure of a training device for a classifier for image clustering provided in one embodiment of the present disclosure;

[0047] Fig.10 A schematic diagram of the structure of a device for merging clustered data sets provided in one embodiment of the present disclosure;

[0048] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0050] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0051] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0052] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0053] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0054] Image clustering is an important technology with wide applications and high practical value. Image clustering can aggregate facial images captured by multiple cameras and group facial images of the same object (such as animals) into one category. In a short time and space range, facial image clustering can improve the ability of visual target analysis, and the target analysis distance is longer; in a long time and space range (such as across days), the clustering results can be used to further analyze the behavior trajectory of the target object, related objects and other information. Image clustering can be applied to multiple scenarios, such as tracking animal trajectories in wildlife reserves, understanding the life patterns of wild animals, etc., and the activity trajectories of animals can be analyzed by image clustering.

[0055] Currently, commonly used image clustering methods usually cluster images based on the vector distance features between them. However, images captured by cameras are of varying quality due to their long distance and the influence of lighting, angle, pixels, etc., making it difficult to extract effective visual features from images. If we only rely on the vector distance features of images to determine whether they are the same object, images that are actually the same object will not be able to be clustered into one category due to large differences in their vector distance features, and thus images with occlusion, lowered heads, or low quality cannot be recalled, resulting in low clustering accuracy.

[0056] Moreover, commonly used image clustering methods can only cluster a single data source (i.e., facial images). Many images in actual scenes that do not capture faces but have bodies cannot be clustered. The clustering data source is single, which is not conducive to target analysis.

[0057] In response to the above problems, the present disclosure provides a method for image clustering, by obtaining target images to be clustered, including facial images and body images, and obtaining image feature information of the target images, including vector feature information and spatiotemporal feature information, and then for an image combination consisting of any two target images, obtaining combined feature information of each image combination based on the image feature information of each target image, and performing image clustering based on the combined feature information of each image combination and a pre-trained classifier, thereby achieving image clustering that integrates vector feature information and spatiotemporal feature information, and taking into account two data sources, facial images and body images, during clustering, achieving association between faces and bodies that are actually the same object, which can not only improve the recall rate (i.e., aggregation degree) of image clustering, but also improve the accuracy of target object analysis in images.

[0058] Figure 1 The flowchart of the method for image clustering provided in one embodiment of the present disclosure can be applied to clustering images containing faces and / or bodies and analyzing target objects. The method for image clustering can be executed by an image clustering device, which can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities, such as a server, a cloud server, a smart phone, a camera and other terminal devices.

[0059] like Figure 1 As shown, the image clustering method provided by the embodiment of the present disclosure may include:

[0060] S101 : Acquire target images to be clustered, where types of the target images include face images and body images.

[0061] Among them, the facial image may be an image containing the facial features of the clustering object, and the body image may be an image containing the body features of the clustering object. The clustering object may be an animal, or other objects with a face and a body. Taking the clustering object as an animal as an example, in the embodiment of the present disclosure, the body image may be an image in which the face of the animal is not captured, such as an image captured when the animal lowers its head, an image captured when the face of the animal is blocked by an obstruction, etc. The body image may also be an image containing the face of the animal but the facial features cannot be recognized due to insufficient clarity, and the present disclosure does not limit this.

[0062] In the disclosed embodiments, the target image to be clustered can be obtained from the images and videos captured by the camera. For example, the face or body image containing the clustering object can be screened out from all the images captured by the camera as the target image. For another example, a video clip of the most recent period (which can be one day, one week, one month, etc.) can be intercepted from the video captured by the camera, and a video frame image containing the face or body of the clustering object can be obtained from the video clip as the target image.

[0063] S102: Acquire image feature information of the target image, where the image feature information includes vector feature information and spatiotemporal feature information.

[0064] Among them, the vector feature information can be a vector feature determined based on the image target in the target image, the image target can be the face or body of the clustering object, and the spatiotemporal feature information can include but is not limited to the shooting time of the image, the longitude and latitude of the camera that shot the target image, etc.

[0065] In the disclosed embodiment, for the face image and the body image, vector feature information and spatiotemporal feature information can be obtained as the image feature information of the face image. In addition, due to shooting angles, occlusions and other reasons, the confidence of the vector distance between the feature vectors determined according to the body in different images is not high, and the vector distance between the feature vectors of the body of the same clustering object is not necessarily close. Therefore, for the body image, the vector feature information can also be ignored, and the spatiotemporal feature information can be used as the image feature information of the body image. By obtaining the spatiotemporal feature information such as the shooting time of the target image and the camera latitude and longitude, it is convenient to associate the body image with the face image. For example, in two images (such as a body image and a face image) with adjacent shooting times collected by the same camera, the probability that the clustering object in the body image and the clustering object in the face image belong to the same object will be very high. Even if there is a large difference between the vector feature information of the body image and the vector feature information of the face image, combined with the spatiotemporal feature information of the image, the body image and the face image may also be judged as a clustering object, so that the spatiotemporal feature information is considered when performing image clustering, which is conducive to improving the clustering accuracy.

[0066] Exemplarily, the vector feature information of the facial image can be obtained through a pre-trained neural network model. The facial image is input into the trained neural network model. The convolution kernel of the neural network model can extract the features of the face in the facial image, and then the vector feature information of the facial image can be obtained from the convolution kernel. The spatiotemporal feature information of the target image can be directly obtained. For example, the shooting time of the target image can be directly determined based on the acquisition time of the target image. Usually, an acquisition time watermark is added to the image captured by the camera, and the shooting time of the target image can be determined based on the acquisition time watermark; the longitude and latitude of the camera can be determined based on the installation location of the camera.

[0067] S103 . For an image combination formed by any two target images, obtain combination feature information of each image combination based on the image feature information of each target image.

[0068] In the disclosed embodiment, for the target images to be clustered, all the target images can be arbitrarily combined in pairs to obtain multiple image combinations, and for any target image, based on the image feature information of each target image, the combined feature information of each image combination can be determined.

[0069] The combined feature information may include but is not limited to at least one of the image feature vector distance between the vector feature information, the structured type combination of the image target, the shooting time difference, the camera position difference, etc. There are three types of structured type combinations of image targets, namely, face-face combination, face-body combination, and body-body combination.

[0070] In the disclosed embodiment, when the image targets in the two images in a certain image combination are both faces, it can be determined that the structured type of the image target corresponding to the image combination is a combination of faces and faces, and the image feature vector distance corresponding to the image combination can be determined based on the vector feature information corresponding to the two target images in the image combination and the existing vector distance calculation method. When the image combination includes a body image, the image feature vector distance of the image combination is not calculated.

[0071] For example, when the scheme of the present disclosure is applied to a scene of clustering analysis of animals, the clustering objects are animals, and the structured type combination of image targets can be divided into three combination forms: animal face and animal face combination, animal face and animal body combination, and animal body and animal body combination. Among them, animals can be but are not limited to tigers, chimpanzees, peacocks, pandas, sika deer, etc.

[0072] S104: Perform image clustering based on the combined feature information of each image combination and a pre-trained classifier.

[0073] The classifier can be pre-trained, and the specific training process will be described in the subsequent embodiments, which will not be described here. The trained classifier can be used to predict whether the image targets in two images in the image combination are the same object.

[0074] In the disclosed embodiment, image clustering may be performed based on the combined feature information of each image combination and a pre-trained classifier.

[0075] Exemplarily, the combined feature information of each image combination can be input into a pre-trained classifier, and the classifier can output whether the image targets in the two target images in the image combination are the same object. For example, when the clustering object is an animal, the classifier can output whether the image targets in the two target images in the image combination are the same animal. Furthermore, if the image targets in the two target images in the image combination are the same object, the two target images in the image combination are labeled with the same identifier; if they are not the same object, the two target images in the image combination are labeled with different identifiers. Using a classifier to determine whether the image targets in the two images in the image combination are the same object is equivalent to clustering the two images in the image combination.

[0076] Furthermore, according to the prediction result of whether the two target images in each image combination are of the same object, all the target images can be clustered, and the target images of the same object can be classified into one category to generate an image clustering result.

[0077] Exemplarily, assuming that in a combination of multiple images containing a same target image, when the image targets in two target images are the same object, the labeled identifiers are the same, then during image clustering, the same identifiers can be classified into one category to generate an image clustering result.

[0078] For example, assuming that the clustering object is a chimpanzee, for multiple image combinations (image A, image B), (image A, image C), (image D, image A) and (image A, image E), the image targets in image A, image B, image D and image E are the same chimpanzee, and the identifiers of each image in the image combinations (image A, image B), image D, image A) and (image A, image E) are the same, all of which are gID1. When performing image clustering, the target images with the same identifier of gID1 can be clustered into one category. When clustering, the target image can be characterized by its vector feature information.

[0079] Exemplarily, assuming that in a combination of multiple images containing the same target image, the labeled identifiers of the image targets in two target images when they are the same object are different, then when clustering the images, clustering can be performed based on the prediction results of whether the image targets are the same object, and the same identifier can be used to represent the image targets in the cluster.

[0080] For example, Figure 2 As shown, assuming that the clustering object is a tiger, for the image combinations (image A, image B), (image A, image E) and (image B, image C), the clustering results determined by the classifier are image A-same tiger-image B, image A-same tiger-image E and image B-same tiger-image C, wherein, for the image combination (image A, image B), the corresponding identifiers of image A and image B are gID1, for the image combination (image A, image E), the corresponding identifiers of image A and image E are gID2, and for the image combination (image B, image C), the corresponding identifiers of image B and image C are gID3. Since image A and image B are the same tiger, image A and image E are the same tiger, and images B and C are also the same tiger, image A, image B, image C and image E can be clustered into one category, and the clustering result containing image A, image B, image C and image E is marked as identifier gID1, or can also be marked as gID2 or gID3, and the present disclosure does not limit this.

[0081] In the disclosed embodiment, the target images to be clustered are combined in pairs to form multiple image combinations, and image clustering is performed based on the combined feature information of each image combination and a pre-trained classifier, thereby converting the clustering problem into several binary classification problems, which greatly reduces the complexity of the algorithm, but significantly improves the performance and clustering effect. By using a classifier to predict the same-object relationship between image combinations, the triples used to characterize the relationship between image objects in the two target images (such as image B-same tiger-image C) are pieced together into a relationship network (for example, Figure 2 As shown in the figure, the target image can be divided into different classes according to this relationship network. This method has strong interpretability, fast running speed, and reduces the difficulty of labeling data. Experiments have shown that the final clustering effect is significantly improved.

[0082] In the disclosed embodiment, target images to be clustered are obtained, including facial images and body images, and image feature information of the target images is obtained, including vector feature information and spatiotemporal feature information. Then, for an image combination consisting of any two target images, combined feature information of each image combination is obtained based on the image feature information of each target image, and image clustering is performed based on the combined feature information of each image combination and a pre-trained classifier. Thus, when performing image clustering, not only the vector feature information of the image is considered, but also the spatiotemporal feature information of the image is considered, so as to comprehensively judge whether the images in the image combination are of the same category, which is conducive to improving the accuracy of image clustering. Moreover, by combining the spatiotemporal feature information of the image for image clustering, the face and body that are actually the same object can be associated, which can not only improve the aggregation degree of image clustering, but also improve the accuracy of target object analysis in the image.

[0083] Usually, cameras installed in zoo areas, roads and other areas capture video streams. In an optional embodiment of the present disclosure, facial images and body images containing the face or body of the clustering object can be extracted from the video clip to obtain the target image to be clustered. The video clip can be a video stream captured by the camera, or a part of the clip intercepted from the video stream captured by the camera. When extracting facial images and body images from the video clip, the facial features or body features contained in the video frame can be identified from the video frame contained in the video clip, and the area where the face or body of the clustering object is located can be cropped out as the facial image or body image. It can be understood that when a video frame contains multiple faces or bodies, multiple facial images or body images can be obtained from a video frame.

[0084] In an optional embodiment of the present disclosure, when constructing an image combination, any two target images can be selected to construct the image combination, wherein the shooting time difference between the two target images in each image combination is less than a first time threshold, and / or the shooting position difference between the two images in each image combination is less than a first distance threshold.

[0085] The shooting position of the image can be represented by the position of the camera (the latitude and longitude of the camera), and the shooting position of two images taken by the same camera is the same. The first time threshold and the first distance threshold can be preset, for example, the first time threshold can be set to 10 seconds, and the first distance threshold is recommended to be set to 0.

[0086] It can be understood that when the shooting time difference between two images is large, the probability that the objects contained in the two images are not the same object will be relatively large. In addition, if the two images are not shot by the same camera, the probability that the objects contained in the two images are not the same object is also relatively large. Therefore, in the embodiment of the present disclosure, when constructing an image combination, any two target images can be selected to construct the image combination, and the shooting time difference and shooting position difference between any two target images can be determined based on the spatiotemporal feature information of each target image, and the image combination in which the shooting time difference between the two images is greater than or equal to the first time threshold is filtered out, and the image combination in which the shooting position difference between the two images is greater than or equal to the first distance threshold is filtered out, and only the shooting time difference between the two target images is less than the first time threshold, and / or the image combination in which the shooting position difference between the two images is less than the first distance threshold is retained for image clustering, thereby avoiding the waste of computing resources caused by image clustering of all image combinations.

[0087] In an optional embodiment of the present disclosure, the spatiotemporal feature information of the target image may include at least one of the following: the shooting spatiotemporal information of the video frame where the target image is located, the positioning information of the target image in the video clip, the appearance time and disappearance time of the target image in the video clip, the number of other target images around the target image in the video clip, the moving speed of the target image in the video clip, the trajectory direction of the target image in the video clip and the discrete coefficient of the fitted trajectory.

[0088] Among them, the shooting spatiotemporal information of the video frame where the target image is located may include the shooting time of the video frame where the target image is located, the longitude and latitude of the camera that shot the video frame; the positioning information of the target image in the video clip may include but is not limited to the capture time of the target image in the video clip, the position information of the target pixel in the target image, wherein the target pixel may be a pixel with obvious features on the image target, such as the nose, the top of the head, the tip of the tail, etc. When determining the position information of the target pixel, the xy coordinates can be established with the upper left corner or the lower right corner of the target image as a reference to determine the position information of the target pixel in the xy coordinate system; the appearance time and disappearance time of the target image in the video clip, that is, the target image The appearance time and disappearance time of the image target in the video clip; the number of other target images around the target image in the video clip, that is, the number of other faces or bodies around the face or body contained in the target image in the video clip, and the number of other faces or bodies can be determined by target detection technology; the moving speed of the target image in the video clip, that is, the moving speed of the face or body in the target image in the video clip; the trajectory direction of the target image in the video clip, that is, the moving trajectory direction of the face or body in the target image in the video clip, and the moving trajectory direction can be determined by fitting. After the moving trajectory direction is fitted, the discrete coefficient of the fitting trajectory is also determined accordingly.

[0089] Furthermore, in an optional embodiment of the present disclosure, the combined feature information of each image combination may include the image feature vector distance, and at least one of the following: a combination of target image types, a temporal overlap of the two target images in the video clip, a difference in the number of other target images around the two target images in the video clip, a difference in the moving speeds of the two target images in the video clip, and a similarity in the trajectories of the two target images in the video clip.

[0090] Among them, the image feature vector distance can be determined according to the vector feature information corresponding to the two target images in each feature combination; the type combination of target images includes a combination of face and face, a combination of face and body, and a combination of body and body. The type combination of target images in the feature combination can be determined according to whether the target image is a face image or a body image. If both target images are face images, the type combination of target images corresponding to the image combination is a combination of face and face; the overlap of the appearance time of the two target images in the video clip can be determined according to the appearance time and disappearance time of the two target images in the video clip respectively; the trajectory similarity of the two target images in the video clip can include the difference between the trajectory directions and the difference between the discrete coefficients of the fitted trajectories. The smaller the difference between the trajectory directions and the difference between the discrete coefficients of the fitted trajectories, the higher the trajectory similarity.

[0091] It can be understood that the smaller the distance between the image feature vectors, the higher the overlap in appearance time, the smaller the difference in the number of other target images around, the smaller the difference in moving speed, and the higher the trajectory similarity, the more similar the image targets in the two target images are.

[0092] In the disclosed embodiment, by obtaining the shooting time and space information of the video frame where the target image is located, the positioning information of the target image in the video clip, the appearance time and disappearance time of the target image in the video clip, the number of other target images around the target image in the video clip, the moving speed of the target image in the video clip, the trajectory direction of the target image in the video clip and at least one of the discrete coefficients of the fitted trajectory as time and space feature information, and determining the combined feature information based on the time and space feature information, and performing image clustering based on the combined feature information, when performing image clustering, the weak connection features such as the appearance and disappearance time, trajectory direction, and the number of surrounding objects are integrated into a decision data set, and the classifier is used to infer the strongly related relationship based on these features and perform the same object prediction, so as to improve the prediction accuracy of judging whether the objects in two images are the same object. The disclosed embodiment introduces feature data of multiple dimensions, combines time, space, and picture together, and constructs a more complete target tracking system, which significantly improves the target object recognition effect.

[0093] In an optional implementation manner of the present disclosure, the image feature information may further include at least one attribute feature and a confidence level corresponding to the attribute feature.

[0094] Optionally, the attribute feature may include but is not limited to at least one of growth stage, height, gender, and clothing color.

[0095] The height may refer to the height between the highest part of the image target and the lowest part of the image target in the target image.

[0096] Optionally, the attribute characteristics may also include at least one of body shape, color, and hair.

[0097] Taking animals as clustering objects as an example, animals of the same species at different growth stages usually have different characteristics, and animals of the same species of different genders also have different characteristics. Therefore, the corresponding growth stage and gender of the animal can be analyzed based on the identified characteristics. For example, a male sika deer grows small tumor-like horns at the age of two, grows into large horns at the age of three, splits into two forks at the age of four, and splits into three forks at the age of five. Therefore, by identifying whether the sika deer in the image has horns, whether the horns are forked, and the number of forks, the growth stage and gender of the sika deer can be identified.

[0098] Some owners will dress up their pets, such as putting beautiful clothes, shoes, and headdresses on their pet dogs. Therefore, in the embodiments of the present disclosure, the color of the clothing can also be used as an attribute feature, which can provide an analysis basis when performing image clustering through the image clustering method provided by the embodiments of the present disclosure to find lost pets.

[0099] In the disclosed embodiment, an attribute feature can be divided into multiple different categories. Taking the clustering object as an animal as an example, the animal can be but not limited to a tiger, chimpanzee, peacock, panda, zebra, clouded leopard, sika deer, pet dog, etc., and the attribute features of the animal may include growth stage, sex, body shape, color, hair and clothing color. Among them, the growth stage can be divided into multiple growth stages, such as infancy, adulthood and old age; or, the growth stage can also be divided into under 2 years old (excluding two years old), 2 to 3 years old (excluding three years old), 3 to 4 years old (excluding four years old), 4 to 6 years old (excluding 6 years old) and over 6 years old; body shape can be divided into small, medium and large; color can be divided into white, black, brown, leopard print, black and white, gray, color, etc.; hair can be divided into hairless, short hair and long hair, etc.; clothing color can include pink, white, red, etc., and clothing can include but is not limited to clothes, hats, shoes, accessories, etc.

[0100] In the disclosed embodiment, the image feature information may include at least one of the growth stage and the corresponding confidence, height and the corresponding confidence, gender and the corresponding confidence, clothing color and the corresponding confidence, body shape and the corresponding confidence, color and the corresponding confidence, hair and the corresponding confidence.

[0101] It can be understood that the confidence corresponding to the attribute feature can reflect the probability that the image target (the face or body of the clustering object) in the target image belongs to a certain category of the attribute feature. The greater the probability, the higher the possibility that the image target belongs to the category.

[0102] Taking the clustering object as animals and the attribute feature as color as an example, multiple color categories can be divided, such as white, black, brown, leopard print, black and white, gray and flowery. The probability that the face or body of the animal in the target image belongs to each color category can be obtained, and the obtained probability of each color category can be determined as the confidence of the corresponding color category, thereby obtaining the color and the corresponding confidence as an image feature information of the target image.

[0103] Taking the sika deer as an example, for a target image containing a sika deer, the color of the sika deer in the target image can be identified by a pre-trained color classification model. Assuming that the probabilities of the sika deer output by the color classification model belonging to each color category are: the probability of belonging to white is 0.14, the probability of belonging to black is 0, the probability of belonging to brown is 0.6, the probability of belonging to leopard print is 0.1, the probability of belonging to black and white is 0, the probability of belonging to gray is 0.06, and the probability of belonging to flower color is 0.1, then in the image feature information corresponding to the target image, the color feature and the corresponding confidence can be expressed as: color (confidence), that is, white (0.14), black (0), brown (0.6), leopard print (0.1), black and white (0), gray (0.06) and flower color (0.1).

[0104] Furthermore, based on the fact that the acquired image feature information includes at least one attribute feature and the confidence level corresponding to the attribute feature, in an optional embodiment of the present disclosure, the combined feature information of each image combination may include whether the attribute features whose confidence levels satisfy a preset threshold are the same.

[0105] The preset threshold may be predetermined, for example, the preset threshold is set to 0.5.

[0106] In the disclosed embodiment, when the image feature information of the target image includes attribute features and corresponding confidence levels, the combined feature information of each image combination obtained based on the image feature information may also include whether the attribute features whose confidence levels meet a preset threshold are the same.

[0107] For example, taking the clustering object as an animal, assuming that image A and image B constitute an image combination, the image feature information of image A and image B both include body attribute features and the confidence of the animals in image A and image B belonging to each body type, wherein the body type category that meets the preset threshold in the confidence of the animals in image A belonging to each body type is medium, if the body type category that meets the preset threshold in the confidence of the animals in image B belonging to each body type is also medium, then the combined feature information of the image combination may include the same body attribute. If the body type that meets the preset threshold in the confidence of the animals in image B belonging to each body type is small, then the combined feature information of the image combination may include different body attributes.

[0108] It should be noted that in the embodiments of the present disclosure, when the confidences of various categories of a certain attribute feature of the face or body in the target image do not meet the preset threshold, the category with the highest confidence can be compared with the category with the highest confidence of another target image or the category whose confidence meets the preset threshold to determine whether the two are the same and generate combined feature information.

[0109] In the disclosed embodiments, by obtaining the attribute features and the corresponding confidence levels of the target image as image feature information, and obtaining combined feature information based on the attribute features and the corresponding confidence levels, it is possible to determine whether clustering objects in two images are the same object from aspects such as the height, growth stage, color, hair, clothing color, etc. of the clustering objects, thereby providing a basis for predicting the same object, which is beneficial to improving the prediction effect and further improving the accuracy of image clustering.

[0110] The image clustering method provided by the embodiments of the present disclosure can be applied to scenarios such as analyzing the living habits and behavioral trajectories of animals in a zoo and searching for missing pets.

[0111] Taking the application of the image clustering method provided in the embodiment of the present disclosure to the scene of analyzing the living habits and behavioral trajectories of animals as an example, there are many kinds of animals mixed in the wildlife park, such as chimpanzees, pandas, Siberian tigers, jaguars, Asian elephants, clouded leopards, zebras, etc. There are many animals of each kind, and the living habits and behavioral trajectories of animals of different kinds are usually different. Even animals of the same kind may have individual differences in living habits and behavioral trajectories. In order to understand the living habits and behavioral trajectories of each animal, cameras can be installed in the wildlife park to shoot videos of each animal, and the image clustering method provided in the embodiment of the present disclosure can be used to perform cluster analysis on multiple images containing animals, and the living habits and behavioral trajectories of each animal can be analyzed based on the clustering results.

[0112] It should be noted that the clustering object described in the embodiment of the present disclosure may also be a person, the facial image may also be a human face image, and the body image may also be a human body image. The image clustering method provided in the embodiment of the present disclosure may also be applied to the scenario of searching for missing persons.

[0113] In the above-mentioned embodiment, the classifier can be pre-trained, and the training process is as follows: Figure 3 shown. Figure 3 A flow chart of a training method for a classifier for image clustering provided in accordance with an embodiment of the present disclosure. The training method for a classifier for image clustering can be performed by a training device for a classifier for image clustering. The device can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities, such as servers, cloud servers, smart phones, cameras and other terminal devices.

[0114] like Figure 3 As shown, the training method of the classifier for image clustering may include the following steps:

[0115] S201. Acquire a target image for training, where types of the target image include a face image and a body image.

[0116] Among them, the facial image may be an image containing the facial features of the clustering object, and the body image may be an image containing the body features of the clustering object. The clustering object may be an animal, or other objects with a face and a body. Taking the clustering object as an animal as an example, in the embodiment of the present disclosure, the body image may be an image in which the face of the animal is not captured, such as an image captured when the animal lowers its head, an image captured when the face of the animal is blocked by an obstruction, etc. The body image may also be an image containing the face of the animal but the facial features cannot be recognized due to insufficient clarity, and the present disclosure does not limit this.

[0117] In the disclosed embodiments, the target image to be clustered can be obtained from the images and videos captured by the camera. For example, the face or body image containing the clustering object can be screened out from all the images captured by the camera as the target image. For another example, a video clip of the most recent period (which can be one day, one week, one month, etc.) can be intercepted from the video captured by the camera, and a video frame image containing the face or body of the clustering object can be obtained from the video clip as the target image.

[0118] S202: Acquire image feature information of the target image, where the image feature information includes vector feature information and spatiotemporal feature information.

[0119] Among them, the vector feature information can be a vector feature determined based on the image target in the target image, the image target can be the face or body of the clustering object, and the spatiotemporal feature information can include but is not limited to the shooting time of the image, the longitude and latitude of the camera that shot the target image, etc.

[0120] In the disclosed embodiment, for the face image and the body image, vector feature information and spatiotemporal feature information can be obtained as the image feature information of the face image. In addition, due to shooting angles, occlusions and other reasons, the confidence of the vector distance between the feature vectors determined according to the body in different images is not high, and the vector distance between the feature vectors of the body of the same clustering object is not necessarily close. Therefore, for the body image, the vector feature information can also be ignored, and the spatiotemporal feature information can be used as the image feature information of the body image. By obtaining the spatiotemporal feature information such as the shooting time of the target image and the camera latitude and longitude, it is convenient to associate the body image with the face image. For example, in two images (such as a body image and a face image) with adjacent shooting times collected by the same camera, the probability that the clustering object in the body image and the clustering object in the face image belong to the same object will be very high. Even if there is a large difference between the vector feature information of the body image and the vector feature information of the face image, combined with the spatiotemporal feature information of the image, the body image and the face image may also be judged as a clustering object, so that the spatiotemporal feature information is considered when performing image clustering, which is conducive to improving the clustering accuracy.

[0121] Exemplarily, the vector feature information of the facial image can be obtained through a pre-trained neural network model. The facial image is input into the trained neural network model. The convolution kernel of the neural network model can extract the features of the face in the facial image, and then the vector feature information of the facial image can be obtained from the convolution kernel. The spatiotemporal feature information of the target image can be directly obtained. For example, the shooting time of the target image can be directly determined based on the acquisition time of the target image. Usually, an acquisition time watermark is added to the image captured by the camera, and the shooting time of the target image can be determined based on the acquisition time watermark; the longitude and latitude of the camera can be determined based on the installation location of the camera.

[0122] It should be noted that the clustering object described in the embodiments of the present disclosure may also be a person, the facial image may also be a human face image, and the body image may also be a human body image.

[0123] S203 . For an image combination consisting of any two target images, obtain combination feature information of each image combination based on the image feature information of each target image, and obtain a clustering labeling result of each image combination.

[0124] The combined feature information may include but is not limited to at least one of the image feature vector distance between the vector feature information, the structured type combination of the image target, the shooting time difference, the camera position difference, etc. There are three types of structured type combinations of image targets, namely, face-face combination, face-body combination, and body-body combination.

[0125] In the disclosed embodiment, when the image targets in the two images in a certain image combination are both faces, it can be determined that the structured type of the image target corresponding to the image combination is a combination of faces and faces, and the image feature vector distance corresponding to the image combination can be determined based on the existing vector distance calculation method according to the vector feature information corresponding to the two target images in the image combination. When the image combination includes a body image, since the feature vector information of the body image is affected by shooting angle, occlusion, etc., the confidence of the vector distance between the body feature vectors is not high, and therefore the image feature vector distance of the image combination is not calculated.

[0126] Optionally, when constructing image combinations, the target images used for training can be combined in pairs, and image combinations in which the shooting time difference is greater than a preset time threshold are filtered out, or image combinations in which the straight-line distance between the cameras that shot the target images is greater than a preset distance threshold are filtered out. The remaining image combinations are used as image combinations for training the classifier, and clustering annotation results of these image combinations are obtained.

[0127] Exemplarily, the clustering annotation result of each image combination may be manually calibrated.

[0128] In an optional embodiment of the present disclosure, the image feature information of the target image may further include attribute features and confidence levels corresponding to the attribute features, and the attribute features may include at least one of growth stage, height, gender, clothing color, body shape, color, and hair. Accordingly, the combined feature information of each image combination may further include whether the attribute features whose confidence levels meet a preset threshold are the same.

[0129] It can be understood that when training a classifier for clustering animals in an image, the target images used for training include animal face images and body images, and the structured type combination of image targets may include a combination of animal face and animal face, a combination of animal face and animal body, and a combination of animal body and animal body. The above animals may be, but are not limited to, tigers, chimpanzees, peacocks, pandas, zebras, etc.

[0130] It should be noted that the relevant descriptions about spatiotemporal feature information, attribute feature information, and combined feature information in the aforementioned embodiments are also applicable to this embodiment and will not be repeated here.

[0131] S204: Training the classifier based on the combined feature information of each image combination and the clustering labeling result of each image combination.

[0132] Among them, the classifier can be a binary classifier, such as an XGBoost classifier, a LightGBM (Light GradientBoosting Machine) classifier, etc.

[0133] When training a classifier, the combined feature information of each image combination can be used as the input data of the classifier, and the clustering annotation result corresponding to each image combination can be used as the output of the classifier. The classifier is iteratively trained, and the parameters of the classifier are continuously optimized during the training process until the prediction accuracy of the classifier reaches the preset value. The training is completed and a trained classifier is obtained. The trained classifier can be used to predict whether an unknown image combination is the same object.

[0134] In the disclosed embodiment, target images for training are obtained, including face images and body images, and image feature information of the target images is obtained, including vector feature information and spatiotemporal feature information. Then, for an image combination consisting of any two target images, combined feature information of each image combination is obtained based on the image feature information of each target image, and a clustering annotation result of each image combination is obtained. Then, based on the combined feature information of each image combination and the clustering annotation result of each image combination, a classifier is trained. Thus, in the input data for training the classifier, not only the vector feature information of the image but also the spatiotemporal feature information of the image is considered. The classifier obtained by training in this way can accurately predict whether two images are of the same object, which is conducive to improving the accuracy of image clustering. Moreover, the classifier is obtained by training in combination with the spatiotemporal feature information of the image. When the classifier is used for image clustering, the face and body that are actually the same object can be associated, which can not only improve the aggregation degree of image clustering, but also improve the accuracy of target object analysis in the image.

[0135] In an optional implementation of the present disclosure, a clustering common sense library can also be obtained as an input of the classifier, and the clustering common sense library is used for human intervention signals, and then the classifier is trained with labeled image data, which can effectively improve the evaluation indicators of the classifier.

[0136] Among them, some prior knowledge can be pre-constructed as a clustering common sense library. The content in the clustering common sense library can be used as a reference standard for judging whether the image targets in two images are the same object (such as animals), and / or can be used to guide the classifier when predicting whether the image targets in two images are the same object (such as animals), which feature information needs to be considered and which feature information does not need to be considered, etc.

[0137] For example, the content in the clustering common sense library may include but is not limited to the following:

[0138] (1) If the confidences of the attribute features corresponding to the two images in the image combination are both greater than a threshold, then the image targets in the two images can be judged as the same object (such as an animal) only when the categories of the attribute features with confidences greater than the threshold are the same. The same category of the attribute features means that if the attribute feature is height, the heights of the objects in the two images must belong to the same height range; if the attribute feature is growth stage, the growth stages of the objects in the two images must belong to the same growth stage; if the attribute feature is color, the color categories of the objects in the two images must be the same; if the attribute feature is clothing color, the clothing colors of the objects in the two images must be consistent.

[0139] (2) When the image combination includes a body image, the feature vector distance is not used, that is, for the judgment of the image combination including the body image, the feature vector distance is not used as the basis for judging whether the objects in the two images are the same object. It can be understood that whether the image combination includes a body image can be determined based on the combination feature information of the image combination, and the combination feature information can include the type combination of images, and the type combination is divided into a face-face combination, a face-body combination, and a body-body combination. Based on the type combination of images, it can be determined whether the image combination includes a body image.

[0140] (3) If the trajectory similarity corresponding to the image targets in two images in the image combination is greater than a similarity threshold, the image targets in the two images can be judged as the same object (such as an animal).

[0141] In the disclosed embodiment, a clustering common sense library can be constructed based on the combined feature information of the image combination. For example, if the combined feature information of the image combination includes a combination of image feature vector distance and the type of target image, the content of the constructed clustering common sense library may include not using the feature vector distance when the image combination includes a body image. For another example, if the combined feature information of the image combination includes the overlap of the appearance time of two target images in a video clip, the content of the constructed clustering common sense library may include that the objects in the two images can only be judged as the same object when the overlap of the appearance time is greater than a preset overlap threshold.

[0142] It is understandable that the content of the clustering common sense library can be flexibly adjusted according to the actual image quality, application scenarios, etc. For example, when the image quality is poor, the content in the clustering common sense library can be modified to relax the conditions for judging as the same object. For example, when the maximum confidence of the attribute feature is not greater than the threshold, the original content "only when the categories of the attribute features with confidence greater than the threshold are the same, the people in the two images can be judged as the same object" is modified to "when the categories of the attribute features corresponding to the maximum confidence are the same, the people in the two images are judged as the same object". For another example, in scenarios where the accuracy of target object analysis is required to be high, the content in the clustering common sense library can be increased to enrich the constraints for predicting the same target object and improve the evaluation indicators of the classifier.

[0143] Therefore, in the embodiment of the present disclosure, based on the combined feature information of each image combination and the clustering labeling result of each image combination, training the classifier may include:

[0144] The classifier is trained based on the combined feature information of each image combination, the clustering common sense library, and the clustering annotation result of each image combination.

[0145] In the disclosed embodiment, a clustering common sense library is introduced as a part of the classifier input when training the classifier, and the clustering common sense library is used for human intervention signals to constrain the same-object prediction process of the classifier. For example, the content in the clustering common sense library is that when the image combination includes a body image, the feature vector distance is not used. Then, in the process of training the classifier, for the combination of image combinations including body images, the feature vector distance of the image combination is not used as a basis for judging whether the persons in the two images in the image combination are the same object, which can improve the evaluation index of the classifier. This is because people know that due to shooting angle, occlusion and other reasons, the confidence of the vector distance between body feature vectors is not high, and it is meaningless to judge whether they are the same object based on the vector distance, but the classifier does not know this. Therefore, the clustering common sense library is introduced to inform the classifier that when the image combination includes a combination of body images, the feature vector distance of the image combination is not used as a basis for judgment, thereby effectively avoiding some obvious errors but difficult for the classifier to learn, thereby improving the evaluation accuracy of the classifier.

[0146] The image clustering method provided in the embodiments of the present disclosure has a wide range of applications, such as visual target analysis, finding missing pets, animal behavior trajectory analysis, person re-identification (ReID), finding missing people, personnel clustering, and other fields.

[0147] Taking the application of the image clustering method provided in the embodiment of the present disclosure in the field of visual target analysis as an example, Figure 4is a diagram of a device architecture supporting visual target analysis capabilities according to an embodiment of the present disclosure. Figure 4 The camera device in the embodiment integrates a classifier trained by the training method for a classifier for image clustering provided by the embodiment of the present disclosure, and an image clustering method provided by the embodiment of the present disclosure, so as to realize image clustering of target images to be clustered. Figure 4 As shown, the object of visual target analysis is a dog, rID1 is the trajectory 1 of visual analysis, and rID2 is the trajectory 2 of visual analysis. Through the image clustering method of the embodiment of the present disclosure, the fitting trajectory gID of rID1 and rID2 can be obtained, rID1 and rID2 are clustered into one category, and the cluster identifier is gID, which effectively improves the visual target analysis capability. Practical verification shows that the image clustering method provided by the present disclosure can achieve a high accuracy rate. Evaluation based on visual data proves that the accuracy rate reaches 99%, and the number of trajectories of visual target analysis increases by 10.4%, which improves the accuracy of target analysis.

[0148] By clustering the image clustering method described in the above embodiment, two different target images to be clustered acquired twice can be obtained, and two different clustering data sets are respectively recorded as a first clustering data set and a second clustering data set. For these two clustering data sets, the clustering data set merging method provided in the embodiment of the present disclosure can be used to merge the first clustering data set and the second clustering data set to obtain a merged clustering data set.

[0149] Figure 5 A flowchart of a method for merging clustered data sets provided in an embodiment of the present disclosure is provided. The method for merging clustered data sets can be executed by a device for merging clustered data sets provided in an embodiment of the present disclosure. The device can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities, such as servers, cloud servers, smart phones, cameras and other terminal devices.

[0150] like Figure 5 As shown, the method for merging the clustering data set may include the following steps:

[0151] S301. Obtain a first clustering data set and a second clustering data set, wherein the first clustering data set includes a plurality of first feature vectors and at least one first cluster identifier obtained by clustering the plurality of first feature vectors, and the second clustering data set includes the plurality of first feature vectors and at least one second feature vector, and at least one second cluster identifier obtained by clustering the plurality of first feature vectors and the at least one second feature vector.

[0152] Among them, the first clustering data set and the second clustering data set can be obtained by clustering multiple images using the image clustering method described in the aforementioned embodiment. The first feature vector and the second feature vector can be the image feature information corresponding to the target image described in the aforementioned embodiment. The clustering identifier refers to the identifier corresponding to each cluster obtained by clustering. The clustering identifier can be used to identify different target objects (such as animals). Each cluster includes at least one feature vector. The number of feature vectors contained in the cluster can be regarded as the number of images corresponding to the target object represented by the clustering identifier.

[0153] It can be understood that the first target image set used for clustering to generate the first cluster data set is different from the second target image set used for clustering to generate the second cluster data set.

[0154] For example, the first target image set may be a set of images acquired by eight cameras with camera identifiers 1-8, and the second target image set may be a set of images acquired by eight cameras with camera identifiers 3-10.

[0155] For another example, the first target image set may be a set of images captured by the camera within the time period [Tn, T], and the second target image set may be a set of images captured by the camera within the time period [T-n+1, ​​T+1].

[0156] S302: Perform diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, where the cluster-affected eigenvector group includes at least one first eigenvector.

[0157] In the embodiment of the present disclosure, at least one second eigenvector is data newly added to the second clustering data set compared to the first clustering data set. Diffusion processing can be performed based on the at least one second eigenvector to obtain a set of eigenvectors affected by clustering.

[0158] Exemplarily, at least one first feature vector of the second clustering identifier that is specifically the same as at least one second feature vector in the second clustering data set can be determined as the clustering affected feature vector group. When at least one second feature vector belongs to a single class, that is, no first feature vector belongs to the same class as the second feature vector, the clustering affected feature vector group is empty.

[0159] S303: Based on the at least one second feature vector and the feature vector group affected by the cluster, update the second cluster identifier in the second cluster data set to obtain a merged third cluster data set.

[0160] Since at least one second eigenvector is a newly added eigenvector compared to the first clustering data set, the group of eigenvectors affected by clustering is the eigenvector affected by the newly added second eigenvector, that is, the group of eigenvectors affected by clustering is data whose cluster identification may change. Therefore, in the embodiment of the present disclosure, re-clustering can be performed based on at least one second eigenvector and the group of eigenvectors affected by clustering to obtain a new cluster identification, so that at least one second eigenvector and the group of eigenvectors affected by clustering correspond to the new cluster identification, and the remaining first eigenvectors still use the old cluster identification, and then the new cluster identification and the second cluster identification corresponding to the remaining first eigenvectors are used to construct the third clustering data set, thereby realizing the update of the cluster identification in the second clustering data set, keeping most of the original clustering results unchanged, and the clustering accuracy is not affected.

[0161] Clustering results are often used in various business scenarios, such as personnel file creation, organizational discovery, etc. A stable clustering (clustering identifiers remain unchanged as much as possible) is crucial. In the clustering methods currently used, clustering based on incremental data usually changes the original classes, resulting in a decrease in clustering accuracy; while the method of full clustering based on the latest data will cause the identifiers of multiple classes in the clustering results to change due to the addition of new data. The disclosed embodiment proposes a selective clustering result merging method, which performs diffusion processing based on at least one second eigenvector to obtain a clustering-affected feature vector group, and then updates the second cluster identifier in the second clustering data set based on at least one second eigenvector and the clustering-affected feature vector group to obtain a merged third clustering data set, thereby realizing the merging of multiple clustering results and filling the gap in clustering merging for multiple data sets. In addition, by performing clustering updates based on the second eigenvector and the affected feature vector group, the cluster identifiers corresponding to the remaining first eigenvectors remain unchanged, keeping most of the original clustering results unchanged.

[0162] In an optional implementation of the present disclosure, performing diffusion processing based on the at least one second feature vector to obtain a feature vector group affected by clustering may include:

[0163] Acquire a second cluster identifier associated with the second feature vector in the second cluster data set as a first association identifier;

[0164] A first feature vector associated with the first association identifier in the second clustering data set is obtained as a first-order affected feature vector, and the clustering affected feature vector group includes the first-order affected feature vector.

[0165] That is, the first eigenvector in the second cluster data set that has the same second cluster identifier as the second eigenvector is determined as the first-order affected eigenvector.

[0166] In an optional implementation of the present disclosure, N times of influence diffusion operations may be performed to obtain at least two-order affected feature vectors, and clustered affected feature vector groups may be constructed based on the affected feature vectors of each order.

[0167] The value of N is a positive integer.

[0168] Specifically, performing N impact diffusion operations includes:

[0169] Acquire a first cluster identifier associated with the 2N-1 order affected feature vector in the first cluster data set as a second association identifier;

[0170] Acquire a first feature vector associated with the second association identifier in the first clustering data set as a 2N-order affected feature vector;

[0171] Acquire a second clustering identifier associated with the 2N-order affected feature vector in the second clustering data set as a third association identifier;

[0172] A first feature vector associated with the third association identifier in the second clustering data set is obtained as a 2N+1-order affected feature vector.

[0173] It can be understood that performing N times of influence diffusion operation is performed on the basis of performing one diffusion process to obtain the first-order affected feature vector, and the cluster affected feature vector group includes at least the first-order affected feature vector. Each time an influence diffusion operation is performed, it is necessary to perform one diffusion process in the first cluster data set and the second cluster data set respectively to obtain two new affected feature vectors, and two new affected feature vectors are added to the cluster affected feature vector group.

[0174] Exemplarily, the diffusion order can be pre-set, and the affected eigenvectors of each order can be determined according to the diffusion order, and then the affected eigenvector group of clustering can be determined based on each order of affected eigenvectors. Wherein, the diffusion order satisfies the formula α=1+2*n, n=0,1,2,…, and the diffusion order can be flexibly adjusted according to the actual data volume, time consumption, etc. When n=0, the first eigenvector in the second clustering data set that has the same second clustering identifier as the second eigenvector is determined as the first-order affected eigenvector, and the affected eigenvector group of clustering includes the first-order affected eigenvector. When n=1, an influence diffusion operation is performed once, and the first cluster identifier associated with the first-order affected feature vector in the first cluster data set is obtained as the second association identifier, and the first feature vector associated with the second association identifier in the first cluster data set is obtained as the second-order affected feature vector, and the second cluster identifier associated with the second-order affected feature vector in the second cluster data set is obtained as the third association identifier, and the first feature vector associated with the third association identifier in the second cluster data set is obtained as the third-order affected feature vector, and a cluster-affected feature vector group is constructed based on the first-order affected feature vector, the second-order affected feature vector, and the third-order affected feature vector. The diffusion processing process when n is other values ​​is similar to the above process and will not be described one by one. By cyclically executing the above diffusion processing operation until the preset diffusion order is met, the cluster-affected feature vector group in the second cluster data set is obtained, and the cluster-affected feature vector group and at least one second feature vector are the objects that need to update the cluster identifier.

[0175] Combine the following Figure 6 An example is given to illustrate the specific process of obtaining the affected eigenvectors of each order.

[0176] Figure 6 Schematic diagram of the first cluster data set and the second cluster data set, assuming that the diffusion order is 3. Figure 6In the example, rID_new1, rID_new2 and rID_new3 are the incremental data (denoted as delta) in the second clustering data set compared to the first clustering data set. First, according to delta, the second clustering data set is traversed to find the clustering identifiers pID1_1 and pID1_2 corresponding to delta, and pID1_1 and pID1_2 are written into the hash table and named pID_set1, that is, pID_set1 = {pID1_1, pID1_2}; based on pID_set1, the second clustering data set is traversed again to find all the cluster identifiers in pID1_1 and pID1_2. The first-order affected stock data (i.e., data other than incremental data) rID1_1, rID1_2, and rID1_3 are used to obtain the first-order affected feature vector, which is recorded as rID_set1, i.e., rID_set1 = {rID1_1, rID1_2, rID1_3}; based on rID_set1, the first clustering data set is traversed, and the first clustering identifiers pID2_1 and pID2_2 associated with each data in rID_set1 are written into the hash table, named pID_set2, i.e., pID_set2 = {pID2_1, pID2_2}, and based on pID_set t2, find all the data rID1_1, rID1_2, rID1_3 and rID2_1 affected by the second order in the first clustering data set, and obtain the second order affected feature vector, recorded as rID_set2, that is, rID_set2 = {rID1_1, rID1_2, rID1_3, rID2_1}; write the second clustering identifiers pID12_2 and pID3_1 corresponding to rID_set2 in the second clustering data set into the hash table, named pID_set3, that is, pID_set3 = {pID12_2, pID3_1}, and traverse the second clustering data Set, use pID_set3 to find all the data rID1_1, rID1_2, rID1_3, rID2_1 and rID3_1 affected by the third order, and get the third-order affected feature vector, recorded as rID_set3, that is, rID_set3 = {rID1_1, rID1_2, rID1_3, rID2_1, rID3_1}. At this point, the diffusion order meets the preset diffusion order 3, and the diffusion processing is stopped to obtain the cluster-affected feature vector group, which includes the first-order affected feature vector, the second-order affected feature vector and the third-order affected feature vector.

[0177] In the embodiment of the present disclosure, an adjustable diffusion order is set, and at least one diffusion process is performed based on the diffusion order to obtain at least one order of affected feature vectors, thereby increasing flexibility and universality in practical applications.

[0178] The first clustering data set and the second clustering data set contain a portion of the same data, namely, multiple first eigenvectors, but since the first clustering data set and the second clustering data set are not exactly the same, even for the same data, after clustering, the cluster identifiers corresponding to the same data are not necessarily the same. In order to make the clustering results as stable as possible, it is necessary to keep most of the cluster identifiers unchanged, that is, for the new clustering results, try to use the cluster identifiers in the old clustering results. In order to achieve this goal, in an optional embodiment of the present disclosure, the class identifier of the class in which only the cluster identifier has changed but the objects in the class have not changed can also be changed to the cluster identifier in the old clustering result in the following way. Because the objects in the class have not changed, the clustering accuracy is not affected. The steps of changing the class identifier of the class in which only the cluster identifier has changed but the objects in the class have not changed include:

[0179] Establishing a first mapping dictionary of the first cluster identifiers and the first feature vectors in the first cluster data set, and a second mapping dictionary of the first cluster identifiers and the number of first feature vectors corresponding thereto;

[0180] Obtaining a second cluster identifier corresponding to the second feature vector in the second cluster data set, and a correspondence table between the second cluster identifier and the first feature vector;

[0181] Based on the correspondence table between the second cluster identifier and the first feature vector and the first mapping dictionary, generate a third mapping dictionary including a mapping relationship between the second cluster identifier and the first cluster identifier, and establish a fourth mapping dictionary for the number of the first feature vectors corresponding to the second cluster identifier;

[0182] When the mapping relationship between the second cluster identifier and the first cluster identifier in the third mapping dictionary is unique, and the number of first feature vectors corresponding to the first cluster identifier with the unique mapping relationship in the second mapping dictionary is the same as the number of first feature vectors corresponding to the second cluster identifier in the fourth mapping dictionary, the second cluster identifier is updated to the first cluster identifier.

[0183] Among them, the correspondence table between the second cluster identifier and the first feature vector refers to the correspondence table between the target second cluster identifier corresponding to the second feature vector in the second cluster data set and the target first feature vector contained in the target second cluster identifier, that is, the second cluster identifier in the correspondence table is the target second cluster identifier corresponding to the second feature vector, and the first feature vector in the correspondence table is the target first feature vector having the same second cluster identifier as the second feature vector. The fourth mapping dictionary can be determined based on the correspondence table between the second cluster identifier and the first feature vector.

[0184] In the disclosed embodiment, the third mapping dictionary records the correspondence between the second cluster identifier and the first cluster identifier. A second cluster identifier may not have a corresponding first cluster identifier, may have a unique corresponding first cluster identifier, or may have at least two corresponding first cluster identifiers. When any second cluster identifier in the third mapping dictionary does not have a corresponding first cluster identifier, or when there are at least two corresponding first cluster identifiers, the second cluster identifier is kept unchanged when the first cluster data set and the second cluster data set are merged to obtain the third cluster data set; when any second cluster identifier in the third mapping dictionary has a unique corresponding first cluster identifier, it is further determined in the fourth mapping dictionary whether the number of first feature vectors corresponding to the second cluster identifier with a unique mapping relationship is the same as the number of first feature vectors corresponding to the first cluster identifier in the second mapping dictionary. If they are the same, the second cluster identifier is updated to the first cluster identifier. In other words, for the classes in the second cluster data set whose mapping relationship of cluster identifiers is unique and whose corresponding number of first feature vectors is also the same, the cluster identifier in the first cluster data set is used to ensure the stability of the clustering results as much as possible.

[0185] To facilitate understanding of the above process, an example is given below to illustrate.

[0186] Assume that the first mapping dictionary is as shown in Table 1, the second mapping dictionary is as shown in Table 2, and the second clustering data set includes the following classes: pID1 (rID1_1, rID1_2), pID2 (rID2_1, rID2_2, rID2_3, rID3_1, rID4_3), pID5 (rID4_1, rID4_2, rID5_1) and pID6 (rID6_1), among which rID4_3, rID5_1 and rID6_1 are newly added feature vectors, i.e., second feature vectors. The second clustering identifiers corresponding to the second feature vectors are pID2, pID5 and pID6. Based on the first feature vectors corresponding to pID2, pID5 and pID6 respectively, a correspondence table between the second clustering identifier and the first feature vector can be established, as shown in Table 3. Based on the correspondence table between the second cluster identifier and the first feature vector shown in Table 3 and the first mapping dictionary shown in Table 1, a third mapping dictionary including the mapping relationship between the second cluster identifier and the first cluster identifier can be generated, as shown in Table 4. Based on the correspondence table between the second cluster identifier and the first feature vector shown in Table 3, a fourth mapping dictionary including the number of first feature vectors corresponding to the second cluster identifier and the first feature vector can also be established, as shown in Table 5. As can be seen from Table 4, the second cluster identifiers pID2 and pID6 are not uniquely corresponding to the first cluster identifier. Therefore, in the second clustering data set, the second cluster identifiers pID2 and pID6 are kept unchanged, the second cluster identifier pID5 is uniquely corresponding to the first cluster identifier pID4, and the number of first feature vectors corresponding to pID5 and pID4 is the same, both 2, which means that the object in the class corresponding to pID5 has not changed, only the identifier has changed, and pID5 in the second clustering data set can be updated to pID4 in the first clustering data set, so that the cluster identifier in the old clustering result is used in the new clustering result, and the clustering accuracy is not affected.

[0187] Table 1

[0188] First cluster identification The first eigenvector pID1 rID1_1, rID1_2 pID2 rID2_1, rID2_2, rID2_3 pID3 rID3_1 pID4 rID4_1, rID4_2

[0189] Table 2

[0190] First cluster identification The number of first eigenvectors pID1 2 pID2 3 pID3 1 pID4 2

[0191] Table 3

[0192] Second cluster identification The first eigenvector pID2 rID2_1, rID2_2, rID2_3, rID3_1 pID5 rID4_1, rID4_2 pID6 null

[0193] Table 4

[0194] Second cluster identification First cluster identification pID2 pID2, pID3 pID5 pID4 pID6 null

[0195] Table 5

[0196] Second cluster identification The number of first eigenvectors pID2 4 pID5 2 pID6 0

[0197] In an optional implementation of the present disclosure, the first clustering data set further includes at least one third feature vector, and the at least one third feature vector and the plurality of first feature vectors are used for clustering to obtain at least one first cluster identifier.

[0198] Among them, the third eigenvector can be the data before the first eigenvector. For example, for a data set with time attributes, each time image clustering is performed, the images of the target image to be clustered that were captured in the last 10 days are clustered every morning. The first cluster data set obtained by clustering this morning is based on the eigenvectors corresponding to the images captured in the [t-10, t-1] time period. When tomorrow morning arrives, the second cluster data set is clustered based on the eigenvectors corresponding to the images captured in the [t-9, t] time period. Among them, t is today, then when tomorrow arrives, the eigenvector corresponding to the image captured on (t-10) is the third eigenvector, the eigenvector corresponding to the image captured today (t) is the second eigenvector, and the eigenvector corresponding to the image captured in the [t-9, t-1] time period is the first eigenvector.

[0199] Usually, in the scenario of population management products, multidimensional feature vectors have time attributes, and because of the limitation of storage space, multidimensional feature vectors also have an validity period. Data stored for more than a certain time (such as 10 days, 30 days) will become invalid. Therefore, each time clustering is performed, the data set undergoes some changes, because some data becomes invalid and some new data is added. Therefore, when merging based on the first clustering data set and the second clustering data set, it is necessary to first delete the third feature vector in the first clustering data set to eliminate invalid data, and then use the remaining data in the first clustering data set to merge with the second clustering data set to ensure the accuracy of the merged result. It can be seen that the clustering data set merging method provided in the embodiment of the present disclosure can be applied to the clustering scenario of data with time attributes, and realizes the merging of multiple clustering data sets, which makes up for the defect that the current technology cannot solve the clustering problem of data sets with time attributes and only clusters a single data set.

[0200] Figure 7 This is a schematic diagram of the result of merging the first clustering data set and the second clustering data set. Figure 7 In the example, each feature vector in the area corresponding to identifier 1 is the third feature vector, each feature vector in the area corresponding to identifier 2 is the first feature vector in the first clustering data set, each feature vector in the area corresponding to identifier 3 is the first feature vector in the second clustering data set, and each feature vector in the area corresponding to identifier 24 is the second feature vector in the second clustering data set. In the first clustering data set and the second clustering data set, the same pattern identifies the same objects in the same class. When merging, each third feature vector in the first clustering data set does not participate in the merging. Figure 7 It can be seen that in the final merge result, for classes with different cluster identifiers but the same objects in the first clustering data set and the second clustering data set, the cluster identifier corresponding to the class in the first clustering data set continues to be used. For classes with changed objects in the class, such as the class where the newly added second eigenvector is located, the cluster identifier corresponding to the class in the second clustering data set is used. Thus, the clustering accuracy is not affected while the stability of the cluster identifier is also guaranteed. Practical verification shows that, taking the evaluation of the population management algorithm as an example (data validity is 30 days, and clusters are merged once a day), the cluster merging of each cycle is completed using the method provided by the embodiment of the present disclosure, and only about 7% of the classes have changed, and the cluster identifiers of most classes remain stable.

[0201] Figure 8 This is a schematic diagram of the structure of an image clustering device provided in one embodiment of the present disclosure. The device can be implemented using software and / or hardware and can be integrated on any electronic device with computing capabilities, such as servers, cloud servers, smart phones, cameras and other terminal devices.

[0202] like Figure 8 As shown, the image clustering device 400 provided in the embodiment of the present disclosure may include an image acquisition module 401, an image feature acquisition module 402, a combined feature acquisition module 403 and a clustering module 404, wherein:

[0203] An image acquisition module 401 is used to acquire a target image to be clustered, where the types of the target image include a face image and a body image;

[0204] An image feature acquisition module 402 is used to acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0205] A combination feature acquisition module 403 is used to acquire, for an image combination consisting of any two target images, combination feature information of each image combination based on image feature information of each target image;

[0206] The clustering module 404 is used to perform image clustering based on the combined feature information of each image combination and a pre-trained classifier.

[0207] Optionally, the image acquisition module 401 is specifically configured to extract a facial image and a body image from a video clip to obtain the target image to be clustered.

[0208] Optionally, the image clustering device further includes:

[0209] The image combination construction module is used to select any two target images to construct an image combination, wherein the shooting time difference between the two target images in each image combination is less than a first time threshold, and / or the shooting position difference between the two images in each image combination is less than a first distance threshold.

[0210] Optionally, the spatiotemporal feature information of the target image includes at least one of the following:

[0211] The shooting time and space information of the video frame where the target image is located, the positioning information of the target image in the video clip, the appearance time and disappearance time of the target image in the video clip, the number of other target images around the target image in the video clip, the moving speed of the target image in the video clip, the trajectory direction of the target image in the video clip and the discrete coefficient of the fitted trajectory.

[0212] Optionally, the combined feature information of each image combination includes an image feature vector distance, and at least one of the following:

[0213] The combination of target image types, the overlap in the appearance time of the two target images in the video clip, the difference in the number of other target images around the two target images in the video clip, the difference in the moving speed of the two target images in the video clip, and the similarity of the trajectories of the two target images in the video clip.

[0214] Optionally, the image feature information further includes at least one attribute feature and a confidence level corresponding to the attribute feature.

[0215] Optionally, the attribute feature includes at least one of growth stage, height, gender, and clothing color.

[0216] Optionally, the combined feature information of each image combination includes whether attribute features whose confidence level satisfies a preset threshold are the same.

[0217] Optionally, the image clustering device further includes a clustering result merging module, which is used to:

[0218] Acquire a first clustering data set and a second clustering data set, wherein the first clustering data set includes a plurality of first feature vectors and at least one first cluster identifier obtained by clustering the plurality of first feature vectors, and the second clustering data set includes the plurality of first feature vectors and at least one second feature vector, and at least one second cluster identifier obtained by clustering the plurality of first feature vectors and the at least one second feature vector;

[0219] Performing diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, wherein the cluster-affected eigenvector group includes at least one first eigenvector;

[0220] Based on the at least one second feature vector and the feature vector group affected by the cluster, the second cluster identifier in the second cluster data set is updated to obtain a merged third cluster data set.

[0221] The first clustering data set and the second clustering data set are obtained by the image clustering device, and the clustering result merging module can obtain the first clustering data set and the second clustering data set from the clustering module 404 .

[0222] The image clustering device provided in the embodiments of the present disclosure that can be configured on an electronic device such as a camera can execute any image clustering method provided in the embodiments of the present disclosure that can be applied to an electronic device such as a camera, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not fully described in the embodiments of the device of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.

[0223] Fig. 9 A schematic diagram of the structure of a training device for a classifier for image clustering provided in one embodiment of the present disclosure; the device can be implemented using software and / or hardware and can be integrated on any electronic device with computing capabilities, such as servers, cloud servers, smart phones, cameras and other terminal devices.

[0224] like Fig. 9 As shown, the training device 500 for a classifier for image clustering provided by an embodiment of the present disclosure may include a training image acquisition module 501, a feature information acquisition module 502, a combination annotation module 503 and a training module 504, wherein:

[0225] A training image acquisition module 501 is used to acquire a target image for training, where the types of the target image include a face image and a body image;

[0226] A feature information acquisition module 502 is used to acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information;

[0227] A combination annotation module 503 is used to obtain, for an image combination consisting of any two target images, combination feature information of each image combination based on image feature information of each target image, and obtain a cluster annotation result of each image combination;

[0228] The training module 504 is used to train the classifier based on the combined feature information of each image combination and the clustering labeling result of each image combination.

[0229] Optionally, the training device for the classifier for image clustering further includes:

[0230] A common sense base acquisition module is used to acquire a cluster common sense base;

[0231] The training module 504 is specifically used to train the classifier based on the combined feature information of each image combination, the clustering common sense library, and the clustering annotation result of each image combination.

[0232] The training device for a classifier for image clustering that can be configured on an electronic device such as a camera provided in the embodiments of the present disclosure can execute any training method for a classifier for image clustering that can be applied to an electronic device such as a camera provided in the embodiments of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. For contents not fully described in the embodiments of the device of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.

[0233] Fig.10 This is a schematic diagram of the structure of a device for merging clustered data sets provided in one embodiment of the present disclosure. The device can be implemented using software and / or hardware and can be integrated on any electronic device with computing capabilities, such as servers, cloud servers, smart phones, cameras and other terminal devices.

[0234] like Fig.10 As shown, the clustering data set merging device 600 provided in the embodiment of the present disclosure may include a data set acquisition module 601, a diffusion processing module 602 and an update module 603, wherein:

[0235] The data set acquisition module 601 is used to acquire a first clustering data set and a second clustering data set, wherein the first clustering data set includes multiple first feature vectors and at least one first cluster identifier obtained by clustering the multiple first feature vectors, and the second clustering data set includes the multiple first feature vectors and at least one second feature vector, and at least one second cluster identifier obtained by clustering the multiple first feature vectors and the at least one second feature vector.

[0236] The first clustering data set and the second clustering data set may be obtained by the image clustering method described in the aforementioned embodiment.

[0237] A diffusion processing module 602 is used to perform diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, wherein the cluster-affected eigenvector group includes at least one first eigenvector;

[0238] The updating module 603 is configured to update the second cluster identifier in the second cluster data set based on the at least one second feature vector and the feature vector group affected by the cluster, so as to obtain a merged third cluster data set.

[0239] Optionally, the diffusion processing module 602 is specifically used for:

[0240] Acquire a second cluster identifier associated with the second feature vector in the second cluster data set as a first association identifier;

[0241] A first feature vector associated with the first association identifier in the second clustering data set is obtained as a first-order affected feature vector, and the clustering affected feature vector group includes the first-order affected feature vector.

[0242] Optionally, the cluster data set merging device further includes:

[0243] An execution module, used for executing N times of influence diffusion operation to obtain at least two-order affected feature vectors, where the value of N is a positive integer;

[0244] The vector group building module is used to build clustering-affected feature vector groups based on affected feature vectors of each order.

[0245] Optionally, the execution module is specifically used to:

[0246] Acquire a first cluster identifier associated with the 2N-1 order affected feature vector in the first cluster data set as a second association identifier;

[0247] Acquire a first feature vector associated with the second association identifier in the first clustering data set as a 2N-order affected feature vector;

[0248] Acquire a second clustering identifier associated with the 2N-order affected feature vector in the second clustering data set as a third association identifier;

[0249] A first feature vector associated with the third association identifier in the second clustering data set is obtained as a 2N+1-order affected feature vector.

[0250] Optionally, the cluster data set merging device further includes:

[0251] A dictionary building module, used to build a first mapping dictionary of the first cluster identifiers and the first feature vectors in the first cluster data set, and a second mapping dictionary of the first cluster identifiers and the number of first feature vectors corresponding thereto;

[0252] A relationship table acquisition module, used to acquire a second cluster identifier corresponding to the second feature vector in the second cluster data set, and a corresponding relationship table between the second cluster identifier and the first feature vector;

[0253] a dictionary generating module, configured to generate a third mapping dictionary including a mapping relationship between the second cluster identifier and the first cluster identifier based on the corresponding relationship table between the second cluster identifier and the first feature vector and the first mapping dictionary, and to establish a fourth mapping dictionary including the second cluster identifier and the number of the first feature vectors corresponding thereto;

[0254] An identifier updating module is used to update the second cluster identifier to the first cluster identifier when the mapping relationship between the second cluster identifier and the first cluster identifier in the third mapping dictionary is unique, and the number of first feature vectors corresponding to the first cluster identifier with the unique mapping relationship in the second mapping dictionary is the same as the number of first feature vectors corresponding to the second cluster identifier in the fourth mapping dictionary.

[0255] Optionally, the first clustering data set further includes at least one third eigenvector, and the at least one third eigenvector and the plurality of first eigenvectors are used for clustering to obtain at least one first cluster identifier.

[0256] The clustering data set merging device provided in the embodiments of the present disclosure and which can be configured on an electronic device such as a camera can execute any clustering data set merging method provided in the embodiments of the present disclosure and which can be applied to an electronic device such as a camera, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not fully described in the embodiments of the device of the present disclosure, reference can be made to the description in any method embodiment of the present disclosure.

[0257] Fig.11 A structural schematic diagram of an electronic device provided for an embodiment of the present disclosure is used to exemplify an electronic device for implementing a method for clustering any image in an embodiment of the present disclosure, or a training method for implementing any classifier for image clustering in an embodiment of the present disclosure, or a method for merging any clustering data sets in an embodiment of the present disclosure, and should not be understood as a specific limitation on the embodiments of the present disclosure.

[0258] like Fig.11As shown, the electronic device 700 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0259] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 700 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0260] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processor 701, it can execute the functions defined in any image clustering method provided by the embodiment of the present disclosure, or execute the functions defined in any training method for a classifier for image clustering provided by the embodiment of the present disclosure, or execute the functions defined in any clustering data set merging method provided by the embodiment of the present disclosure.

[0261] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0262] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0263] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0264] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains target images to be clustered, the types of the target images include face images and body images; obtains image feature information of the target images, the image feature information includes vector feature information and spatiotemporal feature information; for an image combination consisting of any two target images, obtains combined feature information of each image combination based on the image feature information of each target image; and performs image clustering based on the combined feature information of each image combination and a pre-trained classifier. .

[0265] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains target images for training, the types of which include facial images and body images; obtains image feature information of the target images, the image feature information includes vector feature information and spatiotemporal feature information; for an image combination consisting of any two target images, obtains combined feature information of each image combination based on the image feature information of each target image, and obtains a clustering annotation result of each image combination; and trains the classifier based on the combined feature information of each image combination and the clustering annotation result of each image combination.

[0266] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a first clustering data set and a second clustering data set, wherein the first clustering data set includes multiple first feature vectors and at least one first clustering identifier obtained by clustering the multiple first feature vectors, and the second clustering data set includes the multiple first feature vectors and at least one second feature vector, and at least one second clustering identifier obtained by clustering the multiple first feature vectors and the at least one second feature vector; performs diffusion processing based on the at least one second feature vector to obtain a clustering-affected feature vector group, wherein the clustering-affected feature vector group includes at least one first feature vector; and updates the second clustering identifier in the second clustering data set based on the at least one second feature vector and the clustering-affected feature vector group to obtain a merged third clustering data set.

[0267] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0268] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0269] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0270] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0271] In the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0272] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0273] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0274] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for image clustering, characterized in that: include: Acquire target images to be clustered, where types of the target images include face images and body images; Acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information; For an image combination consisting of any two target images, obtaining combination feature information of each image combination based on the image feature information of each target image; Performing image clustering based on the combined feature information of each image combination and a pre-trained classifier; The method further includes: dividing the target image to be clustered to obtain a first clustering data set and a second clustering data set; merging the first clustering data set and the second clustering data set to obtain a merged clustering data set, wherein the first clustering data set includes multiple first feature vectors, and the multiple first feature vectors are used to obtain at least one first clustering identifier for clustering, the first feature vector is the image feature information, the second clustering data set includes the multiple first feature vectors and at least one second feature vector, and the multiple first feature vectors and the at least one second feature vector are used to obtain at least one second clustering identifier for clustering, and the second feature vector is the data newly added to the second clustering data set compared to the first clustering data set.

2. The method according to claim 1, characterized in that The step of obtaining a set of images to be clustered comprises: A face image and a body image are extracted from the video clip to obtain the target image to be clustered.

3. The method according to claim 2, characterized in that The method further comprises: Any two target images are selected to construct an image combination, wherein a shooting time difference between the two target images in each image combination is less than a first time threshold, and / or a shooting position difference between the two images in each image combination is less than a first distance threshold.

4. The method according to claim 2, characterized in that: The spatiotemporal feature information of the target image includes at least one of the following: The shooting time and space information of the video frame where the target image is located, the positioning information of the target image in the video clip, the appearance time and disappearance time of the target image in the video clip, the number of other target images around the target image in the video clip, the moving speed of the target image in the video clip, the trajectory direction of the target image in the video clip and the discrete coefficient of the fitted trajectory.

5. The method according to claim 4, characterized in that The combined feature information of each image combination includes an image feature vector distance, and at least one of the following: The combination of target image types, the overlap in the appearance time of the two target images in the video clip, the difference in the number of other target images around the two target images in the video clip, the difference in the moving speed of the two target images in the video clip, and the similarity of the trajectories of the two target images in the video clip.

6. The method according to claim 2, characterized in that The image feature information also includes at least one attribute feature and a confidence level corresponding to the attribute feature.

7. The method according to claim 6, characterized in that The attribute feature includes at least one of growth stage, height, gender, and clothing color.

8. The method according to claim 6, characterized in that The combined feature information of each image combination includes whether the attribute features whose confidence level satisfies a preset threshold are the same.

9. The method according to claim 1, characterized in that: The method further comprises: The first clustering data set and the second clustering data set are merged to obtain a merged clustering data set, including: Acquire the first clustering data set and the second clustering data set; Performing diffusion processing based on the at least one second eigenvector to obtain a cluster-affected eigenvector group, wherein the cluster-affected eigenvector group includes at least one first eigenvector; Based on the at least one second feature vector and the set of feature vectors affected by the cluster, the second cluster identifier in the second cluster data set is updated to obtain a merged third cluster data set, wherein the merged cluster data set includes the merged third cluster data set.

10. An image clustering device, comprising: An image acquisition module, used for acquiring target images to be clustered, wherein the types of the target images include face images and body images; An image feature acquisition module, used to acquire image feature information of the target image, wherein the image feature information includes vector feature information and spatiotemporal feature information; A combination feature acquisition module, for acquiring, for an image combination consisting of any two target images, combination feature information of each image combination based on image feature information of each target image; A clustering module, used for performing image clustering based on the combined feature information of each image combination and a pre-trained classifier; The device is also used to perform the following steps: dividing the target image to be clustered to obtain a first clustering data set and a second clustering data set; merging the first clustering data set and the second clustering data set to obtain a merged clustering data set, wherein the first clustering data set includes multiple first feature vectors, and the multiple first feature vectors are used to obtain at least one first clustering identifier for clustering, the first feature vector is the image feature information, the second clustering data set includes the multiple first feature vectors and at least one second feature vector, and the multiple first feature vectors and the at least one second feature vector are used to obtain at least one second clustering identifier for clustering, and the second feature vector is the data newly added to the second clustering data set compared to the first clustering data set.

11. An electronic device, comprising: processor; A memory, configured to store instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image clustering method described in any one of claims 1-9.

12. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image clustering method according to any one of claims 1 to 9.

Images