A portrait clustering method, device, equipment and medium

Abstract

The present application relates to the field of video surveillance technology, and more specifically to a portrait clustering method, apparatus, device, and medium for improving the accuracy of portrait clustering. The method comprises: obtaining a plurality of archives, wherein the plurality of archives are divided based on the similarity of a plurality of portrait images, wherein the portrait images are images of faces of people riding in vehicles, and at least one portrait image contained in each archive corresponds to at least one vehicle; generating a bipartite graph based on the plurality of vehicles contained in the plurality of archives and the number of portrait images corresponding to different vehicles contained in each archive; inputting the bipartite graph, a plurality of first eigenvectors associated with the plurality of archives, and a plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; and if the similarity of any two optimized first eigenvectors is greater than a clustering threshold, aggregating the two archives corresponding to the any two optimized first eigenvectors into one archive.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a method, device, equipment and medium for clustering human portraits. Background Art

[0002] With the widespread adoption of intelligent video surveillance equipment, massive amounts of portrait images are accumulated daily. These images consist of either facial features or full-body features (clothing, shoes, hats, etc.). Portrait clustering involves grouping portrait images of the same person captured at different times and locations into a single file. Based on the clustering results, person trajectory analysis can be performed.

[0003] Currently, portrait clustering methods primarily use similarity between images to group massive amounts of portrait images into profiles of different individuals. Due to image quality issues, all portraits of the same person may not be grouped into the same profile. Instead, all portraits of the same person may be scattered across two or even multiple profiles, resulting in low accuracy in portrait clustering. Summary of the Invention

[0004] Embodiments of the present application provide a portrait clustering method, apparatus, device, and medium for improving the accuracy of portrait clustering.

[0005] In a first aspect, the present application provides a portrait clustering method, comprising:

[0006] Acquire a plurality of files, wherein the plurality of files are divided based on similarity of a plurality of portrait images, the portrait images being images of faces of people riding in vehicles, and at least one portrait image contained in each file corresponds to at least one vehicle;

[0007] Generate a bipartite graph based on the number of vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file; wherein the portrait image corresponding to each vehicle is a portrait image of a person with the same license plate number in each file, and the bipartite graph is used to represent the relationship between the multiple files and the multiple vehicles;

[0008] Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; wherein each first eigenvector is used to indicate attribute information of a corresponding file, and each second eigenvector is used to indicate attribute information of a corresponding vehicle, and the plurality of optimized first eigenvectors are obtained by integrating the relationships between the plurality of files and the plurality of vehicles into the plurality of first eigenvectors;

[0009] If the similarity between any two optimized first feature vectors is greater than a clustering threshold, the two files corresponding to the any two optimized first feature vectors are aggregated into one file.

[0010] In a possible embodiment, generating a bipartite graph according to the plurality of vehicles included in the plurality of archives and the number of portrait images corresponding to different vehicles included in each archive includes:

[0011] According to the number of portrait images corresponding to different vehicles contained in each file, the relationship level between each file and different vehicles is determined;

[0012] A bipartite graph is generated according to the plurality of vehicles included in the plurality of archives and the relationship level between each archive and different vehicles.

[0013] In a possible embodiment, the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles are input into a graph neural network to obtain a plurality of optimized first eigenvectors, including:

[0014] Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network, and outputting predicted relationship levels between the plurality of files and the plurality of vehicles; wherein the graph neural network is used to integrate the relationships between the plurality of files and the plurality of vehicles and optimize the plurality of first eigenvectors and the plurality of second eigenvectors;

[0015] calculating a loss between actual relationship levels and the predicted relationship levels between the plurality of profiles and the plurality of vehicles;

[0016] The multiple first eigenvectors and the multiple second eigenvectors are adjusted according to the loss until the loss reaches a minimum value, thereby obtaining multiple optimized first eigenvectors.

[0017] In a possible embodiment, if the similarity between any two optimized first feature vectors is greater than a clustering threshold, before aggregating the two archives corresponding to the any two optimized first feature vectors into one archive, the method further includes:

[0018] Determining the clustering threshold according to the combined file similarity threshold, the similarity between any two optimized first feature vectors, and the maximum adjustable similarity threshold;

[0019] Among them, the combined file similarity threshold is a similarity threshold for determining whether two portrait pictures belong to the same file based on the similarity of the multiple portrait pictures, the maximum adjustable similarity threshold is used to indicate the lower limit of the similarity of two files as the same file, the combined file similarity threshold is positively correlated with the clustering threshold, the similarity of any two optimized first eigenvectors is negatively correlated with the clustering threshold, and the maximum adjustable similarity threshold is negatively correlated with the clustering threshold.

[0020] In a possible embodiment, before generating the bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, the method further includes:

[0021] If a vehicle of a preset type is included in a first portrait picture included in a first file among the multiple files, the first portrait picture is deleted.

[0022] In a possible embodiment, before generating the bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, the method further includes:

[0023] If the license plate numbers in at least one portrait image included in the second file among the multiple files are all the same, the second file is deleted.

[0024] In a second aspect, the present application provides a portrait clustering device, comprising:

[0025] an acquisition module, configured to acquire a plurality of files, wherein the plurality of files are divided based on similarity of a plurality of portrait images, the portrait images being images of faces of people riding in vehicles, and at least one portrait image contained in each file corresponds to at least one vehicle;

[0026] a generating module configured to generate a bipartite graph based on the plurality of vehicles included in the plurality of archives and the number of portrait images corresponding to different vehicles included in each archive; wherein the portrait image corresponding to each vehicle is a portrait image of a person with the same license plate number in each archive, and the bipartite graph is used to represent the relationship between the plurality of archives and the plurality of vehicles;

[0027] an obtaining module, configured to input the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; wherein each first eigenvector is used to indicate attribute information of a corresponding file, and each second eigenvector is used to indicate attribute information of a corresponding vehicle, and the plurality of optimized first eigenvectors are obtained by integrating the relationships between the plurality of files and the plurality of vehicles into the plurality of first eigenvectors;

[0028] The aggregation module is configured to aggregate two files corresponding to any two optimized first feature vectors into one file if the similarity between the two optimized first feature vectors is greater than a clustering threshold.

[0029] In a possible embodiment, the generating module is specifically configured to:

[0030] According to the number of portrait images corresponding to different vehicles contained in each file, the relationship level between each file and different vehicles is determined;

[0031] A bipartite graph is generated according to the plurality of vehicles included in the plurality of archives and the relationship level between each archive and different vehicles.

[0032] In a possible embodiment, the obtaining module is specifically configured to:

[0033] Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network, and outputting predicted relationship levels between the plurality of files and the plurality of vehicles; wherein the graph neural network is used to integrate the relationships between the plurality of files and the plurality of vehicles and optimize the plurality of first eigenvectors and the plurality of second eigenvectors;

[0034] calculating a loss between actual relationship levels and the predicted relationship levels between the plurality of profiles and the plurality of vehicles;

[0035] The multiple first eigenvectors and the multiple second eigenvectors are adjusted according to the loss until the loss reaches a minimum value, thereby obtaining multiple optimized first eigenvectors.

[0036] In a possible embodiment, the apparatus further includes a determining module, wherein the determining module is configured to:

[0037] If the similarity between any two optimized first feature vectors is greater than the clustering threshold, before aggregating the two files corresponding to the two optimized first feature vectors into one file, determine the clustering threshold based on the combined file similarity threshold, the similarity between the two optimized first feature vectors, and the maximum adjustable similarity threshold;

[0038] Among them, the combined file similarity threshold is a similarity threshold for determining whether two portrait pictures belong to the same file based on the similarity of the multiple portrait pictures, the maximum adjustable similarity threshold is used to indicate the lower limit of the similarity of two files as the same file, the combined file similarity threshold is positively correlated with the clustering threshold, the similarity of any two optimized first eigenvectors is negatively correlated with the clustering threshold, and the maximum adjustable similarity threshold is negatively correlated with the clustering threshold.

[0039] In a possible embodiment, the device further includes a deletion module, wherein the deletion module is configured to:

[0040] Before generating a bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, if a first portrait image included in a first file in the multiple files contains a vehicle of a preset type, the first portrait image is deleted.

[0041] In a possible embodiment, the deletion module is further configured to:

[0042] Before generating a bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, if the license plate numbers in at least one portrait image included in a second file in the multiple files are all the same, then the second file is deleted.

[0043] In a third aspect, the present application provides a portrait clustering device, comprising:

[0044] a memory for storing program instructions;

[0045] The processor is configured to call the program instructions stored in the memory and execute the method according to any one of the first aspects according to the obtained program instructions.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods according to the first aspect.

[0047] In an embodiment of the present application, a plurality of archives are divided based on the similarity of a plurality of portrait images, and a bipartite graph is generated based on the plurality of vehicles contained in the plurality of archives and the number of portrait images corresponding to the different vehicles contained in each archive. The bipartite graph, the plurality of first eigenvectors associated with the plurality of archives, and the plurality of second eigenvectors associated with the plurality of vehicles are input into a graph neural network. The powerful node representation and aggregation capabilities of the graph neural network are utilized to perform feature fusion of the relationship between the plurality of archives and the plurality of vehicles with the first eigenvector. In other words, the person-vehicle relationship is fused with the attribute information of the archive, thereby obtaining a plurality of optimized first eigenvectors. If the similarity of any two optimized first eigenvectors is greater than the clustering threshold, the two archives corresponding to the any two optimized first eigenvectors are aggregated into one archive. Based on the optimized first eigenvector, the plurality of archives can be further aggregated, and the portrait images of the same person scattered in different archives can be re-clustered, thereby improving the accuracy of portrait clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of an application scenario of a portrait clustering method provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a process for clustering portraits provided in an embodiment of the present application;

[0051] Figure 3 A matrix diagram of the relationship level between files and vehicles provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a bipartite graph provided in an embodiment of the present application;

[0053] Figure 5 A structural diagram of a portrait clustering device provided in an embodiment of the present application;

[0054] Figure 6 This is a structural diagram of a portrait clustering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Unless there is a conflict, the embodiments in this application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0056] The terms "first" and "second" in the specification and claims of this application and the accompanying drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "comprising" and "including" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to the process, method, product, or device.

[0057] In the embodiment of the present application, "multiple" may mean at least two, for example, two, three or more, and the embodiment of the present application does not limit this.

[0058] With the widespread adoption of intelligent video surveillance equipment, a massive amount of portrait images is accumulated daily. For example, if one million portrait images include 100,000 individuals, the goal of a portrait clustering algorithm is to group these images into 100,000 clusters, with each cluster containing all images of the same person. Archiving these large numbers of portrait images by person is crucial for subsequent human trajectory analysis.

[0059] In addition to capturing images of pedestrians, intelligent video surveillance equipment can also capture facial images of people inside cars through car windows. A single person may drive multiple cars, often referred to as "one person, multiple cars." A single car may also be driven by multiple people, often referred to as "one car, multiple people." In these scenarios, all portrait images of the same person may not be grouped into the same cluster. Instead, they may be scattered across two or even multiple different clusters. Consequently, the accuracy of face clustering is low.

[0060] In order to improve the accuracy of portrait clustering, an embodiment of the present application provides a portrait clustering method, which can be performed by a portrait clustering device. The portrait clustering device can be implemented by a terminal or a server, such as a mobile phone, a personal computer (PC), a laptop computer, a camera device, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to this.

[0061] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios described below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0062] Please refer to Figure 1, is a schematic diagram of an application scenario of a portrait clustering method provided in an embodiment of the present application. The schematic diagram of the application scenario includes portrait images 110 to be clustered and a portrait clustering device 120. The portrait images 110 to be clustered can be collected by the portrait clustering device 120 itself. For example, the portrait clustering device 120 is a device with a camera function. The portrait images 110 to be clustered can also be collected by other camera devices and then sent to the portrait clustering device 120. After the portrait clustering device 120 obtains multiple portrait images 110 to be clustered, it clusters portrait images containing the same person into one category. The specific process of portrait clustering will be described in detail below.

[0063] As described above, the application scenarios are combined with Figure 1 The application scenario shown is Figure 1 Taking the portrait clustering method performed by the portrait clustering device 120 in the example as an example, a portrait clustering method provided by an embodiment of the present application is introduced. Figure 2 , which is a flow chart of a portrait clustering method provided in an embodiment of the present application.

[0064] S201. Acquire multiple files.

[0065] The multiple files can be sent to the portrait clustering device after being divided based on the similarity of multiple portrait images by other camera equipment, or they can be divided based on the similarity of multiple portrait images by the portrait clustering device. The portrait images are images of faces of people riding in a vehicle, where riding in a vehicle means sitting in the main driver's seat or the front passenger seat. Therefore, each portrait image contains a vehicle, which can be a motor vehicle or a non-motor vehicle.

[0066] The following describes the process of dividing multiple files based on the similarity of multiple portrait images.

[0067] Specifically, after other camera devices or portrait clustering devices collect multiple portrait pictures, existing clustering algorithms are used to cluster the multiple portrait pictures. Existing clustering algorithms include k-means clustering algorithm, density-based spatial clustering of applications with noise (DBSCAN) algorithm, etc., which are not specifically limited in the embodiments of the present application. For example, if the similarity between two portrait pictures is greater than the similarity threshold of the combined file, it is determined that the two portrait pictures belong to the same file, and the two portrait pictures are clustered into a cluster. Finally, the multiple portrait pictures are divided into different clusters, and a cluster is a file, thereby obtaining multiple files. Among them, each file contains at least one portrait picture of the same person, and the same person may take different vehicles. Therefore, the at least one portrait picture contained in each file corresponds to at least one vehicle.

[0068] After the portrait clustering device uses the existing clustering algorithm to obtain multiple files, due to image quality issues, all portrait images of the same person cannot usually be classified into the same file. All portrait images of the same person may be scattered in two or even more different files. Therefore, the embodiment of the present application further processes the multiple files and clusters the portrait images of the same person scattered in two or even more different files into one category, thereby improving the accuracy of portrait clustering.

[0069] Considering that drivers of special vehicles such as police cars and training vehicles often have similar clothing or only drive the vehicle briefly, including these portrait images directly in the calculation would introduce a certain amount of noise interference. Therefore, in one possible embodiment, if the first portrait image contained in the first file among the multiple files contains a vehicle of a preset type, the first portrait image is deleted.

[0070] Specifically, the portrait clustering device can detect whether a preset type of vehicle exists in all portrait images contained in each file. The preset type of vehicle, such as a police car or a training vehicle, can be determined based on the vehicle color, shape, license plate number, etc. of each portrait image. If a vehicle of the preset type exists in the first portrait image contained in the first file, the first portrait image is deleted. If a vehicle of the preset type exists in all portrait images contained in the first file, all portrait images contained in the first file are deleted, i.e., the first file is deleted.

[0071] Considering that the file contains only one license plate number, it cannot well reflect the relationship between multiple people in one car and multiple people in one car. Therefore, in a possible embodiment, if the license plate number in at least one portrait image contained in the second file in multiple files is the same, the second file is deleted.

[0072] Specifically, since each portrait image contains a vehicle, each portrait image also contains a license plate number. Each portrait clustering device can detect whether the license plate numbers in all portrait images contained in each file are the same. If the license plate numbers in all portrait images contained in the second file are the same, it means that the second file contains only one license plate number, and the second file is deleted.

[0073] In an embodiment of the present application, the portrait clustering device performs data cleaning and filtering on multiple files. The processed files contain fewer portrait images, which not only helps to improve the efficiency of the algorithm, but also saves computing power.

[0074] S202: Generate a bipartite graph according to the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file.

[0075] After the portrait clustering device obtains multiple files, there are multiple ways to generate a bipartite graph, which are introduced below.

[0076] The first method generates a bipartite graph according to the number of vehicles contained in the multiple files and the number of portrait images corresponding to different vehicles contained in each file.

[0077] Specifically, the portrait clustering device uses multiple archives as archive nodes on a bipartite graph, uses multiple vehicles contained in the multiple archives as vehicle nodes on the bipartite graph, and uses the number of portrait images corresponding to different vehicles contained in each archive as the value on the edge between the archive node and the vehicle node. The different vehicles contained in each archive refer to at least one vehicle corresponding to at least one portrait image contained in each archive. The multiple vehicles contained in multiple archives refer to all vehicles corresponding to all portrait images contained in the multiple archives. Each portrait image has a license plate number, and the portrait image corresponding to each vehicle is the portrait image with the same license plate number in each archive. The bipartite graph is used to represent the relationship between multiple archives and multiple vehicles.

[0078] For example, file 1 contains nine portrait images, eight of which have license plate A, meaning they correspond to the vehicle with license plate A. There is one portrait of the vehicle with license plate B, meaning it corresponds to the vehicle with license plate B. In the resulting bipartite graph, the edge between file node 1 and vehicle node A has a value of 8, while the edge between file node 1 and vehicle node B has a value of 1. File node 1 is the node where file 1 is located, while vehicle node A and vehicle node B are the nodes where vehicles with license plate numbers A and B are located, respectively.

[0079] The second method is to determine the relationship level between each file and different vehicles based on the number of portrait images corresponding to different vehicles contained in each file, and generate a bipartite graph based on the multiple vehicles contained in multiple files and the relationship level between each file and different vehicles.

[0080] Specifically, the portrait clustering device can calculate the number of portrait images corresponding to different vehicles in each profile. Based on the threshold interval within which this number falls, and the correspondence between each threshold interval and each relationship level, it determines the relationship level between each profile and the different vehicles. The relationship level can also be referred to as the interaction level. It should be noted that the interaction level here does not refer to the number of messages that can be sent between the profile and the vehicle, but rather indicates the number of portrait images with the same license plate number in each profile.

[0081] For example, the relationship level corresponding to threshold interval 1 is 1, and the relationship level corresponding to threshold interval 2 is 2. The specific formula is as follows:

[0082]

[0083] Among them, Level represents the relationship level, and num represents the number of portrait images corresponding to each vehicle. For example, the number of portrait images corresponding to vehicle A in file 1 is 100, so the relationship level between file 1 and vehicle A is 6.

[0084] Furthermore, after determining the relationship level between each profile and different vehicles, the portrait clustering device treats the multiple profiles as profile nodes on a bipartite graph, the multiple vehicles contained in the multiple profiles as vehicle nodes on the bipartite graph, and the relationship level between each profile and different vehicles as the value of the edge between the profile node and the vehicle node. The meaning of the different vehicles contained in each profile, the multiple vehicles contained in the multiple profiles, the portrait images corresponding to each vehicle, and the bipartite graph are discussed above and will not be repeated here.

[0085] Please refer to Figure 3 , a matrix diagram illustrating the relationship levels between files and vehicles provided in an embodiment of the present application. Among all the portrait images contained in file 1, there are two vehicles, license plate number A and license plate number C. The relationship level between file 1 and the vehicle with license plate number A is 3, indicating that the number of portrait images corresponding to license plate number A in file 1 is between 4 and 8. The relationship level between file 1 and the vehicle with license plate number C is 1, indicating that there is only one portrait image corresponding to license plate number C in file 1. The situations of other files and vehicles are similar and will not be repeated here.

[0086] Further, Figure 3 The matrix shown is converted into a bipartite graph, please refer to Figure 4 , a schematic diagram of a bipartite graph provided in an embodiment of the present application. Circles represent file nodes, numbers within the circles represent file numbers, squares represent vehicle nodes, letters within the squares represent vehicle license plate numbers, and the values ​​on the edges between the circles and squares represent the relationship level between the file nodes and the vehicle nodes. This bipartite graph includes five file nodes and four vehicle nodes.

[0087] For example, if file node 1 is connected to both vehicle node A and vehicle node C, this means that file number 1 contains two vehicles with license plates A and C. The value between file node 1 and vehicle node A is 3, indicating that the relationship level between file number 1 and vehicle with license plate A is 3. The value between file node 1 and vehicle node C is 1, indicating that the relationship level between file number 1 and vehicle with license plate C is 1. The situations for other files and vehicles are similar and will not be repeated here.

[0088] In the embodiment of the present application, considering that the portrait images corresponding to different vehicles may have a large difference in magnitude, for example, there are hundreds of portrait images corresponding to vehicle 1, but only a few portrait images corresponding to vehicle 2, the levels are divided according to the number of portrait images corresponding to each vehicle. This can eliminate the impact of the magnitude difference on the algorithm, avoid data truncation, and improve the computational efficiency of the algorithm.

[0089] S203 , inputting the bipartite graph, the multiple first eigenvectors associated with the multiple files, and the multiple second eigenvectors associated with the multiple vehicles into a graph neural network to obtain multiple optimized first eigenvectors.

[0090] Both files and vehicles have some fixed attribute information that can be used to distinguish different files and vehicles. The following first introduces the attribute information of files and vehicles respectively.

[0091] File attributes include: the number of portrait images contained in the file, the number of driver's seat and front passenger seat images in the file for different time periods, the N (e.g., 3) most frequently captured locations for different time periods, and the number of vehicles driven by the person in the file within a certain period (e.g., six months). For different time periods, such as morning, noon, afternoon, and evening, driver's seat images refer to images of the person in the driver's seat, while front passenger seat images refer to images of the person in the front passenger seat.

[0092] Vehicle attribute information includes: license plate number, vehicle color, vehicle type, whether it is a preset type of vehicle, the number of people who have driven the vehicle, N (for example, 3) locations where it is most frequently captured at different time periods, etc. Vehicle color, such as white, black, etc., vehicle type, such as sedan, truck, etc., preset types of vehicles, such as training vehicles, police cars, etc., and different time periods, such as morning, noon, afternoon, and evening.

[0093] The portrait clustering device represents the attribute information of each profile using a first feature vector to obtain multiple first feature vectors, each of which indicates the attribute information of a corresponding profile, and associates each first feature vector with the corresponding profile. Furthermore, the attribute information of each vehicle is represented by a second feature vector to obtain multiple second feature vectors, each of which indicates the attribute information of a corresponding vehicle, and associates each second feature vector with the corresponding vehicle.

[0094] Graph neural networks are a recently emerging deep learning method specifically targeted at graph data applications. By aggregating information from surrounding nodes to update node features, they can generate new, more informative node feature vectors. Therefore, after the portrait clustering device completes the construction and feature association of the bipartite graph, it can build a graph neural network to learn and iterate on node features. Graph neural networks can be based on the spectral domain or the spatial domain. Examples of spectral graph neural networks include ChebNet and GCN, while examples of spatial graph neural networks include GraphSAGE and GAT.

[0095] Specifically, the portrait clustering device inputs a bipartite graph, multiple first eigenvectors associated with multiple profiles, and multiple second eigenvectors associated with multiple vehicles into a graph neural network to obtain multiple optimized first eigenvectors. The graph neural network is used to integrate the relationships between multiple profiles and multiple vehicles and optimize the multiple first eigenvectors and multiple second eigenvectors. Therefore, the multiple optimized first eigenvectors are obtained by integrating the relationships between multiple profiles and multiple vehicles into the multiple first eigenvectors.

[0096] In one possible embodiment, the graph neural network can predict the predicted relationship level between multiple profiles and multiple vehicles, also known as the predicted interaction level. After the portrait clustering device divides the profiles based on the similarity of the portrait images, it uses the relationship level between the multiple profiles and multiple vehicles as the actual relationship level, also known as the actual interaction level. The graph neural network is then trained using the loss between the predicted relationship level and the actual relationship level to obtain an optimized first eigenvector. The specific steps are described in S1.1-S1.3.

[0097] S1.1. Input a bipartite graph, a plurality of first eigenvectors associated with a plurality of files, and a plurality of second eigenvectors associated with a plurality of vehicles into a graph neural network, and output predicted relationship levels between the plurality of files and the plurality of vehicles.

[0098] The portrait clustering device uses the image link prediction method and the softmax function to estimate the predicted relationship level between multiple files and multiple vehicles. The specific formula is as follows:

[0099]

[0100] in, represents the second eigenvector associated with the i-th vehicle, represents the first eigenvector of the j-th archive association, C i,j represents the predicted relationship level between the j-th file and the i-th vehicle, is the transformation matrix when the relationship level between the file and the vehicle is l.

[0101] S1.2. Calculate the loss between the actual relationship level and the predicted relationship level between the multiple files and the multiple vehicles.

[0102] For example, the formula of the loss function is as follows:

[0103]

[0104] Among them, N car Represents the total number of multiple vehicles, N id Indicates the total number of multiple files, C i,j represents the predicted relationship level between the j-th file and the i-th vehicle, is the actual relationship level between the j-th file and the i-th vehicle, Represents the loss between the actual relationship rank and the predicted relationship rank.

[0105] S1.3. Adjust the multiple first eigenvectors and the multiple second eigenvectors according to the loss until the loss reaches a minimum value, thereby obtaining multiple optimized first eigenvectors.

[0106] The portrait clustering device optimizes the parameters and node features of the graph neural network by continuously reducing the loss function. The node features include multiple first eigenvectors and multiple second eigenvectors. The optimization method is, for example, the stochastic gradient descent algorithm. When the loss function is at the minimum value, the corresponding node feature is the optimized node feature, thereby obtaining multiple optimized first eigenvectors.

[0107] In a possible embodiment, the portrait clustering device can pre-train the graph neural network, input the bipartite graph, multiple first eigenvectors associated with multiple files, and multiple second eigenvectors associated with multiple vehicles into the trained graph neural network, and directly output the optimized first eigenvector.

[0108] The following uses the training of a graph neural network using a portrait clustering device as an example to illustrate the training process of a graph neural network:

[0109] Specifically, the graph neural network is pre-trained based on a sample dataset, which includes file features associated with multiple sample files, vehicle features associated with multiple sample vehicles, and the actual relationship levels between the multiple sample files and the multiple sample vehicles. The portrait clustering device first constructs a bipartite graph of the multiple sample files and the multiple sample vehicles. This bipartite graph, the file features associated with the multiple sample files, and the vehicle features associated with the multiple sample vehicles serve as input to the graph neural network. The predicted relationship levels between the multiple sample files and the multiple sample vehicles, as well as the actual relationship levels between the multiple sample files and the multiple sample vehicles, output by the graph neural network, serve as feedback data for the graph neural network. The graph neural network's parameter values ​​are continuously adjusted based on this feedback data. After training on a large number of sample files and sample vehicles, the graph neural network's parameters are continuously updated until the error between the predicted relationship levels and the actual relationship levels between the multiple sample files and the multiple sample vehicles falls within a preset range, resulting in a trained graph neural network.

[0110] S204: If the similarity between any two optimized first feature vectors is greater than a clustering threshold, the two files corresponding to the any two optimized first feature vectors are aggregated into one file.

[0111] The clustering threshold may be a pre-fixed threshold, for example, the clustering threshold is pre-set to 0.9. The clustering threshold may also be a dynamic threshold, for example, the clustering threshold is dynamically determined each time according to a formula.

[0112] In a possible embodiment, the portrait clustering device may determine the clustering threshold based on the combined file similarity threshold, the similarity between any two optimized first feature vectors, and the maximum adjustable similarity threshold.

[0113] Among them, the combined file similarity threshold is a similarity threshold for determining whether two portrait pictures belong to the same file based on the similarity of multiple portrait pictures, that is, the combined file similarity threshold in S201 discussed above. The maximum adjustable similarity threshold is used to indicate the lower limit of the similarity of two files as the same file. For example, the combined file similarity threshold is 0.9, and the maximum adjustable similarity threshold is 0.2. If the similarity between the two files is less than 0.9-0.2=0.7, it means that the two files cannot be the same file. The combined file similarity threshold is positively correlated with the clustering threshold, the similarity of any two optimized first eigenvectors is negatively correlated with the clustering threshold, and the maximum adjustable similarity threshold is negatively correlated with the clustering threshold.

[0114] For example, the formula for dynamically determining the clustering threshold is as follows:

[0115] C pic_adj =C pic -S no-pic C bias

[0116] Among them, C pic_adj is the clustering threshold after dynamic adjustment, C bias is the maximum adjustable similarity threshold, generally 0.1 to 0.2. pic It is the similarity threshold of the combined files, which is generally 0.85 to 0.95.

[0117] In the embodiment of the present application, considering that the optimized feature vector incorporates more information after integrating the relationship between the file and the vehicle, its clustering threshold should be appropriately reduced to better judge the similarity between the optimized feature vectors. By dynamically adjusting the clustering threshold, the clustering effect can be further optimized.

[0118] Furthermore, after determining the clustering threshold, the portrait clustering device calculates the similarity of any two optimized first feature vectors, such as cosine similarity. If the similarity is greater than the clustering threshold, the two files corresponding to the any two optimized first feature vectors are aggregated into one file.

[0119] In a possible embodiment, after the portrait clustering device executes S201-S204, it can cluster portrait images of the same person scattered in two or even more different files into one category, thereby obtaining multiple optimized aggregated files, and binding each optimized aggregated file to the ID card of the corresponding person to facilitate personnel trajectory analysis.

[0120] Based on the same inventive concept, this application also provides a portrait clustering device, please refer to Figure 5 , the device comprises:

[0121] An acquisition module 501 is configured to acquire a plurality of files, wherein the plurality of files are divided based on the similarity of a plurality of portrait images, wherein the portrait images are images of faces of people riding in vehicles, and each file contains at least one portrait image corresponding to at least one vehicle;

[0122] Generating module 502, configured to generate a bipartite graph based on the plurality of vehicles included in the plurality of archives and the number of portrait images corresponding to the different vehicles included in each archive; wherein the portrait image corresponding to each vehicle is a portrait image of a person with the same license plate number in each archive, and the bipartite graph is used to represent the relationship between the plurality of archives and the plurality of vehicles;

[0123] An acquisition module 503 is configured to input the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; wherein each first eigenvector is used to indicate attribute information of a corresponding file, and each second eigenvector is used to indicate attribute information of a corresponding vehicle, and the plurality of optimized first eigenvectors are obtained by integrating the relationships between the plurality of files and the plurality of vehicles into the plurality of first eigenvectors;

[0124] The aggregation module 504 is configured to aggregate two files corresponding to any two optimized first feature vectors into one file if the similarity between the two optimized first feature vectors is greater than a clustering threshold.

[0125] In a possible embodiment, the generating module 502 is specifically configured to:

[0126] According to the number of portrait images corresponding to different vehicles contained in each file, the relationship level between each file and different vehicles is determined;

[0127] A bipartite graph is generated according to the multiple vehicles contained in the multiple files and the relationship level between each file and different vehicles.

[0128] In a possible embodiment, the obtaining module 503 is specifically configured to:

[0129] Inputting a bipartite graph, a plurality of first eigenvectors associated with a plurality of files, and a plurality of second eigenvectors associated with a plurality of vehicles into a graph neural network, outputting predicted relationship levels between the plurality of files and the plurality of vehicles; wherein the graph neural network is used to integrate the relationships between the plurality of files and the plurality of vehicles and optimize the plurality of first eigenvectors and the plurality of second eigenvectors;

[0130] Calculating the loss between the actual relationship level and the predicted relationship level between multiple profiles and multiple vehicles;

[0131] The plurality of first eigenvectors and the plurality of second eigenvectors are adjusted according to the loss until the loss reaches a minimum value, thereby obtaining a plurality of optimized first eigenvectors.

[0132] In a possible embodiment, the apparatus further includes a determining module 505, and the determining module 505 is configured to:

[0133] If the similarity between any two optimized first feature vectors is greater than the clustering threshold, before aggregating the two files corresponding to the two optimized first feature vectors into one file, determine the clustering threshold based on the file-merging similarity threshold, the similarity between any two optimized first feature vectors, and the maximum adjustable similarity threshold;

[0134] Among them, the combined file similarity threshold is a similarity threshold for determining whether two portrait pictures belong to the same file based on the similarity of multiple portrait pictures. The maximum adjustable similarity threshold is used to indicate the lower limit of the similarity of two files as the same file. The combined file similarity threshold is positively correlated with the clustering threshold. The similarity of any two optimized first eigenvectors is negatively correlated with the clustering threshold. The maximum adjustable similarity threshold is negatively correlated with the clustering threshold.

[0135] In a possible embodiment, the device further includes a deletion module 506, and the deletion module 506 is configured to:

[0136] Before generating a bipartite graph based on the multiple vehicles contained in the multiple files and the number of portrait images corresponding to different vehicles contained in each file, if a first portrait image contained in a first file in the multiple files contains a vehicle of a preset type, the first portrait image is deleted.

[0137] In a possible embodiment, the deletion module 506 is further configured to:

[0138] Before generating a bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, if the license plate number in at least one portrait image included in a second file in the multiple files is the same, the second file is deleted.

[0139] As an example, Figure 5 The apparatus discussed can be used to perform Figure 2 The portrait clustering method described in the embodiment shown in FIG. 1 is a method for clustering portraits. Therefore, for the functions that can be realized by each functional module of the device, reference can be made to FIG. Figure 2 The description of the illustrated embodiment will not be repeated here.

[0140] It should be noted that although several modules or submodules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0141] Based on the same inventive concept, the present application also provides a portrait clustering device, which is equivalent to the portrait clustering device discussed above. Please refer to Figure 6 , the device includes:

[0142] At least one processor 601, and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 In the example, the processor 601 and the memory 602 are connected via a bus 600. Figure 6 The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 6 The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 may also be referred to as a controller, without limitation to the name.

[0143] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. At least one processor 601 can execute the instructions stored in the memory 602. Figure 2 The processor 601 can also implement any of the portrait clustering methods. Figure 5 The functions of each module in the device shown.

[0144] Among them, the processor 601 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 602 and calling data stored in the memory 602, the various functions of the device and processing data.

[0145] In one possible design, processor 601 may include one or more processing units. Processor 601 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into processor 601. In some embodiments, processor 601 and memory 602 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0146] The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the portrait clustering method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0147] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0148] By designing and programming the processor 601, the code corresponding to the portrait clustering method described in the above embodiment can be fixed into the chip, so that the chip can execute the image clustering method when running. Figure 2 How to design and program the processor 601 is a technique well known to those skilled in the art and will not be described in detail here.

[0149] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium storing a computer program, which includes program instructions. When executed by a computer, the program instructions cause the computer to perform any of the portrait clustering methods discussed above. Because the principles underlying the problem solved by the computer-readable storage medium are similar to those of the portrait clustering method, the implementation of the computer-readable storage medium can be referred to as the implementation of the method, and any repetitions will not be repeated here.

[0150] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable portrait clustering device to produce a machine, so that the instructions executed by the processor of the computer or other programmable portrait clustering device generate instructions for implementing the processes in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions can also be stored in a computer readable memory that can guide a computer or other programmable portrait clustering device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0154] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A portrait clustering method, characterized in that: include: Acquire a plurality of files, wherein the plurality of files are divided based on similarity of a plurality of portrait images, the portrait images being images of faces of people riding in vehicles, and at least one portrait image contained in each file corresponds to at least one vehicle; Generate a bipartite graph based on the number of vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file; wherein the portrait image corresponding to each vehicle is a portrait image of a person with the same license plate number in each file, and the bipartite graph is used to represent the relationship between the multiple files and the multiple vehicles; Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; wherein each first eigenvector is used to indicate attribute information of a corresponding file, and each second eigenvector is used to indicate attribute information of a corresponding vehicle, and the plurality of optimized first eigenvectors are obtained by integrating the relationships between the plurality of files and the plurality of vehicles into the plurality of first eigenvectors; If the similarity between any two optimized first feature vectors is greater than the clustering threshold, the two files corresponding to the two optimized first feature vectors are aggregated into one file; The generating of the bipartite graph according to the plurality of vehicles included in the plurality of files and the number of portrait images corresponding to different vehicles included in each file includes: The multiple archives are used as archive nodes on the bipartite graph, the multiple vehicles included in the multiple archives are used as vehicle nodes on the bipartite graph, and the number of portrait images corresponding to different vehicles contained in each archive is used as the value of the edge between the archive node and the vehicle node.

2. The method according to claim 1, wherein Generating a bipartite graph according to the plurality of vehicles included in the plurality of files and the number of portrait images corresponding to different vehicles included in each file, further comprising: According to the number of portrait images corresponding to different vehicles contained in each file, the relationship level between each file and different vehicles is determined; A bipartite graph is generated according to the plurality of vehicles included in the plurality of archives and the relationship level between each archive and different vehicles.

3. The method according to claim 2, wherein Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors, including: Inputting the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network, and outputting predicted relationship levels between the plurality of files and the plurality of vehicles; wherein the graph neural network is used to integrate the relationships between the plurality of files and the plurality of vehicles and optimize the plurality of first eigenvectors and the plurality of second eigenvectors; calculating a loss between actual relationship levels and the predicted relationship levels between the plurality of profiles and the plurality of vehicles; The multiple first eigenvectors and the multiple second eigenvectors are adjusted according to the loss until the loss reaches a minimum value, thereby obtaining multiple optimized first eigenvectors.

4. The method according to any one of claims 1 to 3, wherein If the similarity between any two optimized first feature vectors is greater than a clustering threshold, before aggregating the two files corresponding to the any two optimized first feature vectors into one file, the method further includes: Determining the clustering threshold according to the combined file similarity threshold, the similarity between any two optimized first feature vectors, and the maximum adjustable similarity threshold; Among them, the combined file similarity threshold is a similarity threshold for determining whether two portrait pictures belong to the same file based on the similarity of the multiple portrait pictures, the maximum adjustable similarity threshold is used to indicate the lower limit of the similarity of two files as the same file, the combined file similarity threshold is positively correlated with the clustering threshold, the similarity of any two optimized first eigenvectors is negatively correlated with the clustering threshold, and the maximum adjustable similarity threshold is negatively correlated with the clustering threshold.

5. The method according to any one of claims 1 to 3, wherein Before generating a bipartite graph based on the multiple vehicles included in the multiple files and the number of portrait images corresponding to different vehicles included in each file, the method further includes: If a vehicle of a preset type is included in a first portrait picture included in a first file among the multiple files, the first portrait picture is deleted.

6. The method according to any one of claims 1 to 3, wherein: The at least one portrait image is a plurality of portrait images; before generating the bipartite graph based on the plurality of vehicles included in the plurality of files and the number of portrait images corresponding to different vehicles included in each file, the method further includes: If the license plate numbers in the multiple portrait images included in the second file among the multiple files are all the same, the second file is deleted.

7. A portrait clustering device, characterized in that: include: an acquisition module, configured to acquire a plurality of files, wherein the plurality of files are divided based on similarity of a plurality of portrait images, the portrait images being images of faces of people riding in vehicles, and at least one portrait image contained in each file corresponds to at least one vehicle; a generating module configured to generate a bipartite graph based on the plurality of vehicles included in the plurality of archives and the number of portrait images corresponding to different vehicles included in each archive; wherein the portrait image corresponding to each vehicle is a portrait image of a person with the same license plate number in each archive, and the bipartite graph is used to represent the relationship between the plurality of archives and the plurality of vehicles; an obtaining module, configured to input the bipartite graph, the plurality of first eigenvectors associated with the plurality of files, and the plurality of second eigenvectors associated with the plurality of vehicles into a graph neural network to obtain a plurality of optimized first eigenvectors; wherein each first eigenvector is used to indicate attribute information of a corresponding file, and each second eigenvector is used to indicate attribute information of a corresponding vehicle, and the plurality of optimized first eigenvectors are obtained by integrating the relationships between the plurality of files and the plurality of vehicles into the plurality of first eigenvectors; an aggregation module, configured to aggregate two files corresponding to any two optimized first feature vectors into one file if the similarity between the two optimized first feature vectors is greater than a clustering threshold; The generation module is specifically used to: use the multiple archives as archive nodes on the bipartite graph, use the multiple vehicles included in the multiple archives as vehicle nodes on the bipartite graph, and use the number of portrait images corresponding to different vehicles contained in each archive as the value of the edge between the archive node and the vehicle node.

8. The device according to claim 7, wherein The generation module is specifically used for: According to the number of portrait images corresponding to different vehicles contained in each file, the relationship level between each file and different vehicles is determined; A bipartite graph is generated according to the plurality of vehicles included in the plurality of archives and the relationship level between each archive and different vehicles.

9. A portrait clustering device, characterized in that: include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 6 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images